Sr Data Engineer -- Pipeline

KAYGEN is an emerging leader in providing top talent for technology-based staffing services. We specialize in providing high-volume contingent staffing, direct hire staffing and project-based solutions to companies worldwide ranging from startups to Fortune 500 and Managed Service Providers (MSP) across a wide variety of industries. • Design, develop, and optimize data pipelines using Databricks and Medallion architecture for structured and unstructured data. • Implement data ingestion strategies from diverse sources (SAP, TrackWise, Salesforce, and others) ensuring high-quality, reliable data flow. • Ingest and process large volumes of data from Azure Blob Storage into Databricks for transformation and analytics. • Build and maintain silver and gold layers for curated, analytics-ready datasets. • Apply advanced statistical methods , including: • Anomaly Detection for identifying irregular patterns in operational and quality data. • Forecasting for demand, defect trends, and resource planning. • Root Cause Analysis to identify drivers of failures or complaints. • Risk Modeling to quantify and predict potential quality or operational risks. • Predictive Analytics for proactive decision-making and maintenance. • Pattern Detection to uncover hidden relationships in large datasets. • Develop and deploy NLP solutions for text-based data analysis, leveraging linguistic schemas for structured interpretation of unstructured data. • Create Power BI dashboards with: • Workflow automation (notifications, reminders, messaging). • AI-driven insights using prepped datasets for predictive and prescriptive analytics. • Implement near real-time tracking systems and alert mechanisms for critical events and operational KPIs. • Collaborate with cross-functional teams to align data solutions with business objectives and ensure visibility across the end-to-end operating cycle . • Document data workflows, maintain traceability, and ensure audit readiness for all data engineering activities. • Leverage knowledge of Quality Management Systems (QMS) to support compliance and continuous improvement. At KAYGEN, we are always looking for dynamic, talented and experienced individuals. We invite you to join our team of talented IT professionals, consulting at client locations across the globe. Our culture is team-orientated; we strive to stand by our core values of respect, honesty and integrity. Our team of experienced staffing experts will work with you to find you the best opportunity. For more information, please visit us at www.kaygen.com. Benefits: • Free Healthcare Insurance • Vision and Dental Insurance • 401(k) Retirement Plan • Free Life Insurance • Sick Time Off Achieve your Kaizen by clicking here. A unique and exclusive talent community supported by Kaygen, that includes programs like: • Certifications • Mentorship Program • Referrals • Family and Wellness benefits • Continuous Growth and Career Development