Senior Data Engineer

About the Role Kanini Software Solutions is seeking an experienced Senior Data Engineer with strong expertise in Azure Databricks, Spark, and Azure data services to design and deliver scalable, cloud-based data solutions for healthcare analytics. The ideal candidate will have hands-on experience in building modern data platforms, implementing CI/CD pipelines, and enabling advanced analytics and machine learning initiatives while ensuring data security and regulatory compliance. Responsibilities The Senior Data Engineer will be responsible for designing, developing, and maintaining scalable, secure, and high-performance ETL/ELT pipelines using Azure Databricks, PySpark, and Synapse Analytics. • Design, develop, and maintain scalable, secure, and high-performance ETL/ELT pipelines using Azure Databricks, PySpark, Synapse Analytics. • Work with structured and unstructured data from various healthcare sources (EHR, claims data, etc.). • Implement and optimize data storage solutions using Azure technologies such as Databricks, Azure Data Lake, Azure SQL, Synapse Analytics. • Develop and deploy machine learning models and predictive analytics solutions to support healthcare analytics. • Ensure data quality, integrity, and security in compliance with HIPAA and other healthcare regulations. • Collaborate with data scientists, analysts, and software engineers to deliver high-performance data solutions. • Automate data workflows using CI/CD and other cloud tools. • Monitor and troubleshoot data pipelines to ensure optimal performance and reliability. • Participate in technical design reviews and contribute to data platform architecture decisions. • Promote data engineering best practices across the organization. Qualifications Required: • 8+ years of experience as a Data Engineer with a focus on Azure cloud services. • Strong understanding of Delta Lake, Lakehouse Architecture, and data warehousing concepts. • Strong expertise in SQL, Python, Spark, and Databricks. • Azure/Databricks certifications such as Microsoft Certified: Azure Data Engineer Associate. • Hands-on experience with Azure Data Factory, Azure Synapse, Azure Data Lake, and Power BI. • Knowledge of machine learning models and data science techniques in a healthcare setting. • Experience working with FHIR, HL7, and other healthcare data standards. • Familiarity with DevOps and CI/CD pipelines for data workflows. • Strong problem-solving skills and ability to work in an agile environment. Preferred: • Experience with MLOps and machine learning model lifecycle management. • Experience with Healthcare AI and Predictive Analytics use cases. • Exposure to Microsoft Fabric and modern analytics platforms. • Experience working with multi-cloud data platforms.