Databricks Data Engineer

Roles & Responsibilities: • Collaborate closely with data scientists, data engineers, and business stakeholders to gather requirements and understand the business objectives driving data pipeline development. • Design, develop, and maintain robust, scalable high-performance Data Pipelines using Databricks. • Leverage Databricks features such as Lakehouse and Delta Lake for efficient data storage and Spark for distributed processing • Develop ETL/ELT pipeline using Databricks • Monitor pipeline health, troubleshoot data issues • Migrate on Prem Pyspark, SAS data pipeline and ML Models to Databricks • Define and implement best practices in Databricks • Evaluate new Databricks features and tools, helping the organization stay at the forefront of innovation in AI and Big Data • Collaborate with cross-functional teams to identify and resolve data-related issues. Qualifications: • Proven expertise in implementing Lakehouse and Delta Lake using Databricks. • Strong PySpark and Python experience • Databricks Certified Data Engineer Professional Certification • Familiarity with ML Ops/LLM Ops and distributed systems. • Experience with Big Data platform like Cloudera Hadoop and Could platforms like AWS, GCP. • Solid understanding of system design patterns, scalability, observability, and performance tuning. • Strong analytical and problem-solving skills. • Passion for exploring and building with emerging technologies. Good to Have Skills: • AWS EKS Experience, Dockers and Containers