Job SummaryJob Description: Must Have Skills – Skill 1 – Yrs of Exp – Azure Databricks Skill 2 – Yrs of Exp – Python Skill 3 – Yrs of Exp – Spark Streaming Good To have Skills – Skill 1 – Yrs of Exp – Java or Scala Skill 2 – Yrs of Exp – CI/CD pipelines Skill 3 – Yrs of Exp –GitHub workflows \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_About the Role:We are looking for a highly skilled Senior Azure Databricks (ADB) Developer to join our Data Engineering team. This role involves developing large-scale batch and streaming data pipelines on Azure Cloud. The ideal candidate will have strong expertise in Python, Databricks Notebooks, Apache Spark (including Structured Streaming), and real-time integration with Kafka. You will work with both relational databases like DB2 and NoSQL systems such as MongoDB, focusing on performance optimization and scalable architecture.Key Responsibilities: Design and Develop: Create real-time and batch data pipelines using Azure Databricks, Apache Spark, and Structured Streaming. Data Processing: Write efficient ETL scripts and automate workflows using Python. Data Integration: Integrate with various data sources and destinations, including DB2, MongoDB, and other enterprise-grade data systems. Performance Optimization: Tune Spark jobs for optimal performance and cost-effective compute usage on Azure. Collaboration: Work with platform and architecture teams to ensure secure, scalable, and maintainable cloud data infrastructure. CI/CD Support: Implement CI/CD for Databricks pipelines and notebooks using tools like GitHub and Azure DevOps. Stakeholder Communication: Interface with product owners, data scientists, and business analysts to translate data requirements into production-ready pipelines. Required Skills: 10+ years of experience in data engineering Python Proficiency: Data Manipulation: Using libraries like Pandas and NumPy for data manipulation and analysis. Data Processing: Writing efficient ETL scripts. Automation: Automating repetitive tasks and workflows. Debugging: Strong debugging skills to troubleshoot and optimize code. Database Management: SQL: Advanced SQL skills for querying and managing relational databases. NoSQL: Experience with NoSQL databases like MongoDB or Cassandra. Data Warehousing: Knowledge of data warehousing solutions like Google BigQuery or Snowflake. Big Data Technologies: Kafka: Knowledge of data streaming platforms like Apache Kafka. Version Control: Git: Using version control systems for collaborative development. Data Modeling: Schema Design: Designing efficient and scalable database schemas. Data Governance: Ensuring data quality, security, and compliance. Database Management: DB2: Understanding of DB2 architecture, SQL queries, and database management. MongoDB: Knowledge of MongoDB schema design, indexing, and query optimization. Programming Skills: Proficiency in languages such as Java, Python, or JavaScript to write scripts for data extraction and transformation. Experience with BSON (Binary JSON) for data conversion. Cloud Services: Experience with cloud platforms like AWS or Azure for deploying and managing databases. Preferred Skills: Experience with Java or Scala in Spark streaming. Familiarity with Azure services like Data Lake, Data Factory, Synapse, and Event Hubs. Background in building data platforms in regulated or large-scale enterprise environments. Job Type: Contract Pay: $62.22 - $65.27 per hour Expected hours: 40 per week Work Location: Hybrid remote in Pleasanton, CA 94566