Software Data Engineer / Databricks

Are you passionate about building scalable data solutions that power real-time insights and machine learning? Join our team as a Software Data Engineer, where you'll design and maintain cutting-edge data architectures on Azure using Databricks. What You’ll Do: • Design and optimize data pipelines and ETL/ELT workflows using Databricks on Azure. • Build and manage modern data platforms including Data Lakes, Warehouses, and Lakehouses for both streaming and batch processing. • Integrate large-scale datasets from diverse sources using Delta Lake and other formats. • Develop feature stores and data prep workflows for ML applications. • Ensure data quality through validation frameworks and governance standards. • Monitor and optimize workflows for performance, cost-efficiency, and reliability. What We’re Looking For: • 3+ years of hands-on experience with Databricks on Azure, Apache Spark (PySpark/Spark SQL), and Delta Lake. • Strong knowledge of Azure Data Factory, ADLS, Synapse Analytics, Azure ML Studio. • Proficiency in Python, SQL, Unix/Linux scripting; Java/Scala is a plus. • Experience with streaming technologies like Apache Kafka, Azure Event Hubs, and Azure Stream Analytics. • Familiarity with CI/CD tools (Azure DevOps, GitHub Actions), workflow orchestration (Airflow, ADF, Databricks Workflows). Bonus Skills: • Experience with ML frameworks (scikit-learn, TensorFlow, PyTorch) and feature engineering on Azure. • Familiarity with LLM applications, prompt engineering, and AI agent frameworks (Azure OpenAI, Semantic Kernel). • Containerization experience with Docker, AKS. • Monitoring tools like Azure Monitor, Datadog, Grafana. Education: • Bachelor’s or Master’s degree in Computer Science, Engineering, or a related field.