Senior Data Engineer (Azure + Databricks Lakehouse)

We are seeking a highly skilled Senior Data Engineer to join a high-performing data engineering team focused on building scalable data pipelines, integrations, and enterprise-grade data products that support analytics and operational use cases. This role requires deep expertise in Azure cloud data engineering within a Databricks Lakehouse environment. The ideal candidate will collaborate with engineers, architects, analysts, and product managers to design and implement robust, scalable, and high-performance data solutions. You will work with minimal supervision, exercise strong technical judgment, and proactively recommend and implement solutions aligned with business and technology goals. Key Responsibilities Data Engineering & Technical Delivery • Design, develop, and maintain scalable data pipelines and ETL/ELT processes using PySpark and SparkSQL • Build and manage orchestration workflows using Azure Data Factory • Work with Azure Data Lake and cloud-based storage systems for large-scale data processing • Implement streaming solutions using Kafka and/or Azure Event Hub • Optimize data pipelines for performance, scalability, reliability, and cost efficiency • Apply best practices for data partitioning, indexing, and storage formats such as Parquet • Analyze DAGs and system performance to identify bottlenecks and improve efficiency • Implement and maintain robust CI/CD pipelines using Azure DevOps System Design & Architecture • Contribute to data architecture design including HLDs, LLDs, and data models • Understand system interactions, dependencies, and cross-platform data flows • Build end-to-end data solutions across ingestion, transformation, and consumption layers • Apply distributed computing concepts such as fault tolerance, idempotency, and scalability • Identify opportunities to automate and optimize existing data processes Cross-Functional Collaboration • Partner with engineering, analytics, and product teams to deliver scalable technical solutions • Translate business requirements into technical designs and implementation strategies • Lead technical discussions and contribute to sprint planning and solution design • Mentor junior engineers and support overall team development Code Quality, Testing & Documentation • Write clean, maintainable, and efficient code aligned with engineering standards • Conduct code reviews and ensure adherence to best practices • Develop and review unit tests and test plans • Maintain technical documentation including architecture diagrams and process documentation • Perform root cause analysis (RCA) and implement quality improvements Project & Delivery Management • Deliver assigned modules and user stories within timelines • Support effort estimation, sprint planning, and release management activities • Monitor delivery progress and ensure compliance with engineering standards • Participate in deployment and production support processes Innovation & Continuous Improvement • Design and implement modern data engineering solutions and frameworks • Evaluate emerging technologies and explore AI/ML and Agentic AI use cases • Continuously improve systems for performance, scalability, and maintainability • Operate effectively in fast-paced and evolving environments Communication & Leadership • Create clear technical documentation and presentations for stakeholders • Communicate architecture decisions, implementation strategies, and technical processes • Mentor engineers and contribute to knowledge-sharing initiatives • Collaborate with stakeholders to clarify requirements and present solutions Required Skills & Qualifications Technical Skills • Strong hands-on experience with the Azure Data Engineering ecosystem, including: • Azure Data Factory • Azure Data Lake • Azure DevOps (CI/CD) • Proficiency in: • SQL (T-SQL, PostgreSQL) • PySpark • SparkSQL • Experience with: • Databricks Lakehouse architecture • Kafka and/or Azure Event Hub • Parquet and modern data storage formats • Strong understanding of: • Data partitioning and indexing • Distributed computing principles • Performance tuning and optimization of data pipelines Professional Skills • Strong analytical and problem-solving abilities • Ability to work independently with minimal supervision • Excellent communication and documentation skills • Experience working in Agile environments (Scrum/Kanban) • Ability to manage multiple priorities in fast-paced environments Preferred Qualifications • Experience designing end-to-end data platforms or lakehouse architectures • Exposure to AI/ML or Agentic AI applications • Prior experience mentoring or leading engineering teams • Relevant Azure or Data Engineering certifications Performance Expectations • Deliver high-quality, scalable, and maintainable solutions • Adhere to coding standards and engineering best practices • Reduce defects and improve system performance • Contribute to team knowledge sharing and continuous improvement initiatives About Brickred Systems: Brickred Systems is a global leader in next-generation technology, consulting, and business process service companies. We enable clients to navigate their digital transformation. Brickred Systems delivers a range of consulting services to our clients across multiple industries around the world. Our practices employ highly skilled and experienced individuals with a client-centric passion for innovation and delivery excellence. With ISO 27001 and ISO 9001 certification and over a decade of experience in managing the systems and workings of global enterprises, we harness the power of cognitive computing hyper-automation, robotics, cloud, analytics, and emerging technologies to help our clients adapt to the digital world and make them successful. Our always-on learning agenda drives their continuous improvement through building and transferring digital skills, expertise, and ideas from our innovation ecosystem.