Data Engineer (SC Clearance)

Role: Data Engineer (SC Cleared) Location: London, UK (Hybrid) Contract Type Clearance: Active SC Clearance is mandatory Role Responsibilities • Lead the design, development, and deployment of scalable, secure, and cost-effective distributed data solutions using Azure services (e.g., Azure Databricks, Azure Data Lake Storage, Azure Data Factory). • Architect and implement advanced data pipelines using Databricks, Delta Lake, Python and Spark, ensuring performance, reliability, and maintainability across cloud and on-prem environments. • Champion data quality, governance, and observability, ensuring data is accurate, timely, and fit-for-purpose for analytics, BI, and operational use cases. • Drive the modernization of legacy systems, leading the migration of data infrastructure to Azure with minimal disruption and long-term scalability. • Act as a technical authority on Azure-native data engineering, guiding best practices and setting standards across the team. • Mentor and coach junior and mid-level engineers, fostering a culture of continuous learning, innovation, and technical excellence. • Collaborate with architects, analysts, and stakeholders to align data engineering efforts with strategic business goals and enterprise data strategy. • Evaluate and introduce emerging technologies, tools, and methodologies to enhance the Bank’s data capabilities. • Own the end-to-end delivery of complex data solutions, from requirements gathering to production deployment and support. • Contribute to the development of reusable frameworks, templates, and patterns to accelerate delivery and ensure consistency across projects. Minimum Criteria • Extensive experience with Azure services including Azure Databricks, Azure Data Lake Storage, and Azure Data Factory. • Advanced proficiency in SQL, Python, and Spark (PySpark), with a strong focus on performance optimization and distributed processing. • Proven experience in CI/CD practices using industry-standard tools (e.g., GitHub Actions, Azure DevOps). • Strong understanding of data architecture principles and cloud-native design patterns. Essential Criteria • Demonstrated ability to lead technical delivery, mentor engineering teams and collaborate with stakeholders to ensure alignment between data solutions and business strategy. • Proficiency in Linux/Unix environments and shell scripting. • Deep understanding of source control, testing strategies, and agile development practices. • Self-motivated with a strategic mindset and a passion for driving innovation in data engineering. Desirable Criteria • Experience delivering data pipelines on Hortonworks/Cloudera on-prem and leading cloud migration initiatives. • Familiarity with: • Apache Airflow • Data modelling and metadata management • Experience influencing enterprise data strategy and contributing to architectural governance.