Data Engineer

Job Description Job Title: Senior Data Engineer Location: UK (Hybrid, 2–3 days per week in-office) Rate: £446/day (Inside IR35) Contract Duration: 6 months Additional Requirements: May require occasional travel to Dublin office About The Role We are looking for an experienced Senior Data Engineer to join a Data & Analytics (DnA) team. You will design, build, and operate production-grade data products across customer, commercial, financial, sales, and broader data domains. This role is hands-on and heavily focused on Databricks-based engineering, data quality, governance, and DevOps-aligned delivery. You will work closely with the Data Engineering Manager, Product Owner, Data Product Manager, Data Scientists, Head of Data & Analytics, and IT teams to transform business requirements into governed, decision-grade datasets embedded in business processes and trusted for reporting, analytics, and advanced use cases. Key Responsibilities • Design, build, and maintain pipelines in Databricks using Delta Lake and Delta Live Tables. • Implement medallion architectures (Bronze/Silver/Gold) and deliver reusable, discoverable data products. • Ensure pipelines meet non-functional requirements such as freshness, latency, completeness, scalability, and cost-efficiency. • Own and operate Databricks assets including jobs, notebooks, SQL, and Unity Catalog objects. • Apply Git-based DevOps practices, CI/CD, and Databricks Asset Bundles to safely promote changes across environments. • Implement monitoring, alerting, runbooks, incident response, and root-cause analysis. • Enforce governance and security using Unity Catalog (lineage, classification, ACLs, row/column-level security). • Define and maintain data-quality rules, expectations, and SLOs within pipelines. • Support root-cause analysis of data anomalies and production issues. • Partner with Product Owner, Product Manager, and business stakeholders to translate requirements into functional and non-functional delivery scope. • Collaborate with IT platform teams to define data contracts, SLAs, and schema evolution strategies. • Produce clear technical documentation (data contracts, source-to-target mappings, release notes). Essential Skills & Experience • 6+ years in data engineering or advanced analytics engineering roles. • Strong hands-on expertise in Python and SQL. • Proven experience building production pipelines in Databricks. • Excellent attention to detail, with the ability to create effective documentation and process diagrams. • Solid understanding of data modelling, performance tuning, and cost optimisation. Desirable Skills & Experience • Hands-on experience with Databricks Lakehouse, including Delta Lake and Delta Live Tables for batch/stream pipelines. • Knowledge of pipeline health monitoring, SLA/SLO management, and incident response. • Unity Catalog governance and security expertise, including lineage, table ACLs, and row/column-level security. • Familiarity with Databricks DevOps/DataOps practices (Git-based development, CI/CD, automated testing). • Performance and cost optimization strategies for Databricks (autoscaling, Photon/serverless, partitioning, Z-Ordering, OPTIMIZE/VACUUM). • Semantic layer and metrics engineering experience for consistent business metrics and self-service analytics. • Experience with cloud-native analytics platforms (preferably Azure) operating as enterprise-grade production services.