Senior Data Engineer – Spark & Databricks Platform Build Location: London (2 days a week on site) Contract: 6-month sign-on Interview process: 2 stages twentyAI’s customer is building the next generation of their data platform, with Databricks and Apache Spark at the core. A PoC is already in place — now they’re looking for someone who’s done this before to lead the full-scale build and help roll the platform out across the organisation. The project involves migrating the existing on-prem Data Lake to a Databricks-based architecture, while continuing to leverage private cloud storage. You’ll be laying the foundations: building ETL pipelines in PySpark, implementing platform best practices, and working closely with teams across trading and finance. Profile: • Strong experience delivering Spark/Databricks implementations in a lead or senior role • Solid hands-on background in PySpark, Spark, and Python • Experience setting up data platforms with a DevOps-first approach (IaC, CI/CD, automation) • Exposure to AWS-based environments • Familiarity with financial/trading systems and working in regulated industries (e.g., banking, commodities) Tech Environment: • Primary Platform: Databricks, Apache Spark • Other Tech: DBT, Airflow, Python, PySpark • Cloud: AWS (preferred), private cloud storage • Data Sources: Financial/trading systems