Senior Data Engineer

Senior Data Engineer ~ £700pd (Inside IR35) ~ Python We are looking for a Senior Data Engineer with strong technical experience in Python, Databricks, SQL and PySpark alongside strong overall data engineering experience, this is an exciting opportunity to join the technology team of a global leader in the energy and commodities trading sector! Key Responsibilities: • Design, develop and maintain scalable, high-performance data pipelines using Databricks on AWS • Build and optimise Delta Lake architectures to ensure reliable, secure and efficient data storage for trading and analytics • Develop and enhance the Front Office data platform, enabling traders and business users to access high-quality data for reporting, analytics and decision-making • Work closely with Data Scientists, Quantitative Analysts and Machine Learning Engineers to prepare data for AI and machine learning models • Implement and maintain CI/CD processes, automated testing and monitoring to improve the reliability and quality of data engineering solutions • Support MLOps processes including model deployment, lifecycle management and monitoring using MLflow Required Skills: • 8+ years' experience delivering enterprise-scale data engineering solutions • Expert-level knowledge in: Apache Spark, PySpark, and Databricks—including experience with Delta Lake, Unity Catalog, MLflow, and Databricks Workflows. • Advanced Python and SQL development skills • Strong experience with AWS cloud services (S3, Lambda, Glue, API Gateway, IAM, EC2, EKS) • Integration of AWS-native components with Databricks • Strong understanding of CI/CD, DevOps and Infrastructure as Code • Experience implementing robust DevOps practices for data engineering • Experience in commodity trading markets is beneficial! The Role: • Senior Data Engineer • Hybrid (3 days in office, 2 from home) • London • £700pd (Inside IR35) If you think you'd be a good fit, don't hesitate to apply with your latest CV attached for immediate consideration! Ella.Lyons@rjcgroup.co.uk