Senior Data Engineer

We are a technology solutions provider that specialises in serving the public sector. Our mission is to help government organisations become data-driven, transform their digital offerings, streamline their processes, improve citizen services and enhance transparency. We are seeking a highly skilled and experienced Senior Data Engineer to join our team working with our Central Government client on a contract basis. This role requires a deep understanding of data engineering best practices, strong hands-on experience with AWS, Azure, Apache Spark, data warehousing, database modelling and SQL. You’ll play a critical role in designing, building, and maintaining our data infrastructure to support scalable, high-performance data pipelines and analytics platforms. Responsibilities will include: • Design, build, and maintain robust, scalable, and secure data pipelines using AWS services and Apache Spark. • Develop and optimize data models for reporting and analytics in Redshift and other DWH platforms. • Collaborate with Data Scientists, Analysts, and Business Stakeholders to understand data requirements and deliver clean, validated datasets. • Monitor, troubleshoot, and optimize ETL/ELT workflows to ensure data quality and pipeline efficiency. • Implement best practices in data governance, security, and compliance within cloud environments. • Lead and mentor junior data engineers, promoting a culture of technical excellence and continuous improvement. • Write clean, maintainable, and efficient code while following best practices in software development. • Debug and resolve issues and implement solutions to improve the performance and functionality of existing applications. • Stay up to date with the latest in cloud and big data technologies, evaluating their potential for use in the platform. • Collaborate with DevOps and Platform teams to deploy and maintain data pipelines and services. • Stay up-to-date with industry trends, best practices, and emerging technologies to enhance the development process and maintain high coding standards. • Document code, systems, and processes to facilitate knowledge sharing and maintainability. Requirements • Proven work experience as a Senior Data Engineer using cloud platform technologies, alongside experience with a variety of database technologies including Oracle, Postgres and MSSQLServer; • Strong expertise in AWS services including AWS DMS, S3, Lambda, Glue, EMR, Redshift, and IAM. • Proficient in Apache Spark (batch and/or streaming) and big data processing. • Solid experience with SQL and performance tuning in data warehouse environments. • Hands-on experience with Amazon Redshift or equivalent, including table design, workload management, and implementing Redshift Spectrum. • Experience building ETL/ELT pipelines using tools like AWS Glue, EMR, or custom frameworks. • Familiarity with data modeling concepts. • Excellent problem-solving and communication skills. • Proficiency in Java and data pipeline development. • Familiarity with version control systems (e.g., Git) and agile development methodologies. • Experience working with Public Sector clients • Consulting experience Preferred Knowledge/Experience • Experience with CI/CD and Infrastructure-as-Code (e.g., Terraform, CloudFormation). • Familiarity with modern data stack components (e.g., dbt, Snowflake, Airbyte). • Experience working in an Agile/Scrum environment. • Knowledge of Python or Java/Scala for data engineering. • Experience with version control systems (eg Git, CVS). Applicants will be required to obtain SC and therefore 5 year's UK address history is essential. IR35 status: Outside Start: ASAP Duration: 3 months (initially) Day rate: negotiable