Senior Data Engineer

Employment Type: Contract Contract Length: 6 months (extendable) Location: London, United Kingdom Work Model: Hybrid About the Data Platform: • The Data Platform will be built and managed “as a Product” to support a Data Mesh organization. • The Data Platform focusses on enabling decentralized management, processing, analysis and delivery of data, while enforcing corporate wide federated governance on data, and project environments across business domains. • The goal is to empower multiple teams to create and manage high integrity data and data products that are analytics and AI ready, and consumed internally and externally. What does a Data Infrastructure Engineer do? • A Data Infrastructure Engineer will be responsible to develop, maintain and monitor the data platform infrastructure and operations. The infrastructure and pipelines you build will support data processing, data analytics, data science and data management across the business. • The data platform infrastructure will conform to a zero trust, least privilege architecture, with a strict adherence to data and infrastructure governance and control in a multi-account, multi-region AWS environment. • You will use Infrastructure as Code and CI/CD to continuously improve, evolve and repair the platform. • You will be able to design architectures and create re-useable solutions to reflect the business needs. Responsibilities will include: • Collaborating across departments to develop and maintain the data platform • Building infrastructure and data architectures in Cloud Formation, and SAM. • Designing and implementing data processing environments and integrations using AWS PaaS such as Glue, EMR, Sagemaker, Redshift, Aurora and Snowflake • Building data processing and analytics pipelines as code, using python, SQL, PySpark, spark, CloudFormation, lambda, step functions, Apache Airflow • Monitoring and reporting on the data platform performance, usage and security • Designing and applying security and access control architectures to secure sensitive data You will have: • 6+ years of experience in a Data Engineering role. • Strong experience and knowledge of data architectures implemented in AWS using native AWS services such as S3, DataZone, Glue, EMR, Sagemaker, Aurora and Redshift. • Experience administrating databases and data platforms • Strong proficiency in Cloud Formation, Python and SQL • Knowledge and experience of relational databases such as Postgres, Redshift • Experience using Git for code versioning, and lifecycle management • Experience operating to Agile principles and ceremonies • Hands-on experience with CI/CD tools such as GitLab • A keen eye for detail, and a passion for accuracy and correctness in numbers Whilst not essential, the following skills would also be useful: • Experience using Jira, or other agile project management and issue tracking software • Experience with Snowflake • Experience with Spatial Data Processing