Azure Data Engineer

We are currently looking for an experienced Azure Data Engineer to support the design, development and operation of a large-scale Azure data platform. You will be responsible for building reliable data pipelines, improving platform scalability and supporting real-time and batch data processing across enterprise datasets. Contract Information • Job title: Azure Data Engineer • Location: London • Contract Length: 6 Months • Working Arrangement: Hybrid, 2 days onsite per week • Engagement: Inside IR35 Responsibilities • Design, build and maintain large-scale data pipelines and ETL workflows across enterprise datasets. • Ingest, provision and manage raw, enriched and curated data assets for analytics platforms. • Improve the reliability, scalability and performance of real-time data ingestion capabilities. • Support and enhance cloud-based data ingestion infrastructure and platform services. • Acquire, transport, cleanse, transform and integrate data from internal and external sources. • Conduct exploratory data analysis and support the onboarding of new data sources. • Develop automated data engineering workflows and optimise data processing pipelines. • Manage data modelling, data warehousing, data integration and data cataloguing activities. • Support real-time analytics and streaming data platforms. • Contribute to cloud infrastructure engineering, platform operations and production support. • Implement cloud cost-optimisation initiatives and improve overall platform efficiency. • Collaborate with engineering, analytics, infrastructure and cybersecurity teams. • Support DevOps automation, monitoring and operational excellence initiatives. • Maintain technical documentation, governance standards and operational procedures. Skills and Experience • Experience applying Site Reliability Engineering principles within Azure environments. • Strong experience with Azure DevOps. • Proficiency in Bash, PowerShell and Azure CLI scripting. • Programming experience with Python, C and Java, including object-oriented development. • Strong SQL and Kusto Query Language experience. • Experience with GitHub, source-control systems and CI/CD pipelines. • Strong Linux administration and scripting skills. • Knowledge of networking, operating systems, storage technologies and core infrastructure principles. • Experience with data acquisition and cloud-based data pipelines. • Experience transporting, cleansing, transforming and integrating data. • Strong data engineering skills, including pipeline automation, productionisation and performance optimisation. • Experience designing and maintaining ETL workflows and large-scale data pipelines. • Knowledge of data warehousing, data modelling and data cataloguing. • Experience supporting real-time analytics and streaming data platforms. • Experience implementing cloud cost-optimisation initiatives. • Strong knowledge of the Microsoft Azure platform, including Microsoft Entra ID, identity and access management, networking, compute, storage, databases and containers. • Hands-on experience with Azure Data Factory, Azure Databricks, Unity Catalog, Azure Functions, Azure Logic Apps, Azure Monitor, Azure Log Analytics, Azure Data Lake Storage, Azure Synapse Analytics and Power BI. • Experience with Kubernetes and cloud-native platforms. • Knowledge of NGINX, Apache, Azure Cosmos DB, Linux, Prometheus, Grafana and Elasticsearch. • Experience with Infrastructure as Code tools such as Terraform, ARM templates, Chef and Ansible. • Experience with Azure Event Hubs, Apache Kafka and Spark Streaming. • Exposure to SIEM and SOAR platforms would be advantageous. • Experience working within highly regulated enterprise environments is preferred.