Lead Data Engineer (Lead II - Data Engineering)

Role Description Role: Lead Data Engineer Fixed term contract Contract Length: Initial 3-6 months with possible extensions Start Date: ASAP Experience Range - 10- 12 Years Location Requirement: Onsite (3 days per week in the office and 2 days remote ) Applicants must be legally authorized to work in the United Kingdom without the need for current or future visa sponsorship Responsibilities • Design, build and optimise robust data ingestion pipelines to acquire, transform and load data vendor platforms into client's data ecosystem. • Develop scalable data engineering solutions using Databricks, PySpark, SparkSQL and associated cloud technologies. • Build and maintain secure, reliable and automated data ingestion processes from external vendor systems, including SFTP-based file transfers and other integration methods. • Ensure data is landed, structured, governed and accessible to support reporting, analytics and business use cases. • Work with Business Analysts, Product Owners, Architects and delivery squads to translate business requirements into technical data solutions. • Support data discovery, profiling and validation activities to understand source data structures, data quality issues and data gaps. • Develop and maintain data transformations, curated datasets and data models required to support reporting and analytical use cases. • Monitor, troubleshoot and resolve data ingestion issues, defects and enhancements identified during development, testing, UAT and production support. • Ensure solutions comply with data architecture standards, engineering best practices, security requirements and governance frameworks. • Produce clear technical documentation for data ingestion processes, data flows and operational support requirements. • Provide comprehensive handover documentation and knowledge transfer to the Data Support team following delivery of data ingestion pipelines. • Collaborate within Agile delivery teams, actively contributing to sprint planning, stand-ups, retrospectives and continuous improvement activities. • Identify opportunities to improve pipeline performance, automation, scalability and maintainability through process and technology enhancements. • Support knowledge sharing and contribute to Engineering and Data Communities of Practice. Essential Skills & Experience • Proven experience designing, building and supporting enterprise-scale data ingestion pipelines and ETL/ELT solutions. • Strong hands-on experience with Databricks, PySpark and SparkSQL. • Experience developing and supporting secure data integrations using SFTP and other file-based or API-driven ingestion mechanisms. • Experience ingesting and processing structured, semi-structured and unstructured data from internal and third-party source systems. • Strong understanding of data modelling, transformation techniques and data warehousing principles. • Experience working with cloud-based data lake and analytics platforms. • Strong understanding of batch and near real-time data processing patterns. • Experience conducting data profiling, discovery and validation activities to assess data quality, completeness and suitability for business requirements. • Experience implementing data quality checks, reconciliations and monitoring processes. • Ability to investigate and resolve ingestion, transformation and data quality issues identified during testing, UAT or production support. • Understanding of data governance, security, data lineage and documentation standards. • Experience producing technical documentation and operational handover materials. • Strong stakeholder engagement skills with the ability to work effectively across business, architecture, engineering and analytics teams. • Experience working within Agile delivery environments. • Knowledge of source control, CI/CD practices and release management processes. • Ability to work independently while collaborating effectively within cross-functional squads. Desirable • Experience integrating data from retail technology platforms, IoT devices or third-party vendor systems. • Experience working with AI Camera, Computer Vision or Electronic Shelf Edge Label (eSEL) technologies. • Knowledge of Azure Data Lake, Azure Data Factory and related Azure data services. • Experience supporting reporting, analytics or BI solutions through the creation of trusted and governed data assets. • Experience working within large-scale retail or data transformation programmes. If interested, please apply with your updated CV for an immediate discussion #UST Skills PySpark, Azure Data Factory, Agile, CI/CD