Principal Data Engineer

Principal Data Engineer Analytics and Data Engineering Bristol Up to £72,000 12 months fixed term contract About us We are the independent regulator of higher education in England. We aim to ensure that every student, whatever their background, has a fulfilling experience of higher education that enriches their lives and careers. We’re proud of our people, our culture and the benefits we offer. We care about supporting and developing our staff. At the OfS we want to create a welcoming and engaging working environment which is free from discrimination. We’re looking for skilled, committed people who care about students and their experience of higher education. Join us and you’ll be part of an energetic, forward-looking organisation with high ambitions and big opportunities for our people. The analytics & data engineering team are a newly formed team in the OfS. The team is there to make sure the analyst team and subsequently the business has the right information to carry out the functions of the organisation. Why work for the OfS The OfS's Data function is currently undergoing an exciting and major transformation and we're looking for a talented data engineer to help lead the change. We're moving away from a legacy on-premises platform to an Azure Databricks platform, unlocking new possibilities for automation and innovation. This is your opportunity to help develop new robust data products from the ground up, set best practice and introduce AI into the organisation to improve how we support the higher education sector and the needs of students. Key responsibilities You will be a key individual contributor and be responsible for: • Building and developing re-useable pipelines for analytics and AI projects • Pushing for innovation within the platform to enable great efficiencies and detailed insights and outputs • Leading key relationships between IT and Data to grow the platform and release new capabilities • Deploying production AI models with automated monitoring from the data pipeline to the model runs and outputs As well as supporting the team responsibilities on: • Work to Extract, Load & Transform (ELT) data sets from a variety of data sources across the OfS enterprise technology stack. With a particular focus to the Extract & Load parts. • Monitoring the execution of data workflows, including identifying and mitigating risks, setting service level indicators and configuring alerts. • Adopting data governance best practice when processing raw data to develop, test and maintain datasets, including the use and maintenance of relevant resources (such as, but not limited to, a data catalogue, data dictionary, logical data models). • Developing coding standards for Python programming to be used across the Data function. • Supporting continuous data quality improvements across the organisation by exploring enhancements to: automated quality assurance processes; the reliability of data workflows; and the effectiveness of data transformations. • Applying dimensional data modelling concepts and practices across the OfS data to develop conceptual, logical and physical data models that support efficiency and effectiveness of business operations • Advocating and supporting an evidence-led culture across the organisation, assisting with driving enhancements to the organisation’s data capability Essential experience & skills: • Strong Python programming knowledge, ideally Pyspark • Strong knowledge of the Azure Databricks platform and associated functionalities • Adaptable, with a willingness to work flexibly as the needs of the organisation evolve. • Working well within a team, and able to work closely with internal and external stakeholders. • An ability to take a logical and analytical approach, and to take a pragmatic, collaborative approach to solving problems. • Adept at communicating technical concepts to a nontechnical audience. • Awareness of the modern data stack and associated methodologies Working for us The OfS regulates the higher education sector on behalf of all students. We value diversity and the wealth of perspectives, experience and ideas that it brings to our work, and we strive to embed equality of opportunity in everything we do. We recruit based on fair and open competition and welcome applications from candidates regardless of age, disability, race and ethnicity, gender reassignment, marriage and civil partnership, pregnancy and maternity, religion or belief, sex or sexual orientation. Flexibility is important to the OfS, as we recognise that it can enable people to work more effectively. A set of organisation-wide principles shape our approach to flexible and hybrid working, which our teams then use as guidance on how they deliver their business goals. This would usually mean working from the office for two days in a typical week. To attract the widest possible field of qualified applicants, flexible working, for example job share and part time working apply to this role. Closing date for applications: 8th September 2025 Interviews: From 15 September 2025