Data Engineer

Must be willing to work with overlap to European hours. Ideally 6:00am-3pm EST We are looking for a Data Engineer to join a dedicated team building and evolving a clinical data platform serving the clinical operations space. You will architect and build the large-scale data pipelines that power clinical insights — processing billions of records across medical claims, clinical trials, publications, and provider data. This is a core infrastructure role. You will be responsible for designing, building, and maintaining ETL frameworks that feed into analytics, machine learning, and product surfaces. You should be deeply comfortable with distributed computing at scale and experienced working alongside ML and data science teams in production environments. What You'll Do ● Design, build, and maintain large-scale ETL pipelines and data frameworks using Apache Spark (PySpark/Scala) on cloud infrastructure ● Architect scalable data models and pipeline patterns to process structured and unstructured healthcare data at volume ● Build and optimize data layers on Azure cloud services, including Databricks, Delta Lake, and supporting compute and storage infrastructure ● Ensure data quality, lineage, and governance across the platform — implementing validation, monitoring, and alerting at scale ● Collaborate with AI Scientists and MLOps teams to build data pipelines that serve model training, inference, and retraining workflows ● Work with data analysts and product teams to ensure curated, reliable data is available for downstream insights and reporting ● Contribute to platform architecture decisions and help define best practices for data engineering within the team What We're Looking For ● 5+ years of experience in data engineering with a focus on large-scale distributed data systems ● Strong proficiency in Python, SQL, and Scala ● Deep hands-on experience with Apache Spark (PySpark, Spark SQL) for building ETL pipelines and data transformations at scale ● Experience with Azure cloud services — including Databricks, Delta Lake, and Azure Data Factory ● Familiarity with Kubernetes and container orchestration for data workloads ● Understanding of MLOps practices and experience building data infrastructure that supports machine learning workflows ● Experience with data quality frameworks, data lineage, and governance tooling ● Background in healthcare, life sciences, pharma, or clinical research is a strong plus ● Comfortable working independently in a remote setting with a distributed, cross-time zone team Who You Are ● A builder who thinks in systems — you design for scale, reliability, and maintainability from the start ● Someone who understands how data engineering connects to ML and analytics, and proactively bridges those gaps ● Confident owning pipeline architecture end-to-end, from ingestion through transformation to serving layers ● Pragmatic and communicative — you flag trade-offs early and keep teams aligned on data dependencies ● Experienced collaborating across time zones with distributed teams including data science, MLOps, and product stakeholder