Senior Platform Data Engineer

Senior Platform Data Engineer We are seeking a Senior Platform Data Engineer to support a high-visibility, year-long platform transformation initiative. This is a production engineering role focused on building and operating a centralized, enterprise-grade data platform that institutionalizes proprietary knowledge and delivers data as a product — not a side effect. You will design, build, and operate models, pipelines, integrations, and infrastructure across the full medallion architecture (Bronze → Silver → Gold). This role requires true end-to-end ownership: design, build, test, deploy, monitor, and continuously improve. What You’ll Do • Build and maintain ingestion pipelines from raw Bronze landing through curated Gold domain models • Develop transformation models using dbt with testing, Slim CI, and exposure tracking • Author and manage production-grade Airflow DAGs with SLA monitoring and backfill strategies • Design and implement streaming and event-driven pipelines (Kafka, Glue Streaming, Delta Live Tables) • Collaborate with data scientists, platform engineers, and product teams to ship reliable data products • Implement data quality frameworks and enforce governance standards • Containerize workloads and deploy to Kubernetes (EKS) • Contribute to infrastructure as code (Terraform provisioning for S3, IAM, Glue, etc.) • Document lineage, metadata, ownership, and sensitivity classification This is not a “pipeline-only” role — it is platform ownership at scale. Core Technical Skills Python • Production-level standards (type hints, Pydantic, pytest, Ruff, mypy) SQL • Advanced analytical queries, window functions, optimization • Experience with Redshift, Snowflake, or Databricks SQL dbt • Gold-layer modeling and testing • Slim CI, exposures, meta configurations Orchestration • Apache Airflow (DAG authoring, task dependencies, SLAs, backfills) • Prefect or Dagster is a plus Streaming & Event Pipelines • Kafka topic consumption • Bronze landing jobs Cloud Data Platforms (hands-on with at least one) • Databricks (Delta Lake, Unity Catalog, Auto Loader, DLT) • AWS (S3, Glue, Lake Formation, Redshift) Data Quality • Great Expectations, dbt tests, or equivalent Docker & Kubernetes • Containerized jobs • Kubernetes Jobs and CronJobs • EKS deployments Infrastructure as Code • Terraform familiarity Data Modeling • Dimensional modeling • Slowly Changing Dimensions • Medallion architecture design • Financial or time-series modeling experience preferred Data Platform-Specific Experience • SaaS ingestion via Fivetran, Airbyte, or custom connectors (REST, SFTP, JDBC) • Change Data Capture (Debezium or connector-based CDC modes) • Handling semi-structured and unstructured data (PDF, Excel, JSON) • Designing Bronze schemas for fidelity and downstream flexibility • Vector embedding pipelines (chunking strategies, embeddings, pgvector/OpenSearch) • Data lineage tooling (OpenLineage, DataHub, Unity Catalog) • Governance standards: PII handling, sensitivity classification, column masking, partition-level access Ways of Working • Agile sprint participation (planning, standups, retros) • Treat pipelines as production software (tests, runbooks, SLAs, alerts) • Migration-safe schema changes (expand/contract pattern) • Proactive data quality ownership • Rigorous peer code reviews • Close collaboration with data science and platform engineering teams Nice to Have • Migration of legacy data assets (Excel, Access, shared drives) • ML feature engineering and point-in-time correct pipelines • Exposure to financial reporting, chart of accounts normalization, or diligence data • Experience in private equity, investment banking, asset management, or professional services • Open-source data tooling contributions Qualifications • Bachelor’s degree or equivalent experience • 5–8+ years of experience developing data products and platforms • Strong background in Agile software development • Demonstrated success in prior engineering roles • Strong analytical and problem-solving capabilities What We’re Looking For • Clear communicator who can document complex data flows • Results-driven and impact-focused • Proactive and self-directed • Comfortable interfacing across technical and business stakeholders • Entrepreneurial mindset with a willingness to innovate If you’re a production-minded data engineer who thrives on building scalable platforms and owning pipelines end-to-end, we’d love to connect.