Senior Python Data Engineer

Senior Python Data Engineer Status: Are you legally eligible to work where you live? We are not able to sponsor VISAs. Resume must be in English Location: Remote Engagement: Contract, Long Term, Full Time About the work We process 1B+ tax forms per season. Our current Snowflake-based pipeline needs to scale. We are re-platforming onto a cloud-native architecture using Azure Container Apps, ADLS Gen2, Polars, and Parquet. We have a POC roadmap in flight and need an engineer to own the implementation and hardening of these components into production. What you’ll do • High-Performance Validation: Build validation engines using Polars for batch processing of large Parquet datasets. • State & Metrics: Develop event-sourced job-state layers using Azure cloud-native services. • Observability: Implement end-to-end tracing and monitoring with OpenTelemetry and App Insights. • Query Layer: Create fast read paths using DuckDB and Parquet to serve UI patterns with sub-200ms latency. • Benchmarking: Run performance tests at scale (1B+ rows) and drive architectural decisions through data. • End-to-End Delivery: Own the full lifecycle including IAC development with Bicep and containerized deployments. What you bring (Must-have) • Python Expert: 5+ years building production Python data systems. • Modern Data Stack: Proficient in Polars (or strong Arrow/Pandas with a willingness to switch). • Azure & IAC: Experience with Container Apps and ADLS Gen2. Comfortable with Bicep and working in Devcontainers. • Columnar Formats: Deep understanding of Parquet (partitioning, column pruning, file-size tuning). • Data Patterns: Practical experience with CDC patterns, idempotency, and data consistency. Nice-to-have • Snowflake: Experience with Streams, Dynamic Tables, or Snowpark. • Embedded DBs: Experience with DuckDB for read-heavy projections. • Telemetry: Hands-on OpenTelemetry instrumentation. • Domain: Experience in tax reporting or similar regulated financial pipelines. • Open Formats: Familiarity with Delta Lake or Apache Iceberg. How we work • AI-First & Agentic: We use AI droids for the heavy lifting. Our engineers architect outcomes and manage agents to drive high throughput. • Startup Mode: We move fast and stay proactive. We value pragmatism and high-impact results. We expect engineers to own features end-to-end and prefer the simplest solution that works at scale. • Flexible Environment: We use local emulators for speed and every engineer gets an isolated Azure sandbox via IAC/Bicep for full-cloud validation. You should be comfortable moving between both. • Efficiency Metrics: We track compute cost and token usage as first-class metrics. Efficiency is as important as throughput. • Docs in the Diff: Architecture decisions and technical guides ship in the same PR as the code.