Senior Data Engineer - Databricks

Senior Data Engineer Contractor US Remote | 3-Month Initial Contract | Senior / Staff Level | Energy-Focused Investment Firm ONLY APPLICATIONS VIA LINKEDIN WILL BE CONSIDERED. ALL DIRECT EMAILS WILL NOT BE READ. Tech Stack: Databrick, Pyspark, Azure, SQL Dataworks is supporting a US-based investment platform focused on energy assets in the search for a Senior Data Engineer Contractor to stabilize, harden, and evolve an existing production data platform. This is a high-impact engagement within a lean, senior team. You will not be building from scratch. You will take ownership of reliability, quality, and production discipline across an Azure-hosted data environment powering investor-grade analytics. About the Environment The client operates a cloud-based data platform running on Azure with multiple production pipelines already in place. The objective is to convert complex third-party operational datasets into reliable, versioned time-series and network-style datasets used for analytics and decision support. Primary consumers are business stakeholders and investment teams. Data accuracy, immutability, and traceability are critical. There is no real-time streaming stack. The focus is production stability, deterministic logic, and high data quality. The Role You will own the engineering backbone of the platform. Key responsibilities include: • Productionizing ingestion from external vendors into structured, reliable datasets • Implementing incremental processing, safe backfills, and reproducible historical replays • Building and maintaining curated “model-ready” datasets with stable keys • Designing and enforcing data quality gates that prevent bad publishes • Ensuring deterministic aggregations and preserving time-series directionality • Establishing run discipline: manifests, run IDs, lineage, rerunnable workflows • Improving orchestration, monitoring, retries, and alerting • Partnering with analytics stakeholders to define stable data contracts This is a senior-level role requiring autonomy. You may be the sole engineer on the project. What “Good” Looks Like Within 30–60 days: • Daily pipeline runs are stable with clear manifests and failure alerts 2026 02 Senior data engineer Co… • Data quality gates catch regressions before publishing • Historical replay and backfills are routine and low risk • Performance and cost are optimized as volumes scale By 90 days: • The pipeline supports modeling and scenario analysis without upstream churn 2026 02 Senior data engineer Co… • An operational playbook exists Ideal Profile We are looking for someone who: • Has built and maintained production-grade data pipelines in the cloud • Understands incremental processing, schema evolution, partitioning, and optimization • Has implemented data quality frameworks and publish gates • Is comfortable being highly autonomous • Has experience in energy, commodities, logistics, or flow-based systems (strong plus) • Can operate independently while participating in stand-ups and team alignment East Coast US timezone overlap preferred. Fully remote. Engagement structure: 1099, C2C, or W-2 depending on compliance and setup. Initial contract: 3 months with strong likelihood of extension.