Senior Data Engineer

Job Title:- Senior Data Engineer Location:- Houston, Texas (On-Site/Hybrid) Job Type:- Long Term Contract Must be Local to Houston, TX Only GC, USC, H4EAD Candidates on W2 or 1099 will be entertained Overview: Delivers the Palantir Foundry exit on a modern Snowflake stack by building reliable, performant, and testable ELT pipelines; recreates Foundry transformations and rule-based event logic; and ensures historical data extraction, reconciliation, and cutover readiness. Years of Experience: 7+ years overall; 3+ years hands-on with Snowflake. Key Responsibilities:Extract historical datasets from Palantir (dataset export, parquet) to S3/ADLS and load into Snowflake; implement checksum and reconciliation controls. Rebuild Foundry transformations as dbt models and/or Snowflake SQL; implement curated schemas and incremental patterns using Streams and Tasks. Implement the batch event/rules engine that evaluates time-series plus reference data on a schedule (e.g., 30–60 minutes) and produces auditable event tables. Configure orchestration in Airflow running on AKS and, where appropriate, Snowflake Tasks; monitor, alert, and document operational runbooks. Optimize warehouses, queries, clustering, and caching; manage cost with Resource Monitors and usage telemetry. Author automated tests (dbt tests, Great Expectations or equivalent), validate parity versus legacy outputs, and support UAT and cutover. Collaborate with BI/analytics teams (Sigma, Power BI) on dataset contracts, performance, and security requirements. Required Qualifications:Strong Snowflake SQL and Python for ELT, utilities, and data validation. Production experience with dbt (models, tests, macros, documentation, lineage). Orchestration with Airflow (preferably on AKS/Kubernetes) and use of Snowflake Tasks/Streams for incrementals. Proficiency with cloud object storage (S3/ADLS), file formats (Parquet/CSV), and bulk/incremental load patterns (Snowpipe, External Tables). Version control and CI/CD with GitHub/GitLab; environment promotion and release hygiene. Data quality and reconciliation fundamentals, including checksums, row/aggregate parity, and schema integrity tests. Performance and cost tuning using query profiles, micro-partitioning behavior, and warehouse sizing policies. Preferred Qualifications:Experience migrating from legacy platforms (Palantir Foundry, Cloudera/Hive/Spark) and familiarity with Trino/Starburst federation patterns. Time-series data handling and rules/pattern detection; exposure to Snowpark or UDFs for complex transforms. Familiarity with consumption patterns in Sigma and Power BI (Import, DirectQuery, composite models, RLS/OLS considerations). Security and governance in Snowflake (RBAC, masking, row/column policies), tagging, and cost allocation. Exposure to containerized workloads on AKS, lightweight apps for surfacing data (e.g., Streamlit), and basic observability practices.