Senior Data Engineer – Databricks / PySpark (AI Platform)

Senior Data Engineer – Databricks / PySpark (AI Platform) Urbandale, IA 50322Duration: 7+ monthsShift: 1st ShiftTarget Start Date: April 1, 2026 Overview We are seeking a highly technical and self-directed Senior Data Engineer to support the development of data processing pipelines for an AI-enabled analytics product serving large agricultural customers. This role is ideal for a mid-career professional with proven experience deploying cloud-native solutions in fast-paced software delivery environments. The engineering team is distributed across Santa Clara, CA and the Des Moines metro area. Key Responsibilities Author and optimize PySpark Databricks ETL and streaming jobs to ensure scalable, reliable data workflows Design and implement Databricks-native solutions including Delta Live Tables, Structured Streaming, and Vector Search Build and maintain CI/CD pipelines using GitHub Actions with a focus on code quality and incremental delivery Contribute infrastructure-as-code using Terraform Debug and resolve data issues to support field testing and customer operations Collaborate closely with data scientists to productionize prototypes and proof-of-concept models Work effectively within a distributed engineering team Required Skills & Experience 5+ years of professional software development experience using Python 3+ years of hands-on Databricks and PySpark experience in production environments Strong experience optimizing PySpark queries and working with Delta Lake Hands-on experience with Structured Streaming and Delta Live Tables Demonstrated proficiency using GitHub (pull requests, code reviews, branching workflows) Experience building CI/CD pipelines (GitHub Actions preferred) Working knowledge of AWS services including S3, Lambda, EC2, and IAM Strong communication skills with the ability to clearly articulate technical concepts Ability to work independently with limited supervision Preferred Experience Experience with event-driven architectures (Kafka, Kinesis, etc.) Experience working in cross-functional teams with product and data science Experience with geospatial data and related libraries Job Disclaimer Equal Opportunity Employer/Veterans/DisabledBenefit offerings include medical, dental, vision, term life insurance, short-term disability insurance, additional voluntary benefits, commuter benefits and 401K plan. Our program provides employees the flexibility to choose the type of coverage that meets their individual needs. Available paid leave may include Paid Sick Leave, where required by law; any other paid leave required by Federal, State or local law; and Holiday pay upon meeting eligibility criteria.Disclaimer: These benefit offerings do not apply to client-recruited jobs and jobs which are direct hire to a clientTo read our Candidate Privacy Information Statement, which explains how we will use your information, please visit https://www.entegee.com/candidate-privacy-information-statement/ #GRPA 1619250 Pay: $60.00 - $65.00 per hour People with a criminal record are encouraged to apply Application Question(s): Do you have 5+ years of professional experience developing production-grade software in Python? Do you have at least 3 years of hands-on Databricks experience in a production environment? Have you worked with Delta Live Tables or Structured Streaming in production? Have you built and optimized PySpark ETL or streaming jobs in Databricks? Have you deployed workloads in AWS using S3, Lambda, or EC2? Have you built or maintained CI/CD pipelines (ideally using GitHub Actions)? Have you used Terraform for infrastructure-as-code? Work Location: In person