Data Engineer (Python, PySpark, Databricks)

Data Engineer (Python / PySpark / Databricks) Long-Term Contract Atlanta, GA or Remote We are seeking a hands-on Data Engineer with strong Python and PySpark development experience in a Databricks environment. This role is ideal for someone with 5+ years of experience designing, developing, and supporting production-grade data pipelines in modern cloud environments. The ideal candidate has extensive experience writing code-based data pipelines and is comfortable working across the full data engineering lifecycle—from data ingestion and transformation through orchestration, deployment, monitoring, and ongoing optimization. We're looking for an engineer who has contributed to large-scale data platforms supporting hundreds of production pipelines and understands the engineering practices required to build reliable, scalable data solutions. Responsibilities Design, develop, and maintain scalable data pipelines using Python and PySpark. Build and optimize batch and streaming data pipelines in a Databricks environment. Develop reusable, production-ready code to ingest, transform, and load large-scale datasets. Implement data quality validation, error handling, monitoring, and alerting to ensure reliable data processing. Optimize Spark jobs for performance, scalability, and cost efficiency. Collaborate with data architects, analytics teams, and business stakeholders to deliver high-quality data solutions. Participate in code reviews and contribute to engineering best practices. Deploy and support data pipelines through CI/CD processes and automated release pipelines. Troubleshoot and resolve production issues while continuously improving pipeline reliability and performance. Contribute to the design and evolution of modern cloud-based data platforms. Required Qualifications 5+ years of professional Data Engineering experience. Strong hands-on experience developing data pipelines using Python and PySpark. Experience working with Databricks. Experience building and supporting production-grade data pipelines processing large datasets. Strong understanding of distributed data processing concepts and Spark optimization techniques. Experience implementing data quality, validation, monitoring, and logging practices. Experience with orchestration tools such as Databricks Workflows, Apache Airflow, or similar. Experience working with Git and modern CI/CD deployment practices. Strong SQL skills and experience developing efficient data transformations. Experience with cloud platforms such as AWS, Azure, or Google Cloud Platform. Excellent troubleshooting, analytical, and problem-solving skills. Experience with Delta Lake and the Databricks Lakehouse Platform. Experience building metadata-driven or reusable pipeline frameworks. Knowledge of streaming technologies such as Apache Kafka or Spark Structured Streaming. Familiarity with modern data architecture patterns, including Medallion Architecture.