Senior Data Engineer

Role: Senior Data Engineer Location: Reading, PA Term: Contract We are seeking a highly skilled Senior Data Engineer to design, build, and operate scalable, high-performance data platforms. This is a hands-on engineering role requiring deep expertise in PySpark, Python Microservices, and Python programming, along with modern data lake technologies such as Apache Iceberg. The ideal candidate will work closely with data architects and platform leads to implement reliable batch and streaming data pipelines on AWS that support analytics and business-critical applications. Key Responsibilities • Hands-On Data Engineering • Design, develop, and maintain large-scale batch and streaming data pipelines using PySpark and . • Write production-grade Python code for complex data transformations, validations, and business logic. • Implement efficient processing of high-volume, high-velocity data across distributed systems. • Streaming & Real-Time Processing • Build and operate real-time and near real-time data pipelines using . • Implement stateful processing, windowing, checkpointing, and fault-tolerant streaming applications. • Ensure low-latency and high-throughput streaming solutions. • Data Lake & Iceberg • Design and manage data lake architectures using Apache Iceberg on cloud storage (S3). • Implement Iceberg capabilities such as schema evolution, partitioning, compaction, and time travel. • Optimize read and write performance for large Iceberg tables. • Cloud & Data Integration • Design, develop, and deploy data pipelines on AWS using services such as S3, EMR, Glue, Lambda, Athena, Redshift, and Aurora (MySQL/PostgreSQL) for data ingestion, processing, and analytics. • Performance, Reliability & Operations • Tune Spark jobs for performance, scalability, and cost efficiency. • Troubleshoot and resolve complex production issues in distributed data systems. • Implement monitoring, alerting, logging, and recovery strategies for data pipelines. • Engineering Excellence & Collaboration • Write clean, testable, and maintainable code following engineering best practices. • Contribute to CI/CD pipelines for data engineering workloads. • Participate in code reviews, technical design discussions, and architecture reviews. Required Qualifications • Bachelor’s or Master’s degree in Computer Science, Engineering, or a related field. • Strong expertise in PySpark and Microservices and Spark SQL for large-scale data processing • Strong expertise in , including streaming fundamentals and stateful processing. • Hands-on experience building and running Kafka-based streaming applications in production environments. • Advanced proficiency in Python for building scalable, production-grade data solutions. • Hands-on experience with Apache Iceberg in production environments. • Solid experience with AWS data services (S3, EMR, Glue, Lambda, Redshift). • Advanced SQL skills and strong understanding of data modeling and data lake architectures.