Senior Data Engineer

Senior Data Engineer Location: Northern Virginia (Remote) Status: USC or GC required Responsibilities include: • Design, develop, and implement robust ETL solutions using Python and PySpark to extract, transform, and load data from various sources into AWS data services. • Optimize ETL processes for performance and scalability utilizing AWS Glue, EMR, Step Functions, and Lambda to ensure efficient data processing and timely delivery. • Ensure data integrity and quality throughout the ETL process by implementing thorough data validation checks and error handling mechanisms. • Manage AWS services such as Glue, EMR, Step Functions, and Lambda, including configuration, monitoring, and troubleshooting to maintain operational excellence. • Collaborate with cross-functional teams including data engineers, data scientists, and business stakeholders to understand data requirements and deliver tailored ETL solutions. • Troubleshoot complex technical issues and provide advanced operational support to internal MITRE customers in AWS Required Qualifications: • Bachelor’s degree with eight years’ related experience, or a master’s degree with six years’ related experience, preferably with a technical major such as engineering, computer science, etc. • Self-motivated, curious, and collaborative, with a passion to learn new technologies and develop new skills • Demonstrated experience in developing ETL pipelines using Python and PySpark, with a strong understanding of data processing techniques. • Expertise in SQL for data manipulation, querying, and optimization to work with various database platforms including Postgres, DynamoDB, Oracle, and Redshift. • Hands-on experience with AWS Glue, EMR, Step Functions, and Lambda for building and orchestrating ETL workflows in a cloud environment. • Experience implementing Continuous Integration/Continuous Deployment (CI/CD) pipelines using AWS CDK or similar tools for automating deployment and testing of ETL solutions. Preferred Qualifications: • Previous experience leading or mentoring a team of developers/engineers in a collaborative environment. • AWS certifications such as AWS Certified Developer or AWS Certified Solutions Architect, demonstrating proficiency in AWS services and best practices. • Familiarity with big data technologies such as Apache Spark, Hadoop, or Kafka for processing large-scale datasets. • Experience in data modeling and schema design for optimizing database performance and scalability. • Experience working in Agile development methodologies, such as Scrum or Kanban, for iterative and collaborative project delivery.