AWS Data Engineer

Role AWS Data Engineer Experience 8+ years Location Remote Time Zone UK Duration 2 months (Extendable) Job Description • Design, development, and implementation of performant ETL pipelines using python API (pySpark) of Apache Spark on AWS EMR. • Writing reusable, testable, and efficient code • Integration of data storage solutions in spark – especially with AWS S3 object storage. Performance tuning of pySpark scripts. • Need to ensure overall build delivery quality is good and on-time delivery is done at all times. • Should be able to handle meetings with customers with ease. • Need to have excellent communication skills to interact with the customer. • Be a team player and willing to work in an onsite-offshore model, mentor other folks in the team (onsite as well as offshore) • 5+ years of experience in programming with python. Strong proficiency in python • Familiarity with functional programming concepts • 3+ years of hands-on experience in developing ETL data pipelines using pySpark on AWS EMR • Experience in building pipelines and data lake for large enterprises on AWS • Good understanding of Spark’s Dataframe and API • Experience in configuring EMR clusters on AWS • Experience in dealing with AWS S3 object storage from Spark. • Experience in troubleshooting spark jobs. Knowledge of monitoring spark jobs using Spark UI • Performance tuning of Spark jobs. • Understanding fundamental design principles behind business processes Process Knowledge and Expertise: • Demonstrated experience in change management processes, including understanding of governance frameworks and preparation of supporting artefacts required for approvals. • Strong clarity on the path to production, with hands-on involvement in deployments, testing cycles, and obtaining business sign-offs. • Proven track record in technical solution design, with the ability to provide architectural guidance and support implementation strategies. Databricks-Specific Skills: • Experience in at least developing and delivering end-to-end Proof of Concept (POC) solutions covering the below: • Basic proficiency in Databricks, including creating jobs and configuring clusters. • Exposure to connecting external data sources (e.g., Amazon S3) to Databricks. • Understanding of Unity Catalog and its role in data governance. • Familiarity with notebook orchestration and implementing modular code structures to enhance scalability and maintainability. Important Pointers: • Candidates must have actual hands-on work experience, not just home projects or academic exercises. • Profiles should clearly state how much experience they have in each skill area, as this helps streamline the interview process. • Candidates must know their CV/profile inside out, including all projects and responsibilities listed. Any ambiguity or lack of clarity on the candidate’s part can lead to immediate rejection, as we value accuracy and ownership. • They should be able to confidently explain their past experience, challenges handled, and technical contributions.