Data Engineer
⭐ - Featured Role | Apply direct with Data Freelance Hub
This role is for a Data Engineer with 8-12 years of experience, focusing on AWS Glue and S3. The contract length is unspecified, with a pay rate of "unknown." Key skills include ETL pipeline development, Python, and AWS services.
🌎 - Country
United Kingdom
💱 - Currency
£ GBP
-
💰 - Day rate
-
🗓️ - Date discovered
May 30, 2025
🕒 - Project duration
Unknown
-
🏝️ - Location type
Unknown
-
📄 - Contract type
Unknown
-
🔒 - Security clearance
Unknown
-
📍 - Location detailed
Telford, England, United Kingdom
-
🧠 - Skills detailed
#Security #AWS (Amazon Web Services) #AWS Glue #Metadata #REST (Representational State Transfer) #Spark (Apache Spark) #Data Pipeline #ML (Machine Learning) #Data Management #Storage #Data Catalog #Migration #Data Processing #Cloud #Lambda (AWS Lambda) #PySpark #Scala #Data Engineering #Data Ingestion #Python #AWS S3 (Amazon Simple Storage Service) #Data Accuracy #Data Architecture #Monitoring #Redshift #Deployment #Datasets #DevOps #IAM (Identity and Access Management) #AWS CloudWatch #"ETL (Extract #Transform #Load)" #Data Science #AWS IAM (AWS Identity and Access Management) #S3 (Amazon Simple Storage Service)
Role description
AWS Data Engineer – AWS Glue & S3 (Band 4) – Exp required 8 -12 years
• Design, develop, and maintain scalable ETL pipelines using AWS Glue to process, transform, and load large datasets into AWS S3, Redshift, and other data stores.
• Collaborate with cross-functional teams, including data architects, analysts, and business stakeholders, to gather data requirements and deliver efficient data solutions.
• Implement data ingestion pipelines from various structured and unstructured data sources into AWS S3, ensuring data consistency and accuracy.
• Develop and maintain data transformation scripts in Python or PySpark within AWS Glue to clean, enrich, and standardize data.
• Optimize AWS Glue jobs for performance, ensuring minimal processing times and efficient resource usage.
• Manage and organize data in AWS S3, implementing best practices for data partitioning, versioning, and lifecycle management to optimize storage costs.
• Create and maintain AWS Glue Data Catalogs for metadata management, ensuring data discoverability, lineage, and governance.
• Implement data validation and quality checks within the ETL pipelines to ensure high data accuracy and integrity.
• Collaborate with DevOps teams to automate data pipeline deployments using AWS services like CloudFormation, Lambda, and CI/CD tools.
• Monitor and troubleshoot AWS Glue jobs and S3 processes, using AWS CloudWatch and other monitoring tools to ensure smooth operation and resolve issues in real-time.
• Implement security best practices for data handling in AWS, including encryption of data at rest and in transit (S3, KMS) and access management using AWS IAM roles and policies.
• Work closely with data scientists, analysts, and other engineers to deliver data models that support advanced analytics, reporting, and machine learning efforts.
• Optimize data retrieval performance from AWS S3, employing partitioning, compression, and other techniques to speed up query performance for downstream users.
• Provide guidance and mentorship to junior data engineers, promoting best practices in data processing and AWS cloud services.
• Perform regular reviews of data pipelines, identifying opportunities for improvements in efficiency, scalability, and cost-effectiveness.
• Stay updated on new AWS services and features, recommending and implementing upgrades or migrations as appropriate.
AWS Data Engineer – AWS Glue & S3 (Band 4) – Exp required 8 -12 years
• Design, develop, and maintain scalable ETL pipelines using AWS Glue to process, transform, and load large datasets into AWS S3, Redshift, and other data stores.
• Collaborate with cross-functional teams, including data architects, analysts, and business stakeholders, to gather data requirements and deliver efficient data solutions.
• Implement data ingestion pipelines from various structured and unstructured data sources into AWS S3, ensuring data consistency and accuracy.
• Develop and maintain data transformation scripts in Python or PySpark within AWS Glue to clean, enrich, and standardize data.
• Optimize AWS Glue jobs for performance, ensuring minimal processing times and efficient resource usage.
• Manage and organize data in AWS S3, implementing best practices for data partitioning, versioning, and lifecycle management to optimize storage costs.
• Create and maintain AWS Glue Data Catalogs for metadata management, ensuring data discoverability, lineage, and governance.
• Implement data validation and quality checks within the ETL pipelines to ensure high data accuracy and integrity.
• Collaborate with DevOps teams to automate data pipeline deployments using AWS services like CloudFormation, Lambda, and CI/CD tools.
• Monitor and troubleshoot AWS Glue jobs and S3 processes, using AWS CloudWatch and other monitoring tools to ensure smooth operation and resolve issues in real-time.
• Implement security best practices for data handling in AWS, including encryption of data at rest and in transit (S3, KMS) and access management using AWS IAM roles and policies.
• Work closely with data scientists, analysts, and other engineers to deliver data models that support advanced analytics, reporting, and machine learning efforts.
• Optimize data retrieval performance from AWS S3, employing partitioning, compression, and other techniques to speed up query performance for downstream users.
• Provide guidance and mentorship to junior data engineers, promoting best practices in data processing and AWS cloud services.
• Perform regular reviews of data pipelines, identifying opportunities for improvements in efficiency, scalability, and cost-effectiveness.
• Stay updated on new AWS services and features, recommending and implementing upgrades or migrations as appropriate.
