Principal Data Engineer
⭐ - Featured Role | Apply direct with Data Freelance Hub
This role is for a Principal Data Engineer on a contract basis, offering $60.00 per hour. Located in Fort Mill, SC or Austin (Hybrid), it requires expertise in AWS, Python, Spark, ETL, SQL, and Pytest, along with a Bachelor's degree.
🌎 - Country
United States
💱 - Currency
$ USD
-
💰 - Day rate
480
-
🗓️ - Date discovered
August 24, 2025
🕒 - Project duration
Unknown
-
🏝️ - Location type
Hybrid
-
📄 - Contract type
Unknown
-
🔒 - Security clearance
Unknown
-
📍 - Location detailed
Fort Mill, SC 29715
-
🧠 - Skills detailed
#Data Pipeline #Big Data #Apache Kafka #S3 (Amazon Simple Storage Service) #SQL Queries #Data Accuracy #Python #Version Control #Automation #Data Governance #Airflow #GIT #Data Processing #Spark (Apache Spark) #SQL (Structured Query Language) #Redshift #Data Ingestion #Terraform #Kafka (Apache Kafka) #Storage #"ETL (Extract #Transform #Load)" #Lambda (AWS Lambda) #Security #Docker #Kubernetes #ML (Machine Learning) #Data Engineering #Databases #Apache Spark #Data Integrity #Pytest #Data Science #BI (Business Intelligence) #Scala #AWS (Amazon Web Services) #Cloud #AWS S3 (Amazon Simple Storage Service)
Role description
Job OverviewWe are looking for a skilled Data Engineer to join our team and help build robust, scalable, and efficient data pipelines. The ideal candidate will have strong expertise in AWS, Python, Spark, ETL Pipelines, SQL, and Pytest. This role involves designing, implementing, and optimizing data pipelines to support analytics, business intelligence, and machine learning initiatives.
Location:
Fort Mill SC or Austin (Hybrid)
Key Responsibilities:
· Design, develop, and maintain ETL pipelines using AWS services, Python, and Spark.
· Optimize data ingestion, transformation, and storage processes for high-performance data processing.
· Work with structured and unstructured data, ensuring data integrity, quality, and governance.
· Develop SQL queries to extract and manipulate data efficiently from relational databases.
· Implement data validation and testing frameworks using Pytest to ensure data accuracy and reliability.
· Collaborate with data scientists, analysts, and software engineers to build scalable data solutions.
· Monitor and troubleshoot data pipelines to ensure smooth operation and minimal downtime.
· Stay up-to-date with industry trends, tools, and best practices for data engineering and cloud technologies.
Required Skills & Qualifications:
· Experience in Data Engineering or a related field.
· Strong proficiency in AWS (S3, Glue, Lambda, EMR, Redshift, etc.) for cloud-based data processing.
· Hands-on experience with Python for data processing and automation.
· Expertise in Apache Spark for distributed data processing.
· Solid understanding of ETL pipeline design and data warehousing concepts.
· Proficiency in SQL for querying and managing relational databases.
· Experience writing unit and integration tests using Pytest.
· Familiarity with CI/CD pipelines and version control systems (e.g., Git).
· Strong problem-solving skills and ability to work in a fast-paced environment.
Preferred Qualifications:
· Experience with Terraform, Docker, or Kubernetes.
· Knowledge of big data tools such as Apache Kafka or Airflow.
· Exposure to data governance and security best practices.
Job Type: Contract
Pay: $60.00 per hour
Application Question(s):
Experience in Data Engineering
Strong proficiency in AWS (S3, Glue, Lambda, EMR, Redshift, etc.)
Experience with Python for data processing and automation.
Expertise in Apache Spark
Experience with Terraform, Docker, or Kubernetes.
Exposure to data governance and security best practices.
Knowledge of big data tools such as Apache Kafka or Airflow.
Education:
Bachelor's (Required)
Work Location: In person
Job OverviewWe are looking for a skilled Data Engineer to join our team and help build robust, scalable, and efficient data pipelines. The ideal candidate will have strong expertise in AWS, Python, Spark, ETL Pipelines, SQL, and Pytest. This role involves designing, implementing, and optimizing data pipelines to support analytics, business intelligence, and machine learning initiatives.
Location:
Fort Mill SC or Austin (Hybrid)
Key Responsibilities:
· Design, develop, and maintain ETL pipelines using AWS services, Python, and Spark.
· Optimize data ingestion, transformation, and storage processes for high-performance data processing.
· Work with structured and unstructured data, ensuring data integrity, quality, and governance.
· Develop SQL queries to extract and manipulate data efficiently from relational databases.
· Implement data validation and testing frameworks using Pytest to ensure data accuracy and reliability.
· Collaborate with data scientists, analysts, and software engineers to build scalable data solutions.
· Monitor and troubleshoot data pipelines to ensure smooth operation and minimal downtime.
· Stay up-to-date with industry trends, tools, and best practices for data engineering and cloud technologies.
Required Skills & Qualifications:
· Experience in Data Engineering or a related field.
· Strong proficiency in AWS (S3, Glue, Lambda, EMR, Redshift, etc.) for cloud-based data processing.
· Hands-on experience with Python for data processing and automation.
· Expertise in Apache Spark for distributed data processing.
· Solid understanding of ETL pipeline design and data warehousing concepts.
· Proficiency in SQL for querying and managing relational databases.
· Experience writing unit and integration tests using Pytest.
· Familiarity with CI/CD pipelines and version control systems (e.g., Git).
· Strong problem-solving skills and ability to work in a fast-paced environment.
Preferred Qualifications:
· Experience with Terraform, Docker, or Kubernetes.
· Knowledge of big data tools such as Apache Kafka or Airflow.
· Exposure to data governance and security best practices.
Job Type: Contract
Pay: $60.00 per hour
Application Question(s):
Experience in Data Engineering
Strong proficiency in AWS (S3, Glue, Lambda, EMR, Redshift, etc.)
Experience with Python for data processing and automation.
Expertise in Apache Spark
Experience with Terraform, Docker, or Kubernetes.
Exposure to data governance and security best practices.
Knowledge of big data tools such as Apache Kafka or Airflow.
Education:
Bachelor's (Required)
Work Location: In person
