E-Solutions
Sr. BIgdata Engineer - Local to NC Only
⭐ - Featured Role | Apply direct with Data Freelance Hub
This role is for a Sr. Bigdata Engineer in Charlotte, NC, with a contract length of "unknown" and a pay rate of "unknown." Key skills include Python/PySpark, Hadoop, Airflow, and strong SQL. Experience in ETL migration and regulated environments is preferred.
🌎 - Country
United States
💱 - Currency
$ USD
-
💰 - Day rate
Unknown
-
🗓️ - Date
May 23, 2026
🕒 - Duration
Unknown
-
🏝️ - Location
On-site
-
📄 - Contract
Unknown
-
🔒 - Security
Unknown
-
📍 - Location detailed
Charlotte, NC
-
🧠 - Skills detailed
#Apache Airflow #Data Engineering #Logging #Hadoop #Infrastructure as Code (IaC) #PySpark #Clustering #Monitoring #Batch #"ETL (Extract #Transform #Load)" #Storage #Airflow #Python #Data Modeling #Spark (Apache Spark) #Data Mapping #ML (Machine Learning) #Data Storage #Debugging #SQL Server #AI (Artificial Intelligence) #Docker #Metadata #GIT #SSIS (SQL Server Integration Services) #SQL (Structured Query Language) #DevOps #YARN (Yet Another Resource Negotiator) #Data Lake #Migration #REST (Representational State Transfer) #Indexing #HDFS (Hadoop Distributed File System) #MongoDB #Programming #Scala #Databases #Compliance #Big Data #Jenkins
Role description
Role : Sr. BIgdata Engineer
Location : Charlotte, NC (Onsite)
Required Skills & Experience
Programming: Python/PySpark, Scala is a plus
Big Data: Hadoop (HDFS, YARN), Hive, Spark (optimization, tuning)
Orchestration: Apache Airflow
Databases/ETL: MongoDB (indexing, sharding, tuning) SQL Server & SSIS (development, migration) Strong SQL & stored procedures
Data Lake: HDFS, Hive, Parquet/ORC, partitioning, compaction
APIs: REST-based ingestion Reverse engineering & lineage tools
CI/CD & DevOps: Git, Jenkins, Docker, IaC
Monitoring: logging, metrics, lineage
Key Responsibilities
Reverse Engineering & Data Mapping
Reverse engineer ETL pipelines (SSIS, Spark, stored procedures) to document data
flows, logic, and transformations.
Perform detailed source-to-target mappings with field-level transformations and business
rules.
Build data dictionaries, lineage, and mapping artifacts.
Collaborate with SMEs to uncover undocumented logic.
Identify data model gaps and recommend remediation.
ETL Pipeline Remediation
Design and refactor pipelines aligned to new source APIs and data contracts.
Re-engineer ETL for 1:1 functional parity during migrations.
Implement schema evolution, transformations, and mapping changes (batch &
streaming).
Eliminate redundancy and optimize legacy logic.
Build modular, reusable pipelines using Spark/PySpark/Scala.
Modernize SSIS and integrate with orchestration frameworks.
Orchestrate workflows in Airflow (DAGs, dependencies, SLAs).
Implement logging, error handling, alerting, and metadata capture.
Data Storage Optimization
Simplify schemas; remove redundant/obsolete data across Hive and MongoDB.
Optimize partitioning, clustering, and file formats (Parquet, ORC, Avro).
Redesign MongoDB indexing, sharding, and collections.
Tune HDFS, Hive, MongoDB, and SQL Server for performance and cost.
Implement lifecycle management, archival, and retention.
Functional Skills
• Experience in ETL migration/remediation projects
• Strong reverse engineering of legacy ETL (SSIS, Spark, scripts)
• Expertise in STM, transformation specs, and lineage artifacts
• Data modeling (dimensional, normalized, denormalized)
• Schema evolution and zero-downtime migrations
• Performance tuning across compute and storage layers
• Strong debugging and problem-solving for distributed systems
Preferred Qualifications
AI/ML-assisted ETL remediation or code conversion
Experience with Wiz or Palo Alto Prisma (APIs, data models, risk metrics)
Prior Prisma to Wiz (or similar CSPM/CNAPP) migrations
Knowledge of CSPM/CNAPP domains (vulnerabilities, identities, exposures)
Experience in regulated, compliance-heavy environments
Role : Sr. BIgdata Engineer
Location : Charlotte, NC (Onsite)
Required Skills & Experience
Programming: Python/PySpark, Scala is a plus
Big Data: Hadoop (HDFS, YARN), Hive, Spark (optimization, tuning)
Orchestration: Apache Airflow
Databases/ETL: MongoDB (indexing, sharding, tuning) SQL Server & SSIS (development, migration) Strong SQL & stored procedures
Data Lake: HDFS, Hive, Parquet/ORC, partitioning, compaction
APIs: REST-based ingestion Reverse engineering & lineage tools
CI/CD & DevOps: Git, Jenkins, Docker, IaC
Monitoring: logging, metrics, lineage
Key Responsibilities
Reverse Engineering & Data Mapping
Reverse engineer ETL pipelines (SSIS, Spark, stored procedures) to document data
flows, logic, and transformations.
Perform detailed source-to-target mappings with field-level transformations and business
rules.
Build data dictionaries, lineage, and mapping artifacts.
Collaborate with SMEs to uncover undocumented logic.
Identify data model gaps and recommend remediation.
ETL Pipeline Remediation
Design and refactor pipelines aligned to new source APIs and data contracts.
Re-engineer ETL for 1:1 functional parity during migrations.
Implement schema evolution, transformations, and mapping changes (batch &
streaming).
Eliminate redundancy and optimize legacy logic.
Build modular, reusable pipelines using Spark/PySpark/Scala.
Modernize SSIS and integrate with orchestration frameworks.
Orchestrate workflows in Airflow (DAGs, dependencies, SLAs).
Implement logging, error handling, alerting, and metadata capture.
Data Storage Optimization
Simplify schemas; remove redundant/obsolete data across Hive and MongoDB.
Optimize partitioning, clustering, and file formats (Parquet, ORC, Avro).
Redesign MongoDB indexing, sharding, and collections.
Tune HDFS, Hive, MongoDB, and SQL Server for performance and cost.
Implement lifecycle management, archival, and retention.
Functional Skills
• Experience in ETL migration/remediation projects
• Strong reverse engineering of legacy ETL (SSIS, Spark, scripts)
• Expertise in STM, transformation specs, and lineage artifacts
• Data modeling (dimensional, normalized, denormalized)
• Schema evolution and zero-downtime migrations
• Performance tuning across compute and storage layers
• Strong debugging and problem-solving for distributed systems
Preferred Qualifications
AI/ML-assisted ETL remediation or code conversion
Experience with Wiz or Palo Alto Prisma (APIs, data models, risk metrics)
Prior Prisma to Wiz (or similar CSPM/CNAPP) migrations
Knowledge of CSPM/CNAPP domains (vulnerabilities, identities, exposures)
Experience in regulated, compliance-heavy environments
