Pyspark Lead Developer
⭐ - Featured Role | Apply direct with Data Freelance Hub
This role is for a Sr PySpark Lead Developer, offering a long-term contract in NYC, NY (Hybrid). Key skills include extensive PySpark experience, Python proficiency, and expertise in data processing frameworks. Familiarity with Kubernetes and big data storage solutions is required.
🌎 - Country
United States
💱 - Currency
$ USD
-
💰 - Day rate
-
🗓️ - Date discovered
June 27, 2025
🕒 - Project duration
Unknown
-
🏝️ - Location type
Hybrid
-
📄 - Contract type
Unknown
-
🔒 - Security clearance
Unknown
-
📍 - Location detailed
New York City Metropolitan Area
-
🧠 - Skills detailed
#SQL (Structured Query Language) #Data Engineering #YARN (Yet Another Resource Negotiator) #Kubernetes #HDFS (Hadoop Distributed File System) #S3 (Amazon Simple Storage Service) #Data Pipeline #Scala #PySpark #"ETL (Extract #Transform #Load)" #Data Storage #Data Processing #Spark SQL #Data Lake #Storage #Spark (Apache Spark) #Big Data #Python #Distributed Computing
Role description
Heading 1
Heading 2
Heading 3
Heading 4
Heading 5
Heading 6
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur.
Block quote
Ordered list
- Item 1
- Item 2
- Item 3
Unordered list
- Item A
- Item B
- Item C
Bold text
Emphasis
Superscript
Subscript
Job Title: Sr PySpark Lead Developer
Location - NYC , NY (Hybrid)
Job Duration: Long Term
Job Description:
• Proven experience as a Lead Spark Developer, Data Engineer, or similar role, with extensive hands-on experience in PySpark.
• Strong proficiency in Python, Spark APIs, and data processing frameworks.
• Deep understanding of distributed computing principles and best practices.
• Expertise in architecting scalable, fault-tolerant data processing systems with PySpark.
• Strong skills in fine-tuning Spark applications, including configuration tuning, memory management, and resource allocation.
• Experience with Spark SQL, DataFrames, and Dataset APIs.
• Experience of cluster management tools like Kubernetes, YARN, Mesos
• Experience with big data storage solutions such as HDFS, S3, Parquet, ORC
• Experience designing and implementing efficient data pipelines, data lakes, ETL workflows.
