Lead PySpark Developer

Job Description • 7+ years of experience in Amazon Web Service(AWS) Cloud Computing. • 10+ years of experience in big data and distributed computing. • Very Strong hands-on experience with PySpark, Apache Spark, and Python. • Strong Hands on experience with SQL and NoSQL databases (DB2, PostgreSQL, Snowflake, etc.). • Proficiency in data modeling and ETL workflows. • Proficiency with workflow schedulers like Airflow. • Hands on experience with AWS cloud-based data platforms. • Experience in DevOps, CI/CD pipelines, and containerization (Docker, Kubernetes) is a plus. • Strong problem-solving skills and ability to lead a team • DBT, AWS Astronomer • Lead the design, development, and deployment of PySpark-based big data solutions. • Architect and optimize ETL pipelines for structured and unstructured data. • Collaborate with Client, data engineers, data scientists, and business teams to understand requirements and provide scalable solutions. • Optimize Spark performance through partitioning, caching, and tuning. • Implement best practices in data engineering (CI/CD, version control, unit testing). • Work with cloud platforms like AWS. • Ensure data security, governance, and compliance. • Mentor junior developers and review code for best practices and efficiency. Additional Notes • Please submit the candidate's resume in PDF format. • Please Note: This position is not available under a Corp-to-Corp (C2C) employment structure. Only candidates authorized to work on a W2 basis will be considered at this time. Skills: pyspark,docker,containerization,sql,cloud computing,nosql,apache spark,big data,airflow,data modeling,dbt,devops,distributed computing,etl workflows,amazon web service (aws),python,ci/cd,kubernetes,aws astronomer,aws cloud computing