Data Engineer

Primary Skills: Databricks, Python, Pyspark, Airflow, Apache Spark Location: San Jose, CA (This is a Hybrid role. 3 days a week in San Jose office.) Duration: 5 months Contract Type: W2 only Responsibilities • Design, develop, and maintain scalable and reliable data pipelines to support large-scale data processing. • Build and optimize data workflows using orchestration tools like Apache Airflow and Spark to support scheduled and event-driven ETL/ELT processes. • Implement complex parsing, cleansing, and transformation logic to normalize data from a variety of structured and unstructured sources. • Collaborate with data scientists, analysts, and application teams to integrate, test, and validate data products and pipelines. • Operate and maintain pipelines running on cloud platforms (AWS) and distributed compute environments (e.g., Databricks). • Monitor pipeline performance, perform root cause analysis, and troubleshoot failures to ensure high data quality and uptime. • Ensure proper security, compliance, and governance of data across systems and environments. • Contribute to the automation and standardization of data engineering processes to improve development velocity and operational efficiency. Required Skills • 9-12 YOE • Proficient in Python and PySpark for data processing and scripting. • Strong experience with SQL for data manipulation and performance tuning. • Deep understanding of distributed data processing with Apache Spark. • Hands-on experience with Airflow or similar orchestration tools. • Experience with cloud services and data tools in AWS (e.g., S3, Lambda, SQS, Gateway, Networking). • Expertise with Databricks for collaborative data engineering and analytics. • Solid understanding of data modeling, data warehousing, and best practices in data pipeline architecture. • Strong problem-solving skills with the ability to work independently on complex tasks. • Familiarity with CI/CD practices and version control (e.g., Git) in data engineering workflows.