GCP Data Engineer

Note : Please apply only if you are a Green Card Holder. Role Summary We are looking for a 6 to 10yrs GCP Data Engineer to design, build, and maintain data pipelines and analytics solutions using Google Cloud Platform services, with a strong focus on BigQuery, Dataproc, Dataflow, Cloud Storage, and Cloud Composer/Airflow in a PySpark-driven environment. The ideal candidate has solid experience in batch and streaming ETL, workflow orchestration, and Bash scripting within GCP. Key Responsibilities • Design, develop, and maintain scalable data pipelines and ETL/ELT workflows using BigQuery, Dataproc, Dataflow, and Cloud Storage. • Build and optimize PySpark-based data processing jobs running on Dataproc or serverless Spark in GCP. • Develop, schedule, and monitor DAGs in Cloud Composer/Apache Airflow for end-to-end workflow orchestration. • Implement data ingestion from varied sources into GCP (APIs, files, databases, streaming) and load into BigQuery data warehouse. • Optimize BigQuery queries, partitioning, clustering, and table design for performance and cost efficiency. • Implement data quality checks, validation rules, and reconciliation for critical data pipelines. • Manage and secure GCS buckets, lifecycle policies, and efficient storage structures for raw, curated, and processed data. • Write and maintain Bash/shell scripts for automation, deployment, and operational tasks across GCP environments. • Collaborate with data architects, analysts, and business stakeholders to translate requirements into technical solutions. • Participate in CI/CD, code reviews, and best practices for version control, testing, and deployment of data pipelines. • Monitor, troubleshoot, and resolve issues in production pipelines with clear root cause analysis and incident reporting. Required Skills and Qualifications • Bachelor’s degree in Computer Science, Engineering, or related field (or equivalent experience). • 6 to 10 years of experience in data engineering with at least 2+ years on Google Cloud Platform. • Strong hands-on experience with: • BigQuery (SQL development, performance tuning, partitioning, clustering). • Dataflow or Dataproc for large-scale data processing. • Cloud Storage (GCS) for data lake design and management. • Cloud Composer / Apache Airflow for DAG development and scheduling. • Strong programming skills in Python and PySpark in a GCP environment. • Proficiency in Linux/Unix and Bash/shell scripting for automation. • Solid understanding of ETL/ELT concepts, data warehousing, and data modeling. • Experience working with large datasets, performance optimization, and cost control on cloud data platforms. • Familiarity with Git and CI/CD practices for data pipelines. • Strong analytical, problem-solving, and communication skills. Good to Have • Google Professional Data Engineer or other GCP certifications. • Experience with additional GCP services such as Pub/Sub, Cloud Functions, Cloud Run, Cloud SQL, Bigtable, or Data Fusion. • Exposure to streaming data pipelines and event-driven architectures. • Experience in Agile/Scrum environments and working with distributed teams.