GCP Data Engineer

Job Title: GCP Data Engineer Location: Phoenix, AZ, USA Job Description: We are seeking a Senior Data Engineer with strong hands-on experience in GCP to design, develop, and operate batch and real-time data pipelines for marketing technology use cases. The ideal candidate will have deep practical experience building Dataflow pipelines, working with Pub/Sub for real-time streaming, and orchestrating workflows using Cloud Composer (Apache Airflow). This role requires a strong engineering mindset, real-world pipeline design experience, and the ability to work across projects and teams to deliver scalable, reliable data solutions. Key Responsibilities Data Pipeline Development • Design, develop, and maintain Dataflow (Apache Beam) pipelines for both batch and real-time data processing. • Implement Pub/Sub-based streaming architectures, including listening to source topics, transforming messages, and publishing to downstream topics. • Build cross-project data pipelines in GCP, ensuring secure and reliable data exchange between projects. Data Transformation & Processing • Perform complex transformations using BigQuery, including: • Advanced SQL queries • Joins across large datasets • Window functions and aggregations • Implement encryption and decryption mechanisms as part of data processing workflows. • Enrich streaming data by querying BigQuery or reference datasets within Dataflow pipelines. Workflow Orchestration • Develop and manage Airflow DAGs using Cloud Composer. • Design DAGs to orchestrate batch and streaming pipelines. • Implement DAG-to-DAG triggering patterns and dependency management. • Monitor, troubleshoot, and optimize pipeline execution and scheduling. Real-Time Processing • Build and enhance real-time data processing components using Pub/Sub and Dataflow. • Handle late-arriving data, retries, error handling, and idempotency in streaming pipelines. • Ensure low-latency and high-throughput data processing for marketing use cases. Platform & Integration • Integrate GCP data pipelines with Adobe Experience Platform (AEP) and related Adobe services. • Support marketing use cases such as customer segmentation, decisioning, and personalization. • Collaborate with analytics, marketing, and platform teams to align data solutions with business needs. Required Technical Skills Google Cloud Platform (GCP) • Strong hands-on experience with: • Cloud Dataflow (Apache Beam) – streaming and batch pipelines • Cloud Pub/Sub – message publishing, subscriptions, and streaming patterns • BigQuery – advanced SQL, performance optimization, partitioning • Cloud Composer (Apache Airflow) – DAG development and orchestration Programming & Querying • Python (advanced): • Writing and maintaining Airflow DAGs • Data processing logic and pipeline components • SQL (advanced): • Complex joins • Window functions • Analytical queries on large datasets Streaming & Pipeline Design • Practical experience designing real-time data processing architectures. • Understanding of: • Event-driven architectures • Exactly-once / at-least-once processing semantics • Error handling and replay strategies Preferred / Nice-to-Have Skills • Experience with Adobe Experience Platform (AEP), Adobe Segmentation, or Adobe Decisioning. • Familiarity with marketing data models (customer profiles, events, attributes). • Knowledge of data security and compliance, including encryption and access controls. • Experience working in multi-project GCP environments. • Exposure to CI/CD practices for data pipelines. Typical Interview Focus Areas (Scenario-Based) Candidates should be able to clearly explain and demonstrate real-world experience with scenarios such as: • Designing a Dataflow pipeline that listens to messages from one Pub/Sub topic and publishes transformed messages to another topic. • Orchestrating batch and streaming pipelines using Airflow DAGs. • Triggering one Airflow DAG from another and managing dependencies. • Handling real-time data processing challenges, such as late data, failures, retries, and scaling. • Designing secure and scalable cross-project GCP pipelines. Ideal Candidate Profile • Strong hands-on implementation experience with GCP data services. • Proven track record of building production-grade batch and real-time pipelines. • Comfortable discussing architecture, trade-offs, and design decisions. • Ability to translate business and marketing requirements into robust data solutions. Priority will be given to candidates with demonstrated, real-world experience implementing Dataflow and Pub/Sub-based streaming pipelines in GCP, especially within marketing or customer data platforms.