ETL Developer

Job description: Role Summary We’re looking for an experienced ETL Developer to build and optimize high-throughput, resilient data pipelines that move and transform data from on-prem sources to cloud destinations. You’ll design, develop, and support Ab Initio graphs and plans, integrate with AWS and/or Azure services, and partner with platform, security, and analytics teams to deliver governed, production-grade data at scale. What You’ll Do • Design & Develop: Build ETL processes using Ab Initio graphs/plans (GDE, Co>Operating System, EME, and Conduct>It) to ingest, transform, and publish data. • Migrate Pipelines: Move data workflows from on-prem to cloud targets (batch and near-real-time), ensuring restartability, parameterization (PDL), metadata management, and resiliency. • Cloud Integration: Integrate with AWS (e.g., S3, Redshift, Glue/Glue Catalog, Lambda, EMR, MSK/Kinesis) and/or Azure (e.g., ADLS Gen2, Synapse, Data Factory, Event Hubs). • Source Systems & CDC: Build connectors/jobs for Oracle, SQL Server, DB2, files, MQ/Kafka; implement incremental loads/CDC patterns where applicable. • Performance & Reliability: Tune memory/parallelism/partitioning, optimize file and database I/O, and implement robust error handling, alerting, and SLA monitoring. • Security & Governance: Apply data quality checks (e.g., Ab Initio DQE), lineage/metadata practices, and enterprise security controls (IAM, KMS/Key Vault, tokenization). • DevOps & CI/CD: Utilize version control (Git/EME), automated build/deploy, environment promotion, and infrastructure coordination with platform teams. • Operations: Troubleshoot production issues, perform capacity planning, and deliver clear runbooks. • Collaboration: Partner with data architects, platform/cloud engineers, and analysts; contribute to standards and best practices. Required Qualifications • 5+ years of hands-on ETL development using Ab Initio (GDE, EME, Co>Operating System; Conduct>It scheduling). • Proven Delivery: Experience with high-volume batch and near-real-time pipelines, including restart/recovery, parameterization, and metadata-driven design. • Database Expertise: Strong SQL and performance tuning across major RDBMS (Oracle/SQL Server/DB2; plus Redshift/Synapse/Snowflake a plus). • Cloud Experience: Production experience integrating Ab Initio with AWS and/or Azure for data landing, processing, and analytics. • Technical Foundations: Solid Linux/Unix fundamentals and scripting (bash, Python preferred). • Messaging & Networking: Experience with Kafka/MQ (publish/subscribe), file transfer patterns, and secure networking (VPC/VNet, PrivateLink/Private Endpoints). • Data Integrity: Familiarity with data quality, lineage, and compliance for sensitive data (e.g., PHI/PII). • Problem Solving: Excellent troubleshooting skills and the ability to own solutions end-to-end. Preferred / “Nice to Have” • Cloud Certifications: AWS Data Analytics / Solutions Architect, or Azure Data Engineer Associate. • Modern Data Stack: Experience with Glue/Spark or Synapse Spark for complementary processing; EMR/Databricks exposure a plus. • Orchestration: Experience with Airflow/ADF alongside Conduct>It; event-driven designs with Lambda/Functions. • Infrastructure as Code (IaC): Awareness of Terraform, CloudFormation, or Bicep to collaborate with platform teams. • Storage Patterns: Experience with Snowflake or Delta Lake patterns on S3/ADLS. • Observability: Proficiency with CloudWatch, Azure Monitor, Prometheus/Grafana, or Splunk. • Methodology: Agile/SAFe delivery in regulated environments (healthcare/financial services).