Principal Data Engineer – Data & Intelligence

DATA PIPELINE DEVELOPMENT • Architect, design, and oversee development of enterprise-scale ELT/ETL pipelines for finance and revenue data (billing, revenue, GL, opex). • Define and enforce standards for batch, incremental, and streaming ingestion patterns (CDC, watermarking, event-driven ingestion). • Ensure idempotent, fault-tolerant, and highly scalable pipeline design across platforms. • Establish frameworks for error handling, retry strategies, dead-letter queue patterns, and operational resiliency. • Provide technical leadership for multi-source, high-volume data integration pipelines. Must have skills – • skill 1 – 7yrs of exp – Snowflake, Delta Lake • skill 2 – 7yrs of exp Databricks, , DLT, Unity Catalog • skill 3 – 5yrs of exp dbt, PySpark, PLATFORM & TOOLING • Lead architecture and adoption of Snowflake and Databricks platforms for large-scale data processing and analytics. • Define best practices for: • Snowflake (Snowpipe, streams, tasks, query optimization, cost efficiency) • Databricks (PySpark, Delta Live Tables, Unity Catalog, job optimization) • dbt (modular design, testing frameworks, CI/CD integration, reusable components) • Establish and govern orchestration frameworks using Airflow / Azure Data Factory, including DAG standards, dependency design, and monitoring. • Evaluate and drive tooling strategy and platform standardization across teams. CLOUD INFRASTRUCTURE • Architect and optimize cloud-native data platforms on Azure (ADLS Gen2, Event Hub, ADF, Key Vault) or AWS equivalents. • Define standards for infrastructure-as-code (Terraform, Bicep) and environment provisioning. • Drive cost optimization strategies (compute sizing, storage design, partitioning, workload isolation). • Ensure platforms are scalable, secure, and production-ready. LANGUAGES & FRAMEWORKS • Provide deep technical leadership in: • Advanced SQL (query tuning, execution optimization, complex transformations) • Python / PySpark for distributed data processing • Guide teams on best practices, reusable frameworks, and performance optimization. • Oversee development standards for Spark, Scala (where applicable), and automation scripting. STREAMING & REAL-TIME • Architect real-time and near real-time data processing solutions using Kafka / Event Hub and Spark Structured Streaming. • Define patterns for stateful processing, watermarking, checkpointing, and fault tolerance. • Lead implementation of real-time finance/revenue use cases such as reconciliation, anomaly detection signals, and operational reporting. DATA QUALITY & TESTING • Establish enterprise frameworks for data quality, validation, and observability. • Define standards for: • Automated testing (unit, integration, regression) • Data validation (completeness, accuracy, consistency) • Data quality tools (dbt tests, Great Expectations, custom frameworks) • Ensure SLA monitoring, alerting, and data freshness tracking across all pipelines. • Drive proactive data quality and governance practices across teams. DATA MODELING SUPPORT • Interpret and implement architect-defined enterprise data models (star, snowflake, data vault). • Provide guidance on: • SCD (Type 1/2) strategies • Partitioning, clustering, and performance optimization • Collaborate with architects to evolve scalable and reusable data models. • Support semantic layer enablement for analytics and reporting. DEVOPS & ENGINEERING PRACTICES • Define and enforce CI/CD standards for data engineering (GitHub Actions, Azure DevOps). • Establish code quality, versioning, and deployment best practices (branching strategies, PR reviews, release pipelines). • Standardize environment promotion (dev $B" • (J QA $B" • (J prod) and release management. • Drive adoption of engineering excellence practices including reusable frameworks and templates. SECURITY & GOVERNANCE • Lead implementation of enterprise-grade security and governance controls: • RBAC, row/column-level security • PII and CPNI compliance (TISS-310) • Define standards for secrets management and secure pipeline design. • Ensure data lineage, auditability, and compliance readiness across platforms. FINANCE DOMAIN KNOWLEDGE • Deep understanding of finance and revenue data domains, including: • Billing and revenue systems • GL structures and financial reporting • Revenue recognition and reconciliation • Period-end close cycles • Guide engineering teams on accurate implementation of finance logic. • Ensure high data integrity standards for regulated financial data. SOFT SKILLS & COLLABORATION • Act as a technical leader and escalation point across engineering teams. • Partner with architects, product managers, analysts, and business stakeholders. • Drive cross-team alignment and solution consistency. • Communicate complex technical topics clearly to both technical and non-technical audiences. • Lead incident reviews and ensure continuous improvement. PRINCIPAL-LEVEL EXPECTATIONS • Own and drive enterprise-level data engineering strategy and execution. • Lead delivery of large, complex, multi-domain data platforms. • Mentor senior engineers and define technical direction for the team. • Drive tooling, architecture, and platform decisions across programs. • Identify and lead technical debt reduction and modernization initiatives. • Establish best practices, reusable components, and platform standards at scale. • Influence cross-functional teams and leadership decisions on data platform strategy.