Sr. Data Engineer

We are Seeking- Data Engineer needed for an Azure-native third-party data enrichment platform built on Databricks/Spark and Snowflake. The platform augments first-party datasets with external identity and attribute data to support analytics, activation, and research. Enriched datasets are consumed by multiple downstream systems including the Customer Data Platform (CDP) and analytics/research stakeholders. A major focus is building reliable, governed, vendor-agnostic datasets while ensuring privacy/compliance, data governance, and cost efficiency. This is an early-stage, fast-moving product — the first live month was recent. You'll be self-driven, ownership-oriented, and contributing to evolving architecture from day one. Key Responsibilities Data Ingestion & Pipeline Development • Build and enhance ingestion pipelines for large batch and event-driven paths (streaming may evolve over time). • Integrate data from third-party enrichment vendors (identity + attributes, very large volumes), digital platforms via Conversion API (CAPI), and Rewards/Promotions systems (e.g., TMT) for offer issuance/redemption/consumption data. Data Quality, Reliability & Operations • Implement strong data validation, idempotency, replay/backfill strategies, and deduplication to prevent quality drift. • Own monitoring, alerting, dashboarding, and operational readiness around core pipelines. • Troubleshoot failures with root cause analysis — interpret Spark logs, diagnose performance issues (shuffle, skew, partitioning), and improve stability and SLA adherence. Governance & Compliance • Apply privacy, compliance, and governance requirements across pipelines and datasets. • Support governance standards: Unity Catalog, lineage, access controls, PII vs non-PII access management. • Document tables, schemas, catalogs, and cluster usage. Cost Governance & Performance Optimisation • Design pipelines with cost awareness from day one: cluster sizing, workload tuning, efficient compute/storage usage. • Make trade-off decisions balancing cost vs quality vs SLA. Collaboration & Ownership • Work in a small, fast-moving team; be self-driven and ownership-oriented. • Raise and manage data quality escalations when issues are detected. • Contribute to evolving architecture decisions. Must-Have Skills • PySpark + SQL – hands-on, not only orchestration (7+ years) • Databricks – notebooks/jobs, performance tuning, medallion patterns (7+ years) • Spark fundamentals – partitioning, skew/shuffle optimisation, failure diagnosis via logs • Snowflake – data modeling and usage for analytics/warehousing workloads (7+ years) • Power BI & Data Modeling (7+ years) • Azure Data Factory (ADF) – orchestration and Azure-native integrations • Data reliability patterns – validation, idempotency, replay/backfills, deduplication, auditability • Data governance – Unity Catalog (preferred), lineage, access control patterns • Microsoft Fabric – must-have platform experience Role Expectations • Interpret application/feature/component designs and develop in accordance with specifications. • Code, debug, test, document, and communicate product/component/feature development stages. • Validate results with user representatives; integrate and commission the overall solution. • Select appropriate technical options — reuse, improve, or reconfigure existing components, or create own solutions. • Optimise efficiency, cost, and quality across deliverables. • Set FAST goals for self and team; influence and improve customer satisfaction. Performance Areas Code & Documentation • Follow coding standards, templates, and checklists; review code for team and peers. • Create/review deliverable documents: design documentation, requirements, test cases/results. Testing & Configuration • Create and review unit test cases, scenarios, and execution. • Review test plans; provide clarifications to the testing team. • Define and govern configuration management plans; ensure team compliance. Design & Architecture • Contribute to creation of HLD, LLD, and SAD for applications, features, and data models. • Present design options to customers; conduct product demos. Project & Defect Management • Manage delivery of modules and user stories. • Perform defect RCA and mitigation; identify defect trends and take proactive quality improvement measures. • Create and provide input for effort estimation. Team & Knowledge Management • Set FAST goals and provide feedback; understand team aspirations and provide growth opportunities. • Consume and contribute to project-related documents, SharePoint, and libraries. • Execute and monitor release processes. • Take relevant domain/technology certifications. ABOUT BRICKRED SYSTEMS BrickRed Systems is a global leader in next-generation technology, consulting, and business process services. The company helps organizations accelerate digital transformation through innovative, scalable, and high-impact solutions across industries worldwide. With ISO 27001 and ISO 9001 certifications and over a decade of experience, BrickRed Systems leverages cloud computing, analytics, hyper-automation, robotics, cognitive computing, and emerging technologies to deliver measurable business outcomes. The organization fosters a strong continuous learning culture, enabling clients and talent to build future-ready digital capabilities and maintain a competitive edge in a rapidly evolving global marketplace.