Data Engineer

We are seeking an experienced Data Engineer to join a data engineering team supporting an Azure-native third-party data enrichment platform. The role focuses on building reliable, scalable, governed, and cost-efficient data pipelines using Databricks, Spark, Snowflake, Azure Data Factory (ADF), Python, and SQL. The platform enriches first-party datasets with external identity and attribute data to support analytics, customer activation, research, and Customer Data Platform (CDP) use cases. The ideal candidate will have strong hands-on data engineering experience, excellent Spark troubleshooting skills, and a strong understanding of data quality, privacy, governance, and performance optimization. Candidates with backgrounds in FAANG, product-based, or Tier-1 technology companies are preferred. Key Responsibilities Data Ingestion & Pipeline Development • Design, develop, and enhance scalable data ingestion pipelines supporting large-volume batch and event-driven workloads. • Build robust ETL/ELT pipelines using PySpark, Python, SQL, Databricks, and Azure Data Factory. • Integrate data from third-party enrichment vendors, including large-scale identity and attribute datasets. • Integrate digital platform data through Conversion API (CAPI) and middleware-based integrations. • Integrate data from rewards and promotions systems, including offer issuance, redemption, and consumption data. • Develop scalable and reusable data engineering frameworks and components. Data Quality, Reliability & Operations • Implement strong data validation, deduplication, auditability, and data quality controls. • Design and implement idempotency, replay, backfill, and recovery strategies to maintain pipeline reliability. • Build and maintain monitoring, alerting, dashboards, and operational readiness capabilities. • Troubleshoot data pipeline failures using root-cause analysis rather than simply rerunning failed jobs. • Analyze Spark logs and diagnose issues related to shuffle, skew, partitioning, memory, and performance. • Improve pipeline stability, reliability, and SLA adherence. Databricks & Spark Engineering • Develop and optimize large-scale data processing solutions using Databricks and Apache Spark. • Apply Spark fundamentals such as partitioning, caching, shuffle optimization, and workload tuning. • Troubleshoot complex Spark failures and performance bottlenecks. • Implement modern Databricks and Delta Lake patterns, including Medallion Architecture. • Contribute to Delta Live Tables (DLT) and other Databricks-based data engineering workflows where applicable. Snowflake & Data Warehousing • Develop and support data solutions using Snowflake for analytics and warehousing workloads. • Apply effective data modeling and query optimization techniques. • Ensure data structures are scalable, maintainable, and optimized for downstream analytical consumption. Data Governance, Privacy & Compliance • Apply data privacy, security, governance, and compliance requirements throughout the data lifecycle. • Work with Unity Catalog and other governance frameworks for access control, lineage, and data management. • Implement appropriate controls for PII and non-PII data. • Maintain documentation for tables, schemas, catalogs, pipelines, and cluster usage. • Support data lineage, auditability, and governance standards across the platform. Cost & Performance Optimization • Design data pipelines with cost efficiency, scalability, and performance in mind. • Optimize cluster sizing, compute utilization, storage, and workload configurations. • Identify and resolve performance bottlenecks across Spark, Databricks, Snowflake, and Azure services. • Balance cost, quality, reliability, and SLA requirements when making technical decisions. Required Skills & Qualifications • 5+ years of hands-on Data Engineering experience. • Strong hands-on development experience with PySpark and SQL. • Strong experience with Python for data engineering and pipeline development. • 5+ years of experience with Databricks, ETL, and Azure. • Strong experience with Azure Data Factory (ADF) for orchestration and data integration. • Strong understanding of Apache Spark fundamentals, including: • Partitioning • Shuffle • Data skew • Performance tuning • Spark troubleshooting • Cluster optimization • Experience with Snowflake and analytics/warehouse workloads. • Experience designing and implementing scalable ETL/ELT pipelines. • Strong understanding of data engineering reliability patterns, including: • Data validation • Idempotency • Replay and backfills • Deduplication • Auditability • Experience with data governance, lineage, access controls, and PII handling. • Strong analytical and problem-solving skills with the ability to perform detailed root-cause analysis. • Ability to work independently and take ownership of technical deliverables with minimal supervision. • Strong understanding of software development lifecycle, coding standards, testing, documentation, and Agile methodologies. Preferred / Nice-to-Have Skills • Experience with event-driven or real-time/streaming data ingestion. • Experience with Delta Lake and Delta Live Tables (DLT). • Experience building configuration-driven data pipelines and reusable frameworks. • Experience with Azure-native data services and integrations. • Experience supporting Customer Data Platforms (CDP) or customer/marketing data ecosystems. • Experience working with third-party data enrichment, identity, or customer attribute datasets. • Knowledge of data privacy, compliance, and enterprise data governance. • Experience working in FAANG, product-based, or Tier-1 technology organizations. Architecture & Engineering Responsibilities • Contribute to HLD, LLD, solution architecture, and data model design. • Evaluate technical options and select appropriate design patterns and reusable components. • Design solutions that optimize performance, scalability, maintainability, quality, and cost. • Translate technical and business requirements into scalable data engineering solutions. • Participate in design reviews, code reviews, and technical discussions. • Identify opportunities to improve existing architectures, frameworks, and engineering practices. Testing, Documentation & Quality • Develop, review, and execute unit and integration test cases. • Validate solutions against technical specifications and business requirements. • Perform defect analysis, root-cause analysis, and remediation. • Maintain technical documentation, data models, pipeline documentation, standards, and operational procedures. • Follow established coding standards, development processes, templates, and quality guidelines. • Support release activities and ensure production readiness. Collaboration & Communication • Collaborate with Data Engineers, Architects, Product teams, Analytics teams, and business stakeholders. • Clarify requirements and provide technical guidance to development and cross-functional teams. • Communicate technical solutions and design decisions effectively to both technical and non-technical stakeholders. • Manage multiple priorities, dependencies, risks, and deliverables in a fast-paced environment. • Proactively identify issues and drive them to resolution. • Contribute to knowledge sharing, reusable assets, and continuous improvement initiatives. ABOUT BRICKRED SYSTEMS BrickRed Systems is a global leader in next-generation technology consulting and workforce solutions, specializing in delivering high-quality talent across digital, engineering, marketing, analytics, finance, operations, and business transformation domains. With a strong emphasis on innovation, scalability, and client success, BrickRed Systems helps organizations solve complex business challenges by providing skilled professionals across strategy, technology, creative, and operational functions. BrickRed Systems fosters a culture of continuous learning, collaboration, and excellence, enabling professionals to contribute to high-impact global initiatives while advancing their careers