Senior Data Engineer

We are seeking a Senior Data Engineer / Technical Lead to architect, develop, and lead enterprise-scale finance and revenue data solutions. The ideal candidate will have strong expertise in Snowflake, Databricks, Python, PySpark, SQL, ETL/ELT, real-time data processing, and cloud data platforms, along with the ability to provide technical leadership across engineering teams. Key Responsibilities Data Pipeline Development • Architect and oversee enterprise-scale ETL/ELT pipelines for finance and revenue data, including billing, revenue, GL, and OPEX. • Design batch, incremental, and streaming ingestion patterns using CDC, watermarking, and event-driven architectures. • Build highly scalable, fault-tolerant, and idempotent data pipelines. • Establish standards for error handling, retries, dead-letter queues, monitoring, and operational resiliency. • Provide technical leadership for high-volume, multi-source data integration. Data Platforms & Tooling • Lead the architecture and adoption of Snowflake and Databricks for large-scale data processing and analytics. • Implement Snowflake capabilities including Snowpipe, Streams, Tasks, query optimization, and cost optimization. • Drive Databricks best practices across PySpark, Delta Live Tables, Unity Catalog, and job optimization. • Establish best practices for dbt, including modular development, testing, CI/CD, and reusable components. • Govern orchestration frameworks using Airflow and Azure Data Factory (ADF). • Evaluate and standardize data engineering tools and platforms across teams. Cloud Infrastructure • Architect cloud-native data platforms using Azure ADLS Gen2, Event Hub, ADF, Key Vault, or equivalent AWS services. • Define infrastructure-as-code standards using Terraform and Bicep. • Drive cloud cost optimization through compute sizing, storage design, partitioning, and workload isolation. • Ensure data platforms are secure, scalable, highly available, and production-ready. Programming & Data Processing • Provide technical leadership in Advanced SQL, Python, and PySpark. • Optimize complex queries, transformations, and distributed processing workloads. • Establish coding standards, reusable frameworks, and performance optimization practices. • Support Spark, Scala, and automation scripting where applicable. Streaming & Real-Time Data • Architect real-time and near-real-time solutions using Kafka, Azure Event Hub, and Spark Structured Streaming. • Implement patterns for stateful processing, watermarking, checkpointing, and fault tolerance. • Support real-time finance and revenue use cases such as reconciliation, anomaly detection, and operational reporting. Data Quality & Testing • Establish enterprise frameworks for data quality, validation, testing, and observability. • Define automated unit, integration, and regression testing standards. • Implement data validation for completeness, accuracy, and consistency. • Utilize dbt tests, Great Expectations, and custom data quality frameworks. • Ensure SLA monitoring, alerting, and data freshness tracking. Data Modeling • Work with enterprise data models including Star Schema, Snowflake Schema, and Data Vault. • Provide guidance on SCD Type 1/2, partitioning, clustering, and performance optimization. • Collaborate with architects to develop scalable and reusable data models. • Support semantic-layer enablement for analytics and reporting. DevOps & Engineering Practices • Establish CI/CD standards using GitHub Actions and Azure DevOps. • Enforce code quality, version control, branching, peer reviews, and release management practices. • Standardize environment promotion across Development, QA, and Production. • Drive adoption of reusable frameworks, templates, and engineering best practices. Security & Governance • Implement enterprise security and governance controls, including RBAC, row-level and column-level security. • Ensure PII/CPNI compliance, secure secrets management, and secure pipeline design. • Maintain data lineage, auditability, and compliance readiness across platforms. Finance Domain • Demonstrate strong understanding of billing, revenue, GL, financial reporting, revenue recognition, reconciliation, and period-end close processes. • Guide engineering teams in accurately implementing finance-related business logic. • Maintain high data-integrity standards for regulated financial data. Technical Leadership & Collaboration • Serve as a technical leader and escalation point across engineering teams. • Partner with architects, product managers, analysts, and business stakeholders. • Translate complex technical concepts into clear business and technical communications. • Provide guidance on HLD, LLD, SAD, architecture, data models, and solution design. • Conduct design reviews, code reviews, technical presentations, and product demonstrations. • Lead incident reviews, root-cause analysis, defect mitigation, and continuous improvement. • Mentor team members, set goals, provide feedback, and foster strong team engagement. Software Development & Delivery • Interpret application, feature, and component designs and translate them into production-ready solutions. • Code, debug, test, document, and communicate throughout the development lifecycle. • Review and create unit test cases, test scenarios, and execution plans. • Manage user stories, module delivery, defects, releases, and project timelines. • Provide effort estimates and proactively identify dependencies, risks, and technical challenges. • Ensure solutions meet quality, performance, cost, security, and compliance standards. Required Knowledge & Skills • Strong understanding of Software Development Life Cycle (SDLC) and Agile/Scrum or Kanban methodologies. • Proficiency in multiple programming and technology skill clusters. • Strong knowledge of DBMS, data modeling, operating systems, software platforms, and IDEs. • Experience with RAD, modeling technologies, and interface definition concepts. • Excellent analytical, problem-solving, decision-making, and communication skills. • Ability to manage multiple priorities and work effectively under pressure. • Strong customer-facing and stakeholder-management skills. Performance Expectations • Adherence to engineering processes, coding standards, and project timelines. • High-quality, defect-free code and deliverables. • Timely completion of development, testing, documentation, and releases. • Compliance with security, governance, and mandatory training requirements. • Continuous improvement in application performance, cost efficiency, quality, and customer satisfaction. ABOUT BRICKRED SYSTEMS BrickRed Systems is a global leader in next-generation technology consulting and workforce solutions, specializing in delivering high-quality talent across digital, engineering, marketing, analytics, finance, operations, and business transformation domains. With a strong emphasis on innovation, scalability, and client success, BrickRed Systems helps organizations solve complex business challenges by providing skilled professionals across strategy, technology, creative, and operational functions. BrickRed fosters a culture of continuous learning, collaboration, and excellence, enabling professionals to contribute to high-impact global initiatives while advancing their careers.