Data Engineer

We are hiring a Data Engineer to join a modern cloud data engineering team responsible for building scalable data pipelines, cloud integrations, and enterprise data products. The ideal candidate will have strong expertise in Azure Data Engineering, Azure Databricks Lakehouse, Azure Data Factory (ADF), SQL, Spark, PySpark, and ETL development. Full Stack Engineers from leading technology companies are also encouraged to apply, provided they have strong hands-on experience in Azure Data Engineering technologies and meet the required data engineering skill set. This role involves designing scalable cloud-based data solutions, optimizing distributed data processing workloads, and collaborating with architects, analysts, and engineering teams to deliver enterprise-grade data platforms. Key Responsibilities • Design and develop scalable cloud-based data pipelines. • Build ETL/ELT workflows using Azure Data Factory and Databricks. • Develop high-performance PySpark and Spark SQL applications. • Design data ingestion, transformation, and integration pipelines. • Optimize Spark jobs for performance and scalability. • Implement data orchestration using Azure Data Factory. • Work with Azure Data Lake Storage and cloud-based data storage solutions. • Develop reusable, scalable, and fault-tolerant data engineering solutions. • Build streaming data pipelines using Kafka/Event Hub technologies. • Apply best practices for indexing, partitioning, and query optimization. • Optimize distributed data processing workloads. • Participate in Agile development and DevOps processes. • Collaborate with architects, analysts, and product teams. • Prepare technical documentation and system design documents. • Mentor junior engineers and participate in knowledge-sharing sessions. • Identify automation opportunities and improve engineering processes. • Support production deployments, debugging, and performance tuning. • Design low-level technical solutions for multiple components. Required Technical Skills • Azure Data Factory (ADF) • Azure Databricks Lakehouse • Azure Data Lake Storage (ADLS) • SQL • PySpark • Spark SQL • ETL Development • Azure DevOps • Azure Cloud • Kafka / Event Hub • Parquet • Data Streaming • Data Pipeline Development • Performance Optimization • Indexing • Partitioning • Distributed Computing • Fault Tolerance • Idempotency • Service-Oriented Architecture (SOA) • CI/CD • Agile / Scrum Preferred Skills • Experience building enterprise-scale cloud data platforms. • Strong understanding of Spark execution plans and DAG optimization. • Experience optimizing large-scale ETL workloads. • Knowledge of distributed cloud computing concepts. • Experience implementing automated data engineering solutions. • Exposure to AI-enabled or Agentic AI data engineering solutions. • Strong documentation and technical communication skills. • Experience mentoring junior engineers. Responsibilities Across SDLC • Develop clean, reusable, and scalable code. • Perform unit testing and code reviews. • Debug production issues and perform root cause analysis. • Estimate effort for development tasks. • Create technical documentation. • Execute release and deployment activities. • Design Low-Level Design (LLD) documentation. • Follow coding standards and engineering best practices. • Ensure quality, scalability, and maintainability of applications. Qualifications • Bachelor's degree in Engineering, Computer Science, Information Technology, MCA, BCA, or equivalent. • Strong analytical and problem-solving skills. • Excellent written and verbal communication. • Experience working in Agile environments. • Ability to collaborate with cross-functional teams. Mandatory Requirements • 1–3 years of Data Engineering experience • Strong experience with Azure Databricks Lakehouse • Strong expertise in Azure Data Factory (ADF) • Hands-on experience with SQL, PySpark, Spark SQL, and ETL • Experience with Azure Data Lake Storage (ADLS) • Knowledge of Kafka/Event Hub • Experience with Azure DevOps and CI/CD • Strong understanding of Indexing, Partitioning, and Performance Optimization • Experience with Distributed Computing Concepts (Fault Tolerance, Idempotency, SOA) ABOUT BRICKRED SYSTEMS BrickRed Systems is a global leader in next-generation technology consulting and workforce solutions, specializing in delivering high-quality talent across digital, engineering, marketing, analytics, finance, operations, and business transformation domains. With a strong emphasis on innovation, scalability, and client success, BrickRed Systems helps organizations solve complex business challenges by providing skilled professionals across strategy, technology, creative, and operational functions. BrickRed Systems fosters a culture of continuous learning, collaboration, and excellence, enabling professionals to contribute to high-impact global initiatives while advancing their careers.