Senior Data Engineer

Senior Data Engineer We are looking for a data engineer with 8+ years of experience responsible for designing and building reliable data pipelines and transforming raw data into clean, consumable datasets. The role focuses on developing and maintaining ETL and ELT workflows, data modeling, and ensuring data quality and lineage for downstream users. You will collaborate with product and analytics teams to translate business requirements into scalable data products. This position is hands-on and involves coding, optimizing queries, troubleshooting pipelines, and implementing best practices for data governance and reliability. Role & Responsibilities: • Design, develop, and maintain scalable and reliable data pipelines and platforms to support enterprise analytics and reporting. • Collaborate with business and technical teams to understand data requirements and translate them into efficient data engineering solutions. • Build data ingestion frameworks using Kafka Confluent and integrate data from multiple cloud and on-premises sources. • Implement ELT and ETL pipelines using PySpark, SQL, Dataform, DBT, and Dataproc for efficient data transformation and processing. • Design and manage data models, schemas, and data warehousing structures optimized for performance and scalability. • Ensure data quality, consistency, and integrity through effective data validation and monitoring mechanisms. • Implement data governance practices including data lineage, metadata management, privacy, and compliance. • Develop automated monitoring and alerting mechanisms to ensure reliability, availability, and performance of data pipelines. • Collaborate with cross-functional teams to enable downstream reporting, analytics, and AI/ML model consumption. • Follow best practices for version control, documentation, and CI/CD workflows using Git and infrastructure-as-code tools. • Provide operational support for data pipelines and proactively troubleshoot performance issues. • Continuously optimize data processing, query execution, and resource utilization in Azure environments. • Build and maintain RESTful API integrations to support data ingestion from and synchronization with web applications and enterprise systems. • Contribute to lightweight web application development efforts where data pipelines interface directly with front-end or full-stack components. Requirements: • Bachelor Degree in Computer Science or related field and 8 years related professional experience Required or Required • Associate Degree and 10 years related professional experience Required or Required • High School Diploma/GED and 12 years related professional experience Required Required Skills • Strong programming skills in Python and PySpark for large-scale data processing. • Proficiency in SQL for data manipulation, analysis, and performance tuning. • Experience with Dataform, Dataproc, and BigQuery for data pipeline development and orchestration. • Understanding of Data Governance principles and implementation practices. • Experience using Git for version control and CI/CD pipelines for automated deployments. • Experience building and consuming RESTful APIs, including enterprise integrations (OAuth2, OpenID Connect, webhooks, pagination, rate limiting, retry logic, error handling). • Some hands-on web application development experience (full-stack or front-end). Nice to Have : • Experience configuring and building pipelines within Databricks • Working knowledge of Infrastructure as Code (IaC) for cloud resource management and automation • Familiarity with DBT, Machine Learning, and AI concepts is a good advantage • Hands-on experience with Kafka and Confluent for real-time data streaming • Knowledge of Cloud Scheduler and Dataflow for automation and workflow management • Full-stack application development experience using Node.js, TypeScript, React, and Express • Experience designing contract-first APIs (OpenAPI, Orval, Zod) • Proficiency with React, Vite, Tailwind CSS, shadcn/ui, and TanStack Query • Experience with PostgreSQL and ORM frameworks (Drizzle ORM preferred) • Experience integrating engineering, construction, ERP, or project management systems • Experience working with metadata-driven applications, configurable mappings, and schema transformation frameworks • Candidates must be legally authorized to work permanently (i.e., without time limitations, without restrictions or without need for work sponsorship) in the country where this position is located. • Disclaimer: Brooksource, Medasource, and Calculated Hire are part of the Eight Eleven Group family of companies and operate under Eight Eleven Group, LLC. All employees receive the same benefits, policies, and terms of employment. • EEO: We are committed to creating an inclusive environment for all employees and applicants. We do not discriminate on the basis of race, color, religion, creed, sex, sexual orientation, gender identity or expression, national origin, ancestry, age, disability, genetic information, marital status, military or veteran status, citizenship, pregnancy (including childbirth, lactation, and related conditions), or any other protected status in accordance with applicable federal, state, and local laws. • Benefits & Perks: Brooksource offers competitive medical, dental, vision, Health Savings Account, Dependent Care FSA, and supplemental coverage with plans that can fit each employee’s needs. We offer a 401k plan that includes a company match and is fully vested after you become eligible, paid time off, sick time, and paid company holidays. We also offer an Employee Assistance Program (EAP) that provides services like virtual counseling, financial services, legal services, life coaching, etc. • Pay Disclaimer: The pay range for this job level is a general guideline only and not a guarantee of compensation or salary. Additional factors considered in extending an offer include (but are not limited to) responsibilities of the job, education, experience, knowledge, skills, and abilities, as well as internal equity, alignment with market data, applicable bargaining agreement (if any), or other law.