Lead Data Engineer (Only W2 or Selfcorp)

Job Title: Lead Data Engineer (Only W2 or Selfcorp) Location: Silver Spring, MD (Onsite usually every Wednesday) Position Type: Contract Top Skills • Databricks • Azure • CI/CD automation experience • Data Lake • SQL/Python • Someone who can mentor and lead. (will not be managing anyone) • Association background – Nice to have Job Description: What You Will Do: • Design, build, and optimize scalable, high-performance data pipelines to support analytics, AI/ML, and business intelligence workloads. • Troubleshoot data pipeline failures, latency issues, and infrastructure bottlenecks, implementing proactive monitoring and alerting. • Lead the adoption of DevOps and automation practices, including CI/CD pipelines, Infrastructure as Code (IaC), and workflow orchestration. • Implement ETL/ELT solutions for efficient data ingestion, transformation, and integration from multiple sources. • Build and manage RESTful APIs to integrate data from various data sources with strong proficiency in handling OAuth authentication, JSON data parsing, and building scalable data pipelines. • Translate complex business requirements into technical solutions, ensuring alignment with enterprise data strategies. • Manage and optimize Azure cloud data storage solutions for cost and performance efficiency. • Ensure data quality, integrity, and consistency by implementing validation, monitoring and error-handling frameworks. • Evaluate and recommend emerging data technologies and architectures to enhance scalability, efficiency, and innovation. • Define and enforce data governance, security, and compliance policies, ensuring alignment with industry standards. • Optimize query performance, indexing, and partitioning strategies to enhance data accessibility and speed. • Collaborate with data architects, data analysts and business stakeholders to deliver data-driven solutions that support decision-making. • Mentor and provide technical guidance to junior data engineers, fostering a culture of best practices and continuous improvement. • Takes authority, responsibility, and accountability for exploiting the value of enterprise information assets and of the analytics used to render insights for decision making automated decisions and augmentation of human performance. Education • Bachelor’s degree in computer science, engineering, or related field. Related Work Experience • Seven (7+) plus years building enterprise-level data solutions on cloud platforms. Skills: • Primary expertise in Databricks, including Databricks Workflows for orchestrating scalable, production-grade data pipelines and ML workflows. • Advanced programming skills in Python and SQL for developing robust data engineering, analytics, and machine learning solutions. • Proficient in Apache Spark (PySpark) for large-scale distributed data processing and real-time analytics on big data platforms. • Experience in integrating data from social media platforms using RESTful APIs, with strong proficiency in handling OAuth authentication, JSON data parsing, and building scalable data pipelines. • Strong DevOps capabilities, including CI/CD pipeline design, automated deployments, and monitoring for data solutions using tools like Shelling Scripting, Azure DevOps, GitHub Actions, and Terraform. • Deep experience with data pipeline design (ETL/ELT), including structured streaming and batch ingestion, transformation, and orchestration using tools like Databricks, Airflow, and Kafka. • Proven expertise in working with relational (PostgreSQL, SQL Server) and NoSQL (MongoDB, Cassandra) databases, as well as data warehouse platforms such as Azure Synapse. • Experience in data governance frameworks, including data quality, lineage, cataloging, access control, and compliance with security and regulatory standards (e.g., GDPR, CCPA). • Experience implementing data quality controls, metadata management, and lineage tracking using Unity Catalog, Great Expectations, and custom rule-based validation frameworks. • Proficient in cloud-native data architectures, especially on Azure, leveraging services such as Azure Data Factory, Azure Key Vault, and Azure Monitor for end-to-end solution management. • Proven experience designing and managing the full machine learning lifecycle using tools such as MLflow, encompassing model deployment, performance tracking, and scalable retraining processes. • Familiarity with containerization and orchestration using Docker and Kubernetes for deploying scalable and portable data services is a plus. • Excellent collaborator with cross-functional teams including data scientists, business analysts, and IT, translating complex technical requirements into actionable, value-driven data solutions. • Strong analytical, debugging, and optimization skills for improving pipeline performance, data quality, and reliability. • Adept at aligning technical data solutions with business goals by understanding data sources, analytical objectives, and enterprise data strategy. If you believe you are qualified for this position and are currently in the job market or interested in making a change, please email me the resume along with contact details at roshni@nytpcorp.com