Databricks Data Engineer

Software Engineering III – Databricks Data Engineer (Remote | U.S. Citizenship Required) Remote – quarterly travel to Gaithersburg, MD We are seeking a highly skilled Software Engineering III – Databricks Data Engineer to join our growing team and play a key role in building scalable, secure, and high-performing data solutions within an enterprise Microsoft Azure cloud environment. This role is ideal for a hands-on Data Engineer with strong expertise in Databricks, Apache Spark, Azure data services, and modern data lake architecture. You will be responsible for designing, developing, and optimizing enterprise-grade data pipelines, supporting advanced analytics and AI-driven workflows, and collaborating with both technical and business stakeholders to deliver impactful data solutions. If you enjoy solving complex data engineering challenges, working with large-scale datasets, and contributing to mission-critical cloud analytics platforms, this is a great opportunity to make a meaningful impact. Key Responsibilitie • sDesign, develop, and optimize scalable data pipelines and analytics solutions using Databricks and Medallion Architectur • eBuild and maintain data ingestion pipelines for structured, streaming, and unstructured dat • aManage and support ingestion routines for multi-terabyte datasets across multiple Databricks workspace • sTranslate business and technical requirements into robust Azure-based data engineering solution • sIntegrate data from multiple enterprise sources to enable high-quality analytics, reporting, and insight • sMaintain and enhance ETL/ELT processes with a strong focus on data quality, reliability, and performanc • eMonitor platform health, troubleshoot production issues, and optimize Spark workloads, cluster performance, and platform stabilit • yImplement data management strategies to ensure data integrity, availability, accessibility, and governanc • eIdentify and drive cost optimization opportunities across data storage, compute, and analytics workload • sIntegrate and work with key Azure services, including • :Azure Function • sAzure Storage Service • sAzure Data Factor • yAzure Log Analytic • sAzure User Management / Access Control • sProvision and manage cloud infrastructure using Infrastructure-as-Code (IaC • )Apply best practices for data security, compliance, governance, and federal/Public Trust standard • sPartner with cross-functional stakeholders, including business users, architects, analysts, and engineering team • sSupport AI/ML and advanced analytics workflows by enabling reliable and scalable data foundation • sProvide client-facing support, technical consultation, and clear solution documentatio • nCreate and maintain architecture diagrams, interface diagrams, and technical documentatio • nMentor and review the work of junior technical team members as neede d Required Technical Skil • lsStrong hands-on experience with Databricks platform development, administration, and maintenan • ceExpertise in Apache Spa • rkProficiency in Python and Spark (R experience is a plu • s)Strong experience with Azure cloud services, includin • g:Azure Functio • nsAzure Storage Servic • esAzure Data Facto • ryAzure Log Analyti • csUser / Identity Manageme • ntExperience building data pipelines for structured, streaming, and unstructured da • taStrong understanding of ETL/ELT development, orchestration, and data quality manageme • ntDeep knowledge of Medallion Architecture and modern data lake / lakehouse design patter • nsExperience with Infrastructure-as-Code (Ia • C)Knowledge of data governance, includin • g:Metadata manageme • ntEnterprise data catal • ogData quali • tyData security / access contro • lsExperience with CI/CD automation for data engineering workflo • wsFamiliarity with Agile / Scrum methodologi • esExposure to cloud platforms such as Azure (required) and AWS (preferre d) Preferred / Nice-to-Have Ski • llsExperience with .NET developm • entAzure cloud certificati • onsUnderstanding of FinOps / cloud cost management princip • lesExposure to AI/ML platforms, advanced analytics, or modern AI technolog ies Qualifications & Experi • enceBachelor’s degree in Computer Science, Engineering, or related field with 3+ years of relevant experie • nc • e,ORMaster’s degree in a related field with 2+ years of relevant experi • enceDatabricks Data Engineer Certification strongly preferred / requ • ired2+ years of hands-on experience supporting and maintaining the Databricks platform and Spark-based develop • ment3+ years of experience designing and developing cloud-based data ingestion and processing pipel • inesProven experience working directly with clients / end users and providing front-line technical sup • portStrong ability to create and maintain technical architecture and interface documenta tion Work Authorization & Clearance Require • mentsU.S. Citizenship is req • uiredMust be eligible to obtain or currently hold a Public Trust clea rance Work Location / • TravelRemote po • sitionMust be comfortable with quarterly travel to Gaithersbu • rg, MDCandidates located in the Greater DC / Maryland / Virginia area are highly pre ferred