Databricks Data Engineer

Job Title: Databricks Data Engineer Location : Los Angeles CA or New York, NY (Hybrid) Contract / Fulltime Job Summar yWe are looking for an experienced Databricks Data Engineer with strong DevOps expertise to join our data engineering team. The ideal candidate will design, build, and optimize large-scale pipelines on the Databricks Lakehouse Platform on AWS, while driving automated CI/CD and deployment practices. This role requires strong skills in PySpark, SQL, AWS cloud services, and modern DevOps tooling. You will collaborate closely with cross-functional teams to deliver scalable, secure, and high-performance data solutions . Must Demonstrate (Critical Skills & Architectural Competencie • s)Designing and implementing Databricks-based Lakehouse architectures on A • WSClear separation of compute vs. serving laye • rsAbility to design low-latency data/API access strategies (beyond Spark-only pattern • s)Strong understanding of caching strategies for performance and cost optimizati • onData partitioning, storage optimization, and file layout strate • gyAbility to handle multi-terabyte structured or time-series datase • tsSkill in requirement probing, identifying what matters architectural • lyA player-coach mindset: hands-on engineering + technical leadersh ip Key Responsibilit ies1. Data Pipeline Developm • entDesign, build, and maintain scalable ETL/ELT pipelines using Databricks on A • WS.Develop high-performance data processing workflows using PySpark/Spark and S • QL.Integrate data from Amazon S3, relational databases, and semi/non‑structured sourc • es.Implement Delta Lake best practices including schema evolution, ACID, OPTIMIZE, ZORDER, partitioning, and file-size tuni • ng.Ensure architectures support high-volume, multi-terabyte workloa ds. 1. DevOps & C • I/CDImplement CI/CD pipelines for Databricks using Git, GitLab, GitHub Actions, or AWS-native to • ols.Build and manage automated deployments using Databricks Asset Bund • les.Manage version control for notebooks, workflows, libraries, and environment configurat • ion.Automate cluster policies, job creation, environment provisioning, and configuration managem • ent.Support infrastructure-as-code via Terraform (preferred) or CloudFormat ion. 1. Collaboration & Business Su • pportWork with data analysts and BI teams to prepare curated datasets for reporting and analy • tics.Collaborate closely with product owners, engineering teams, and business partners to translate requirements into scalable implementat • ions.Document data flows, technical architecture, and DevOps/deployment workf lows. 1. Performance & Optimi • zationTune Spark clusters, workflows, and queries for cost efficiency and compute perfor • mance.Monitor pipelines, troubleshoot failures, and maintain high reliab • ility.Implement logging, monitoring, and observability across workflows and • jobs.Apply caching strategies and workload optimization techniques to support low-latency consumption pat terns. 1. Governance & S • ecurityImplement and maintain data governance using Unity C • atalog.Enforce access controls, security policies, and data compliance requir • ements.Ensure lineage, quality checks, and auditability across data flows. Technica • l SkillsStrong hands-on experience with Databricks, in • cluding:De • lta LakeUnity • CatalogLakehouse Arch • itectureDelta Live P • ipelinesDatabricks • RuntimeTable • TriggersDatabricks W • orkflowsProficiency in PySpark, Spark, and advan • ced SQL.Expertise with AWS cloud services, in • cl • udi • ng:S3IAMGlue / Glue • Catal • ogLambdaKinesis (optional but ben • eficial)Secrets • ManagerStrong understanding of DevOp • s tools:Git • / GitLabCI/CD p • ipelinesDatabricks Asset • BundlesFamiliarity with Terraform is • a plus.Experience with relational databases and data warehouse c oncepts. Preferred E • xperienceKnowledge of streaming technologies like Structured Streaming/Spark S • treaming.Experience building real-time or near real-time p • ipelines.Exposure to advanced Databricks runtime configurations and performanc e tuning. Certifications • (Optional)Databricks Certified Data Engineer Associate / Pr • ofessionalAWS Data Engineer or AWS Solutions Architect cer tification