Scala/Data Bricks Engineer

We’re partnering with a fast-growing technology company in New York that is scaling its data engineering and analytics platforms. They are seeking a highly skilled Scala/Databricks Engineer on a contract basis to design, optimize, and maintain large-scale data pipelines that power mission-critical insights across the business. This role sits within the company’s core Data Engineering team and will be central to modernizing their big data ecosystem, building production-grade pipelines, and enabling advanced analytics and machine learning use cases. Key Responsibilities • Design and build scalable, distributed data pipelines in Databricks (Spark) using Scala. • Develop and optimize ETL/ELT workflows for structured and unstructured data sources. • Implement best practices in data modeling, partitioning, and performance tuning. • Collaborate with Data Science and Analytics teams to productionize ML pipelines. • Work with cloud-native data platforms (Azure, AWS, or GCP) to deploy and monitor workloads. • Ensure data quality, governance, and compliance across the pipeline ecosystem. • Contribute to CI/CD automation for data engineering workflows. • Troubleshoot and optimize Spark jobs to improve efficiency and reduce cost. Required Skills & Experience • 5+ years of professional experience in Scala development, with a strong background in functional programming. • 3+ years of hands-on experience with Apache Spark (preferably in Databricks). • Strong expertise in building and tuning large-scale ETL pipelines. • Experience with cloud data platforms such as Azure Data Lake, AWS S3, or GCP BigQuery. • Solid knowledge of SQL and distributed query engines (e.g., Hive, Presto, Delta Lake). • Familiarity with ML pipeline integration and working alongside Data Science teams. • Strong understanding of CI/CD tools (Jenkins, GitHub Actions, or Azure DevOps). • Excellent problem-solving skills, with the ability to work independently and in fast-paced environments. Preferred Skills • Experience with Delta Lake and Databricks MLflow. • Knowledge of Python for data engineering tasks. • Background in financial services, fintech, or large-scale SaaS data environments. • Familiarity with streaming frameworks (Kafka, Structured Streaming).