Cloud Data Engineer Databricks, Snowflake & Azure | Contract | New York, NY (Onsite)

Dice is the leading career destination for tech experts at every stage of their careers. Our client, Anagha Techno Soft, is seeking the following. Apply via Dice today! Cloud Data Engineer Databricks, Snowflake & Azure Location: New York, NY (Onsite) Job Type: Contract Position Overview We are seeking an experienced Cloud Data Engineer with strong expertise in Scala, Spark, Databricks, Snowflake, and Azure to support large-scale data modernization initiatives. This role involves designing, developing, testing, and optimizing enterprise data pipelines while ensuring data quality, scalability, and operational excellence across cloud-based platforms. The ideal candidate will have hands-on experience migrating legacy ETL workloads to modern cloud architectures and implementing robust data engineering best practices. Key Responsibilities • Design, develop, and optimize scalable data pipelines using Scala and Apache Spark. • Build and maintain data solutions on Databricks leveraging Serverless Compute, Unity Catalog, Databricks CLI, and Asset Bundles. • Analyze, refactor, and translate complex SQL-based data transformation logic into modern cloud-native architectures. • Develop and optimize Snowflake data models, schemas, and integrations using the Spark-Snowflake Connector. • Implement data ingestion, transformation, and orchestration workflows using Azure services and Apache Airflow. • Create automated testing frameworks, including unit, integration, and behavior-driven testing approaches. • Develop data quality controls, reconciliation processes, and automated validation mechanisms. • Collaborate with cross-functional teams to modernize legacy ETL environments and execute migration strategies. • Implement CI/CD pipelines for code deployment, testing automation, and release management. • Ensure adherence to data governance, lineage, observability, and security best practices. Required Skills & Qualifications Data Engineering • Strong hands-on experience with Scala and Apache Spark in production environments. • Expertise with Spark DataFrame APIs, joins, window functions, partitioning strategies, and performance tuning. • Advanced SQL development and optimization skills. Databricks & Snowflake • Experience with Azure Databricks, including Serverless Compute, Unity Catalog, Asset Bundles, and Databricks CLI. • Strong understanding of Snowflake architecture, schema design, query optimization, and connector integrations. Cloud & Orchestration • Experience working with Azure services including ADLS, identity management, and cloud security fundamentals. • Hands-on experience developing and maintaining Apache Airflow DAGs, sensors, retries, and SLA monitoring. DevOps & Testing • Experience with Git-based development workflows and CI/CD automation. • Familiarity with artifact management and automated deployment pipelines. • Strong background in Test-Driven Development (TDD), unit testing, and behavior-driven testing methodologies. Data Quality & Migration • Experience building automated data validation, reconciliation, and drift-detection frameworks. • Proven success migrating legacy ETL platforms to modern cloud-based data ecosystems. • Strong understanding of Medallion Architecture, schema evolution, lineage, observability, and idempotent pipeline design. Preferred Qualifications • Financial Services or Regulatory Reporting domain experience. • Python development experience for data engineering utilities and automation. • Experience with specification-driven development methodologies. • Knowledge of Gradle and JVM ecosystem tooling. Why Join? • Work on enterprise-scale cloud modernization initiatives. • Utilize modern technologies including Databricks, Snowflake, Azure, Scala, and Spark. • Collaborate with highly skilled engineering teams on complex data transformation projects. • Opportunity to contribute to large-scale migration and cloud adoption programs.