Sr. Big Data Engineer Weehawken, NJ Strong Banking Domain Experience Is Required Only Local candidates to NJ/NY Position type: W2 contract. Our Challenge • As a Big Data Engineer, candidate will be instrumental in developing and optimizing big data technologies such as Apache Spark and Azure Databricks. • Candidate will create robust data pipelines, ensure high data quality, and collaborate closely with cross-functional teams to implement scalable cloud-native solutions on major cloud providers, predominantly Microsoft Azure. The Role Responsibilities • Develop, optimize, and maintain big data pipelines using Apache Spark, Azure Databricks, and related tools. • Write and deploy complex production systems in Scala, Java, or Python, ensuring high performance and reliability. • Leverage Terraform for automating cloud infrastructure deployment on Azure or other major cloud providers. • Design and implement CI/CD pipelines for data workflows using modern tools like GitLab, promoting automation and continuous improvement. • Apply best practices for testing, instrumentation, observability, and alerting to maintain data pipeline health and performance. • Contribute to system architecture and low-level design, ensuring modularity, scalability, and security. • Understand and implement data models, data structures, and algorithms for efficient data processing. • Use containerization technologies such as Docker and Kubernetes for development, build, and runtime environments. • Work within Agile development teams, participating in planning, daily stand-ups, and iterative releases. • Collaborate effectively in a global team, influencing key architectural decisions and sharing best practices. Requirements • 10+ Years of experience is required. • Proven experience developing big data solutions with Apache Spark and Azure Databricks. • Hands-on experience with Terraform for infrastructure as code (IaC) on major cloud platforms, ideally Azure. • Programming expertise in Scala, Java, or Python for complex production systems. • Extensive Python experience, especially in data engineering contexts. • Background in platform engineering roles on cloud platforms, with strong knowledge of Azure services. • Practical knowledge of testing frameworks, instrumentation, observability, and alerting tools. • Experience building and maintaining CI/CD pipelines with modern cloud-friendly systems like GitLab. • Deep understanding of information modelling, data structures, and algorithms. • Hands-on experience with containerization (Docker) and orchestration tools (Kubernetes). • Strong understanding of technical architecture and low-level system design. • Familiarity with Agile methodologies and best practices in software development. Preferred, But Not Required • Experience working with financial or risk management data. • Knowledge of data governance, security standards, and compliance in cloud environments. • Familiarity with additional cloud providers and multi-cloud deployment strategies. • Contributions to open-source data projects or communities