Databricks Data Engineer

This is a great opportunity to join one of the largest integrated energy and commodity trading companies in the world. We are seeking a highly skilled and experienced Data Architect with strong expertise in Databricks and Python development. The ideal candidate will have a deep understanding of modern data architecture principles and experience with data migration from traditional RDBMS systems to cloud-based platforms. You will work closely with end-users to understand business processes, gather requirements, and design solutions that deliver tangible value. Key Responsibilities • Design, develop, and implement robust data architecture solutions utilizing modern cloud data platforms like Databricks. • Ensure scalable, reliable, and secure data environments that support advanced analytics and meet business requirements. • Lead the migration of data from traditional RDBMS systems to Databricks environments. • Architect and design scalable data pipelines and infrastructure to support the organization’s data needs. • Develop and manage ETL processes using Databricks, ensuring efficient data extraction, transformation, and loading. • Optimize ETL workflows for performance and maintain data integrity. • Ensure seamless data transitions with minimal disruption to operations. • Monitor and tune performance of data systems to ensure scalability, reliability, and cost-effectiveness. • Collaborate with cross-functional teams including data engineers, data scientists, analysts, and product managers. • Define and enforce best practices and standards for data engineering and data warehouse processes. • Evaluate and implement new technologies and tools to improve the efficiency and reliability of data pipelines. • Provide technical leadership and mentorship to junior data team members. • Partner with DevOps and infrastructure teams to deploy and manage data systems in production. • Ensure compliance with data governance, security, and regulatory standards. • Align data architecture with enterprise architecture and IT strategy. Skills and Experience • BS or MS in Computer Science or a related field. • Extensive experience with Databricks, including ETL processes and cloud-based data architecture. • Strong proficiency in Python development, including scripting and building data pipelines. • Experience with cloud platforms such as AWS. • Strong background in data warehousing, data modeling, and advanced SQL. • Experience with traditional RDBMS systems and transitioning data to cloud-native platforms. • Knowledge of data governance, data quality management, and security principles. • Familiarity with containerization (e.g., Docker) and orchestration tools (e.g., Kubernetes). • Excellent analytical, problem-solving, and communication skills. • Ability to work effectively across geographically distributed, cross-functional teams.