Data Engineer

Sorry we are not considering any candidates that require sponsorship or transfer of any kind as the client needs to be able to convert this person without any kind of transfer or sponsorship of any visa. Local candidates only. I am not interested in working with any third party agencies please do not respond or send me your candidates in response to my posting I will simply delete you; only working with direct candidates. Data Engineer to help build and scale its modern cloud data platform. This role will be responsible for designing and developing data pipelines, integrating internal and external data sources, and building scalable cloud solutions using Azure Databricks, Amazon Web Services (AWS), and Python. The ideal candidate is passionate about cloud technologies and data engineering, enjoys building applications and automation tools, and has experience creating reliable, high-performance data solutions that support analytics and business intelligence. This position will work closely with Data Scientists, Business Analysts, and business stakeholders to transform raw data into trusted, actionable insights. Key Responsibilities Data Engineering • Design, develop, and maintain scalable ETL/ELT pipelines using Azure Databricks and PySpark. • Build robust data integration solutions for structured and unstructured data sources. • Design and optimize data models to support reporting, analytics, and machine learning. • Monitor, troubleshoot, and optimize data pipelines for performance, reliability, and scalability. • Implement data quality checks and governance best practices. Cloud Engineering (AWS) • Design and maintain cloud-based data storage solutions using Amazon S3. • Configure secure AWS environments for data ingestion from internal and third-party data providers. • Build automated processes for ingesting files into S3 and orchestrating downstream processing. • Work with AWS services including: • Amazon S3 • AWS Glue • Lambda • IAM • CloudWatch • Support future cloud-native initiatives utilizing AWS container technologies such as ECS, EKS, or Kubernetes. Application Development • Develop internal applications, automation tools, APIs, and utilities using Python. • Build reusable services that streamline business processes and improve operational efficiency. • Collaborate with cross-functional teams to deliver scalable, data-driven solutions. Collaboration & Analytics • Partner with Sales, Marketing, Finance, Supply Chain, and Operations teams to understand business requirements. • Support Data Scientists by preparing high-quality datasets for advanced analytics and AI initiatives. • Work closely with IT and business stakeholders to implement enterprise data solutions. Required Qualifications • Bachelor's degree in Computer Science, Information Systems, Data Engineering, Software Engineering, or a related field. • 3+ years of experience in Data Engineering or a similar role. • Hands-on experience with Azure Databricks. • Experience working within Amazon Web Services (AWS). • Strong programming skills in Python. • Advanced SQL development skills. • Experience building scalable ETL/ELT pipelines. • Experience with Git and source control. • Strong understanding of relational databases and data warehousing concepts. Preferred Qualifications • Experience working with syndicated market data providers such as NielsenIQ, Circana (IRI), SPINS, or retailer POS data. • Experience with: • PySpark • Spark SQL • Delta Lake • Azure Data Factory • Snowflake • Amazon Athena • Amazon Redshift • Experience with Docker, Kubernetes, or containerized applications in AWS. • Familiarity with CI/CD pipelines and DevOps practices.