Senior/Lead Data Engineer-Python,Spark & SQL (Only W2)

Description We are in search of a Senior Data Engineer who is adept in SQL, Python, and Spark, and ideally with an experience in AI-related tasks and API integrations for chatbot development/integration. • SQL Expertise: Essential for complex query handling and database manipulation. • Python Proficiency: Required for automation, data processing, AWS Bedrock API integration, and chatbot development workflows using Boto3 SDK • Spark Familiarity: Needed for large-scale data processing. Responsibilities • Partner closely with stakeholders to define requirements for data, data processing, reporting, and analytics. • Design and maintain scalable data models and extraction processes. • Develop and integrate chatbot functionality using AWS Bedrock foundation models, Knowledge Bases, and Agents • Implement AWS Bedrock Converse API for multi-turn conversations and tool use patterns • Build AI-driven features including conversational interfaces, intelligent automation workflows, and contextual response systems • Approach tasks with flexibility, adapting to ad-hoc requirements as needed. • Foster a continuous improvement mindset for data practices and visualizations. Requirements • A solid background in database engineering and software development. • Experience with AI Agents Design and Integration • Demonstrated ability with Python and Spark within data-intensive environments • Experience with cloud services, particularly integrating foundation models through Bedrock runtime APIs • Understanding of prompt engineering, system messages, and model parameter tuning for optimal AI responses • Eager to learn and apply new technologies in data visualization, automation, AI. Nice To Have • Experience working on Amazon Web Services (in particular using AWS Glue, AWS Athena) • Familiarity with AWS Bedrock Agents, Knowledge Bases, and RAG patterns • Prior experience building chatbots using various foundation models (Claude, Llama, Titan) available on AWS Bedrock • Knowledge of multi-modal AI applications and streaming response implementations • Understanding of AWS security best practices, IAM policies, and model access management for Bedrock.