AI Quality and Safety Engineer

Position: Senior/Principal AI Quality and Safety Engineer Location: Onsite In Lehi (Hybrid) Duration: 10 Month Contract Must Have: • 5-8 years of experience • Extensive experience with eval frameworks for LLM quality and safety • Deep understanding of LLMs (Gemini, OpenAI, Llama, etc.) • Experience with prompt engineering and fine-tuning • Strong Python skills and experience with LangChain or similar frameworks Pluses: • Bachelor’s degree in related field • Detect and mitigate bias in datasets and model predictions. • Identify potential risks in AI systems (e.g., harmful outputs, adversarial attacks). • Implement safeguards to prevent unintended consequences. • Ensure compliance with safety standards and regulations • Continuously monitor AI systems in production for anomalies or drift. • Audit decision-making processes for transparency and explainability. • Document testing procedures and safety measures for accountability. • Work with legal and compliance teams to meet regulatory requirements. • Partner with data scientists, ML engineers, and product teams to integrate safety checks. Responsibilities: • Validate that AI models meet performance benchmarks (accuracy, precision, recall, etc.). • Ensure AI systems align with ethical principles (fairness, accountability, transparency). Day to Day: One of Insight Global’s clients is seeking an AI Quality and Safety Engineer to join their team. In this role, the engineer will review system dashboards to monitor AI models in production, evaluating them for anomalies, performance drift, and safety alerts. They will assess potential risks within AI systems, including harmful outputs or adversarial attacks, and implement safeguards to prevent unintended consequences. They will also conduct stress tests on new models to validate accuracy, fairness, and robustness, as well as perform risk assessments to identify potential failure scenarios and document mitigation strategies. Additional responsibilities include ensuring models comply with regulatory standards such as GDPR, maintaining audit logs, and updating safety documentation to support transparency. Additionally, they will work closely with developers to implement safety guardrails such as content filters or fallback mechanisms and will provide feedback to guide model improvements. Finally, they will report findings to stakeholders, highlighting critical issues and areas for improvement, while dedicating time to continuous learning to stay current with emerging AI safety frameworks and tools. Pay: $70-85/hr Based on Experience