Data Analyst

The AI Business Data Analyst will support the Customer Experience digital transformation by providing day-to-day analytics quality monitoring KPI definition evaluation and insight generation for production generative AI and conversational AI applications. This role will help ensure AI-powered customer experiences are accurate reliable measurable and continuously improving. The analyst will monitor production LLM application performance evaluate response quality build dashboards identify risks and opportunities and synthesize insights that inform product roadmap decisions feature requirements backlog prioritization and operational improvements. The ideal candidate combines strong analytical skills product sense technical curiosity and practical understanding of generative AI applications including conversational AI RAG prompt-based systems AI evaluation metrics and LLM-as-a-judge methodologies. 1. Key Accountabilities AI Product Performance Evaluation Insights - Establish monitor and report on KPIs related to AI product performance customer experience user behavior operational efficiency and business value. - Analyze production AI application data including conversation logs usage patterns customer feedback quality signals escalation trends and operational telemetry. - Identify issues and opportunities related to response quality accuracy hallucination risk retrieval relevance content drift deflection satisfaction coverage and task completion. - Design and support AI evaluation processes including response quality reviews evaluation rubrics benchmark test sets human-in-the-loop review and LLM-as-a-judge approaches. - Build dashboards scorecards and recurring business reviews that communicate AI application health trends risks and improvement opportunities. - Translate findings into actionable recommendations product requirements backlog items acceptance criteria and release validation plans. Product Analytics Discovery Roadmap Support - Conduct exploratory analysis to understand user behavior customer intent conversation topics feature usage and AI-assisted customer journeys. - Identify friction points emerging needs product gaps and opportunities to improve AI-powered customer experiences. - Partner with Product Managers and CX stakeholders to translate data-driven insights into roadmap recommendations feature concepts and experimentation priorities. - Support product discovery through analysis of customer feedback behavioral data qualitative research competitive trends and emerging AI capabilities. - Design and evaluate experiments or A/B tests related to prompts workflows retrieval strategies response patterns and feature changes. Agile Product Delivery Cross-Functional Partnership - Lead or participate in the gathering analysis and documentation of business requirements user stories acceptance criteria use cases process flows and technical dependencies for AI product enhancements. - Participate in Agile ceremonies including backlog refinement sprint planning daily syncs demos sprint reviews and retrospectives. - Support UAT release validation and post-release monitoring for prompt updates indexed content changes retrieval logic changes model updates workflow enhancements and dashboard/reporting releases. - Partner with Product Owners AI engineers data teams QA CX stakeholders vendors and business leaders to ensure AI products meet customer operational and business requirements. - Translate technical AI behavior and data findings into clear business implications for technical and non-technical stakeholders. - Support change management user support training materials and stakeholder communications related to AI product performance and enhancements. 1. Qualifications Experience Skills Minimum Educational Qualifications - BA/BS degree in Business Economics Marketing Analytics Statistics Information Science Computer Science Cognitive Science Data Science or a related field or equivalent professional experience supported by relevant subject matter training. Minimum Experience - 3 years of experience in business analysis product analytics data analytics product operations AI operations or related analytical roles. - Experience supporting digital products customer-facing applications conversational AI AI agents chatbots or generative AI systems. - Experience defining KPIs measuring product performance building dashboards and communicating trends risks and opportunities to stakeholders. - Experience analyzing user behavior usage data conversation logs customer journeys operational metrics or product performance data. - Good understanding of generative AI applications including LLMs prompt-based systems conversational AI AI agents and retrieval-augmented generation. - Familiarity with AI evaluation concepts such as groundedness relevance factuality hallucination risk retrieval quality response accuracy human evaluation and LLM-as-a-judge methods. Knowledge of RAGAS or similar frameworks helpful. - Experience with or knowledge of LLM observability platforms e.g. LangSmith Arize etc. beneficial but not required. Technical Tool Skills - Experience with SQL or similar query languages for data extraction and analysis. Knowledge of Python and Pandas helpful. - Strong Excel skills including creation of automations. - Experience with Power BI Tableau Adobe Analytics or similar analytics and visualization tools. - Experience working with content metadata taxonomies or categorical data. - Experience building dashboards recurring reports operational scorecards and performance monitoring views. - Familiarity with AWS technologies beneficial especially DynamoDB and CloudWatch. Desired Traits - Strong data analytical and technical aptitude. - Excellent product sense and ability to connect user behavior AI system performance and business outcomes. - Solid understanding of which data is needed to understand how the product is used and uses data to understand how customers use the product. - Strong at independently identifying insights from data and the UX to refine and innovate the product makes recommendations to Product Owner based on data. - Rigorous attention to detail and strong judgment around quality customer experience and brand risk. - Curiosity about why AI systems behave the way they do and persistence in diagnosing root causes. - Ability to tell clear stories with data and make complex AI performance topics accessible to non-technical audiences. - Comfortable working in ambiguity and operating in a live evolving production environment. - Self-motivated execution-oriented and able to manage multiple stakeholders and competing priorities. The pay range for this position is 42-44/hr. Benefits available to contract/temporary professionals, include medical, vision, dental, and life and disability insurance. Hired contract/temporary professionals are also eligible to enroll in our company 401(k) plan. Visit roberthalf.gobenefits.net for more information. Our specialized recruiting professionals apply their expertise and utilize our proprietary AI to find you great job matches faster. Los Angeles, CA postings only – add this sentence (in addition to the verbiage above): Robert Half will consider for employment qualified applicants with arrest or conviction records in accordance with the Los Angeles County Fair Chance Ordinance for Employers and the California Fair Chance Act.