Business Intelligence Analyst

Job Description Responsibilities 1. Deterministic Testing & Data Validation • Validate generative AI tool outputs for structured, rules-based use cases by reconciling results against trusted data sources and established SQL-based metrics. • Ensure consistency, explainability, and auditability of outputs by confirming alignment with existing data pipelines and query logic • Expand and maintain test coverage across prioritized use cases to establish a robust, high-confidence baseline for the platform • Partner with data engineering and analytics teams to identify and resolve discrepancies in underlying data or logic 1. Non-Deterministic Testing & Scenario Evaluation • Design and execute scenario-based testing for more complex, AI-driven outputs where direct validation is not always possible • Evaluate results based on intent accuracy, reasonableness, and confidence thresholds rather than exact match validation • Prioritize testing across higher-risk and high-impact use cases using curated question sets and real-world scenarios • Identify patterns in output variability and drive iterative refinement to improve reliability and user trust 1. Human-in-the-Loop Review & Continuous Monitoring • Conduct ongoing review of generative AI tool interactions post-launch, validating outputs and ensuring quality across all user scenarios • Identify edge cases, inconsistencies, and emerging risks, and escalate findings to product and engineering teams • Synthesize insights from testing and live usage to inform enhancements, training data improvements, and governance practices • Serve as an accountable reviewer, providing a critical control point for responsible AI deployment and continuous improvement Qualifications Required Skills & Experience • Robust SQL skills required. • Robust analytical background with experience in data validation, SQL, and analytics workflows • Ability to assess outputs both quantitatively (data accuracy) and qualitatively (reasonableness, business context) • Demonstrated critical thinking and sound judgment, especially in ambiguous or non-deterministic environments • Experience working with large datasets, reporting tools, or analytics platforms Preferred Qualifications • Exposure to AI/ML or generative AI tools and associated testing or validation frameworks • Experience in scenario-based testing, UAT, or model validation • Familiarity with financial services, retirement, or plan sponsor analytics Required Skills: SQL, AI