Data Scientist

e&e is seeking a Data Scientist for a hybrid contract opportunity in Philadelphia, PA! We are seeking a highly analytical and hands-on Data Scientist to drive the evaluation, optimization, and continuous improvement of advanced AI and machine learning systems. This role is responsible for developing data-driven evaluation frameworks, measuring model performance, and ensuring AI outputs meet high standards for accuracy, reliability, and business value. The ideal candidate combines strong statistical expertise, machine learning knowledge, and experience with large language models (LLMs) to translate complex business objectives into measurable AI performance metrics while collaborating closely with engineering, product, and business stakeholders. Key Responsibilities • Design, build, and maintain evaluation frameworks for AI and multi-agent LLM pipelines, including classification, retrieval, data valuation, PII detection, segmentation, and content synthesis. • Develop, curate, and manage synthetic and real-world datasets to support comprehensive model testing across multiple business domains. • Measure and analyze AI model performance using metrics such as precision, recall, calibration, confidence intervals, drift detection, and agreement with human-validated data. • Design and evaluate prompt engineering experiments, agent workflows, and structured output schemas using statistically sound methodologies. • Improve vector search and retrieval performance by evaluating embedding models, taxonomy design, and retrieval metrics such as Recall@K and Mean Reciprocal Rank (MRR). • Analyze production AI outputs to identify issues such as hallucinations, classification errors, and missed sensitive data, then develop experiments to improve model performance. • Collaborate with engineering, product management, and subject matter experts to define measurable success criteria for AI initiatives. • Develop dashboards, reports, and executive-level presentations that communicate model performance, experiment results, and recommendations. • Support continuous improvement of AI systems by monitoring model quality, identifying trends, and recommending data-driven enhancements. • Ensure evaluation processes are repeatable, scalable, and integrated into ongoing development and deployment workflows. Required Qualifications • Bachelor's degree in Computer Science, Data Science, Statistics, Machine Learning, or a related quantitative field (Master's or PhD preferred). • 3+ years of professional experience in Data Science, Machine Learning, AI Engineering, or AI model evaluation. • Strong programming skills in Python, including experience with Pandas, NumPy, scikit-learn, PyTorch, or TensorFlow. • Experience evaluating large language model (LLM) applications, including prompt engineering, structured output validation, retrieval evaluation, and hallucination detection. • Strong foundation in statistical analysis, including hypothesis testing, confidence intervals, power analysis, calibration, and experimental design. • Proficiency writing SQL queries and performing data analysis using relational databases such as PostgreSQL. • Experience working with vector databases, embedding models, and semantic search technologies. • Strong analytical, problem-solving, and communication skills with the ability to translate technical findings into business insights. • Experience writing production-quality code and collaborating within software development and CI/CD environments. Preferred Qualifications • Experience with multi-agent AI frameworks such as LangGraph, LangChain, CrewAI, AutoGen, or similar platforms. • Familiarity with workflow orchestration tools such as Airflow, Temporal, or equivalent. • Experience working with modern LLM APIs, tool calling, structured outputs, and prompt optimization. • Knowledge of synthetic data generation and AI evaluation dataset development. • Experience with experiment tracking platforms such as MLflow, Weights & Biases, or LangSmith. • Background in NLP, financial data analysis, data valuation, or PII/PHI detection. • Familiarity with Go, TypeScript, or similar programming languages used within AI application development.