LLM Evaluation Engineer

LLM Evaluation Engineer Location: Remote Duration: 12+ Months Required Skills • Strong understanding of LLMs and generative AI concepts, including model behavior and output evaluation • Experience with AI evaluation and benchmarking methodologies, including baseline creation and model comparison • Hands-on expertise in Eval testing, creating structured test suites to measure accuracy, relevance, safety, and performance • Ability to define and apply evaluation metrics (precisionrecall, BLEUROUGE, F1, hallucination rate, latency, cost per output)Prompt engineering and prompt testing experience across zero-shot, few-shot, and system prompt scenarios • Python other programming languages, for automation, data analysis, batch evaluation execution, and API integration • Experience with evaluation tools/frameworks (OpenAI Evals, HuggingFace evals, Promptfoo, Ragas, DeepEval, LM Eval Harness) • Ability to create datasets, test cases, benchmarks, and ground truth references for consistent scoring • Test design and test automation experience, including reproducible evaluation pipelines • Knowledge of AI safety, bias, security testing, and hallucination analysis