Data Annotator

Harvey Nash USA has been engaged to find a talented Relevance Evaluator for one of our enterprise clients. This is an exciting opportunity for detail-oriented professionals who enjoy evaluating search results and AI-generated responses, understanding user intent, and improving the quality of search and conversational AI experiences. Job Title: Relevance Evaluator Location: Remote (U.S.) Duration: 4-month contract with possible extension Day-to-Day Responsibilities • Evaluate the relevance of documents, images, videos, and AI-generated responses against user search queries. • Review query-document pairs and assign relevance scores based on established quality guidelines and evaluation rubrics. • Analyze user intent by understanding what the user is trying to accomplish and determine whether the presented result meets that need. • Assess AI assistant responses for accuracy, completeness, clarity, and overall usefulness. • Conduct research on unfamiliar topics or terminology to ensure accurate and consistent evaluations. • Utilize Dash Search and Dash Assistant to investigate search behavior and validate results. • Formulate effective search queries, including ChatGPT-style prompts and KQL (Kibana Query Language) searches when applicable. • Follow detailed annotation guidelines while maintaining high quality and consistency across evaluations. • Collaborate with internal teams by providing feedback that helps improve search relevance and AI response quality. Must-Have Skills • Strong English reading comprehension and written communication skills. • Experience evaluating content quality, search relevance, data annotation, or AI-generated responses. • Ability to understand user intent and objectively assess search results against defined guidelines. • Strong analytical and research skills with excellent attention to detail. • Ability to formulate effective search queries using search engines or AI tools such as ChatGPT. • Comfortable working independently while following detailed evaluation rubrics and quality standards. • Strong critical thinking and problem-solving abilities. Nice-to-Have Skills • Previous experience with search relevance labeling, data annotation, or AI model evaluation. • Familiarity with Large Language Models (LLMs) and generative AI tools. • Basic knowledge of Python scripting. • Experience with Elasticsearch or Kibana Query Language (KQL). • Experience in Quality Assurance (QA), Search Quality, Trust & Safety, or Information Retrieval. Education Bachelor's degree preferred, or an equivalent combination of education and relevant professional experience. A reasonable, good faith estimate of the minimum and maximum hourly wage for this position is $35-40/hour on W2. About Us Harvey Nash is a national, full-service talent management firm specializing in technology positions. Our company was founded with a mission to serve as the talent partner of choice for the information technology industry. Benefits will be available, and details are available below: 401(k) Plan: Our employees work hard, which is why Harvey Nash is proud to contribute to their hard-earned savings with a 401(k) retirement plan that includes a 10% company match on all deferrals. We also offer a Roth 401(k) for even more flexibility. Employees 21 years of age or older who have completed 3 months of service are eligible to participate & Health Benefits