Prompt Response Evaluators

Role: Prompt Response Evaluators – Coding (Multiple Programming Languages) Location: Remote (USA) Job Type: Contingent (Hourly) – 3 Months About the Role: We are initiating a large-scale evaluation project that involves assessing and annotating coding prompts across 10 programming languages. The work includes reviewing model-generated responses, validating code correctness, evaluating logic, and ensuring adherence to required technical standards. This role requires strong coding knowledge, analytical skills, and high attention to detail. Assessment Process: Candidates will undergo a two-level evaluation as part of the shortlisting process: 1. Level 1 – Coding Assessment: Evaluates candidates’ technical coding proficiency, problem-solving skills, and accuracy in their primary programming language. 1. Level 2 – Guidelines & Attention-to-Detail Assessment: Candidates who clear Level 1 will complete a second assessment focused on guideline comprehension, analytical ability, and attention to detail required for evaluating model-generated outputs. Both levels must be successfully completed to qualify for the project. Project Requirements (Programming Languages & Expected ICs): Programming Language No. of ICs Required Python 10 Java 10 JavaScript & TypeScript (combined) 10 C/C++ 10 Rust 5 PHP 5 HTML/CSS 5 SQL 5 Bash/Shell 5 Total ~65 (subject to change) Priority Distribution Guidance: High-priority languages account for 80% of annotations: Python, Java, JavaScript, C/C++, HTML/CSS, Bash/Shell. Among these, Python, Java, JavaScript, and C/C++ should contribute equally to the 80% split. Medium-priority languages (20%): TypeScript, Rust, PHP, SQL. Preferred Qualifications: · Education: Technical degree preferred . · Experience: Minimum 3+ years of relevant experience. · Skills: o Strong coding expertise in the required tech stack o SDE-level technical understanding o Excellent analytical skills and attention to detail · Tools: SRT/Gala Tool · Mode: Remote Additional details like AHT, total work hours, and weekday/weekend support will be shared as soon as they are finalized.