Audio Generalist Evaluator Expert at Mercor

This description is a summary of our understanding of the job description. Click on 'Apply' button to find out more.

Role Description

Mercor is seeking detail-oriented writing experts to contribute to a high-impact audio AI research project with a leading lab. Freelancers will author prompt–golden answer pairs that train and evaluate advanced language models. This is a short-term, flexible opportunity for professionals with strong academic backgrounds and a knack for instructional clarity. Ideal for those who enjoy distilling complex concepts into well-crafted text.

• Design and Optimize Prompts : Create detailed audio prompts with multiple constraints and instructions.
• Define and Document Evaluation Standards : Establish high-level expectations for correct responses in general audio consumer contexts, and develop comprehensive rubric.
• Conduct Model Testing and Grading : Run prompts through models and assess preliminary outputs against expectations.
• Support Benchmarking and Quality Assurance : Collaborate in QA review processes to ensure prompt tasks and rubrics meet rigor, maintaining consistency and reliability before integration into official benchmarks.

Qualifications

• BS or BA from a reputable institution completed or in progress
• Strong writing and critical thinking skills
• Ability to work independently and meet deadlines
• Significant familiarity with ChatGPT or similar tools for personal decision-making or hobbies/general interests

Requirements

• 2+ years of experience in teaching or research

Application & Onboarding Process

• Complete an AI-led interview, this should take around 15 minutes
• Complete a 45-minute written assessment that will guide you through writing rubrics
• If selected, you will be invited to work on the project

More Details About This Role

• This is a remote and asynchronous role — work on your own schedule
• Expect to contribute at least 20 hours per week
• Expect a commitment of around 1 month
• You’ll be working in a structured project environment with clear goals and tools

Company Description

• Our team is based in San Francisco, CA
• We specialize in recruiting experts for top AI labs
• Our investors include Benchmark, General Catalyst, Adam D’Angelo, Larry Summers, and Jack Dorsey