Evaluation Scenario Writer - AI Agent Testing SpecialistMindrift • KY, US
Evaluation Scenario Writer - AI Agent Testing Specialist
Mindrift • KY, US
job_description.job_card.30_days_ago
serp_jobs.job_preview.job_type
- serp_jobs.job_card.part_time
- serp_jobs.filters.remote
- serp_jobs.filters_job_card.quick_apply
job_description.job_card.job_descriptionBachelor's and / or Master’s Degreein Computer Science, Software Engineering, Data Science / Data Analytics, Artificial Intelligence / Machine Learning, Computational Linguistics / Natural Language Processing (NLP), Information Systems or other related fields. Background in QA, software testing, data analysis, or NLP annotation. Good understanding of test design principles (e.g., reproducibility, coverage, edge cases). Strong written communication skills in English. Comfortable with structured formats like JSON / YAML for scenario description. Can define expected agent behaviors (gold paths) and scoring logic. Basic experience with Python and JS. Curious and open to working with AI-generated content, agent logs, and prompt-based behavior. You are ready to learn new methods, able to switch between tasks and topics quickly and sometimes work with challenging, complex guidelines. Our freelance role is fully remote so, you just need a laptop, internet connection, time available and enthusiasm to take on a challenge. Experience in writing manual or automated test cases. Familiarity with LLM capabilities and typical failure modes. Understanding of scoring metrics (precision, recall, coverage, reward functions). Get paid for your expertise, with rates that can go up to $60 / hour depending on your skills, experience, and project needs. Take part in a flexible, remote, freelance project that fits around your primary professional or academic commitments. Participate in an advanced AI project and gain valuable experience to enhance your portfolio. Influence how future AI models understand and communicate in your field of expertise.
This opportunity is only for candidates currently residing in the specified country. Your location may affect eligibility and rates. Please submit your resume in English and indicate your level of English.
At Mindrift , innovation meets opportunity. We believe in using the power of collective human intelligence to ethically shape the future of AI.
What we do
The Mindrift platform, launched and powered by Toloka , connects domain experts with cutting-edge AI projects from innovative tech clients. Our mission is to unlock the potential of GenAI by tapping into real-world expertise from across the globe.
About the Role
We’re looking for someone who can design realistic and structured evaluation scenarios for LLM-based agents. You’ll create test cases that simulate human-performed tasks and define gold-standard behavior to compare agent actions against. You’ll work to ensure each scenario is clearly defined, well-scored, and easy to execute and reuse. You’ll need a sharp analytical mindset, attention to detail, and an interest in how AI agents make decisions.
Although every project is unique, you might typically :
- Designing structured test scenarios based on real-world tasks.
- Defining the golden path and acceptable agent behavior.
- Annotating task steps, expected outputs, and edge cases.
- Working with devs to test your scenarios and improve clarity.
- Reviewing agent outputs and adapting tests accordingly
How to get started
Simply apply to this post, qualify, and get the chance to contribute to projects aligned with your skills, on your own schedule. From creating training prompts to refining model responses, you’ll help shape the future of AI while ensuring technology benefits everyone.
Requirements
Nice to Have
Benefits
Contribute on your own schedule, from anywhere in the world. This opportunity allows you to :
serp_jobs.job_alerts.create_a_job
Evaluation Testing • KY, US
Job_description.internal_linking.related_jobs
Earn up to $15 / hour + performance bonuses.Outlier, a platform owned and operated by Scale AI, is looking for.If you're passionate about improving models and excited by the future of AI, this is you...serp_jobs.internal_linking.show_more
Now Hiring : Allied Healthcare Speech Therapist - Cynthiana, KY.Contact us for Pay Information.Actual amount dependent upon market adjustments.
Are you a passionate Allied Healthcare professional loo...serp_jobs.internal_linking.show_more
The Process Engineer will assist in feasibility studies and implementing process improvements and will create necessary documentation and aides in communication to personnel.This role will be a ful...serp_jobs.internal_linking.show_more
Product Testers are wanted to work from home nationwide in the US to fulfill upcoming contracts with national and international companies.
We guarantee 15-25 hours per week with an hourly pay of bet...serp_jobs.internal_linking.show_more
AlliedTravelCareers is working with Prime Time Healthcare to find a qualified Physical Therapist (PT) in Paris, Kentucky, 40361!.
Now Hiring : Allied Healthcare Physical Therapist - Paris, KY.Contact...serp_jobs.internal_linking.show_more
DoorDash is the #1 category leader in food delivery, food pickup, and convenience store delivery in the US, trusted by millions of customers every day.
As a Dasher, you’ll stay busy with a var...serp_jobs.internal_linking.show_more
What is Uber? Driving with Uber is an alternative to a part-time or full-time job and can get you earning cash on the road quickly.
Driving with Uber allows you to earn quick cash while maintaining ...serp_jobs.internal_linking.show_more
Posting Title : PRN X-Ray Technologist Overview : .At Fast Pace Urgent Care, X-Ray Techs provide clinical and administrative support essential for effective patient care under the direct supervision o...serp_jobs.internal_linking.show_more
AlliedTravelNetwork is working with Prime Time Healthcare to find a qualified Physical Therapist (PT) in Paris, Kentucky, 40361!.
Now Hiring : Allied Healthcare Physical Therapist - Paris, KY.Contact...serp_jobs.internal_linking.show_more
Physical Therapist for a travel job in Richmond, Kentucky.Job Description & Requirements.CompHealth exists to make the locums process easier.
Not only will we search for jobs that fit your interests...serp_jobs.internal_linking.show_more This opportunity is only for candidates currently residing in the specified country.Your location may affect eligibility and rates.
Please submit your resume in English and indicate your level of En...serp_jobs.internal_linking.show_more 
TravelNurseSource is working with Ardor Health Solutions to find a qualified Home Health RN in Richmond, Kentucky, 40475!.
Access to 24 / 7 telephone support during your contract assignment!.Ardor Hea...serp_jobs.internal_linking.show_more
TMC is Now Hiring Student Truck Drivers!.TMC is Hiring Student Drivers in Your Area!.New drivers are earning $100,000+ annually!\n\t.
Drivers average $1,350 - $1,600 weekly.Home Weekends - Our flatb...serp_jobs.internal_linking.show_more
Are you looking to earn extra income from the comfort of your home? We're seeking motivated individuals to explore and test a variety of remote side hustle opportunities featured on FinanceBuzz.Thi...serp_jobs.internal_linking.show_more
Remote Opportunity : Link Posting Marketing Rep (No Experience Needed!) Are you self-driven, reliable, and ready to earn from anywhere? We’re looking for motivated individuals to jo...serp_jobs.internal_linking.show_more AlliedTravelNetwork is working with Prime Time Healthcare to find a qualified Speech Language Pathologist (SLP) in Cynthiana, Kentucky, 41031!.
Now Hiring : Allied Healthcare Speech Therapist - Cynth...serp_jobs.internal_linking.show_more Earn at least $2230 driving with Uber when you complete your first 162 trips in 30 days.Why Uber? Driving is an easy way to boost your income while maintaining the flexibility your schedule require...serp_jobs.internal_linking.show_more Earn up to $15 / hour + performance bonuses.Outlier, a platform owned and operated by Scale AI, is looking for.If you're passionate about improving models and excited by the future of AI, this is you...serp_jobs.internal_linking.show_more
AI Trainer -Remote Writing Evaluator
Outlier • Remote, KY, United States
serp_jobs.filters.remote
serp_jobs.job_card.full_time
serp_jobs.last_updated.last_updated_variable_days • serp_jobs.job_card.promoted
Travel Speech Language Pathologist (SLP) - $2,551 to $2,852 per week in Cynthiana, KY
Prime Time Healthcare • Cynthiana, KY, US
serp_jobs.job_card.full_time
serp_jobs.last_updated.last_updated_variable_days • serp_jobs.job_card.promoted
Process Engineer I
Avantor • Paris, KY, United States
serp_jobs.job_card.full_time
serp_jobs.last_updated.last_updated_30 • serp_jobs.job_card.promoted
Get Paid to Test Products at Home – Flexible Hours, Weekly Pay
OCPA • Paris, Kentucky, us
serp_jobs.job_card.part_time +1
serp_jobs.last_updated.last_updated_30 • serp_jobs.job_card.promoted
Travel Physical Therapist (PT) - $2,050 to $2,291 per week in Paris, KY
AlliedTravelCareers • Paris, KY, US
serp_jobs.job_card.full_time
serp_jobs.last_updated.last_updated_variable_days • serp_jobs.job_card.promoted
Dashers - Sign Up and Start Earning
DoorDash • Bighill, KY, US
serp_jobs.job_card.full_time +1
serp_jobs.last_updated.last_updated_variable_hours • serp_jobs.job_card.promoted • serp_jobs.job_card.new
Earn Extra Income Driving
Uber • Dow, Kentucky, United States
serp_jobs.job_card.full_time +1
serp_jobs.last_updated.last_updated_variable_days • serp_jobs.job_card.promoted
X-Ray Technologist - Sign-On Bonus
Fast Pace Health • Winchester, Kentucky, United States
serp_jobs.job_card.full_time
serp_jobs.last_updated.last_updated_variable_days • serp_jobs.job_card.promoted
Travel Physical Therapist (PT) - $2,050 to $2,291 per week in Paris, KY
AlliedTravelNetwork • Paris, KY, US
serp_jobs.job_card.full_time
serp_jobs.last_updated.last_updated_variable_days • serp_jobs.job_card.promoted
Travel Physical Therapist - $2,276 per week
CompHealth Inc. • Richmond, KY, United States
serp_jobs.job_card.permanent
serp_jobs.last_updated.last_updated_30 • serp_jobs.job_card.promoted
AI Agent Evaluation Analyst (Freelance)
Mindrift • KY, US
serp_jobs.filters.remote
serp_jobs.job_card.part_time +1
serp_jobs.filters_job_card.quick_apply
serp_jobs.last_updated.last_updated_variable_days
Travel Nurse RN - Home Health in Richmond, KY
TravelNurseSource • Richmond, KY, US
serp_jobs.job_card.full_time
serp_jobs.last_updated.last_updated_30 • serp_jobs.job_card.promoted
Entry Level Truck Driver - Home Weekends - Earn Up to $100,000 / Year
TMC • Richmond, KY, US
serp_jobs.job_card.full_time
serp_jobs.last_updated.last_updated_30 • serp_jobs.job_card.promoted
Remote Side Hustle Evaluator - Flexible Online Gig Work
Finance Buzz • Cynthiana, Kentucky, US
serp_jobs.filters.remote
serp_jobs.job_card.temporary
serp_jobs.last_updated.last_updated_30 • serp_jobs.job_card.promoted
Work from Home / Online Opportunity - Part-time with Flexible Scheduling & No Experience Necessary!
HomeJobFinder • Cynthiana, KY
serp_jobs.filters.remote
serp_jobs.job_card.part_time
serp_jobs.last_updated.last_updated_variable_days • serp_jobs.job_card.promoted
Travel Speech Language Pathologist (SLP) - $2,551 to $2,852 per week in Cynthiana, KY
AlliedTravelNetwork • Cynthiana, KY, US
serp_jobs.job_card.full_time
serp_jobs.last_updated.last_updated_variable_days • serp_jobs.job_card.promoted
Drive with Uber : Earn at least $2230 when you complete 162 trips, guaranteed
Uber • Richmond, Kentucky, United States
serp_jobs.job_card.full_time +1
serp_jobs.last_updated.last_updated_variable_days • serp_jobs.job_card.promoted
Remote AI Writing Evaluator
Outlier • Remote, KY, United States
serp_jobs.filters.remote
serp_jobs.job_card.full_time
serp_jobs.last_updated.last_updated_variable_days • serp_jobs.job_card.promoted