Talent.com
AI Agent Evaluation Analyst (Freelance)
AI Agent Evaluation Analyst (Freelance)Mindrift • SC, US
AI Agent Evaluation Analyst (Freelance)

AI Agent Evaluation Analyst (Freelance)

Mindrift • SC, US
job_description.job_card.variable_days_ago
serp_jobs.job_preview.job_type
  • serp_jobs.job_card.part_time
  • serp_jobs.job_card.permanent
  • serp_jobs.filters.remote
  • serp_jobs.filters_job_card.quick_apply
job_description.job_card.job_description

This opportunity is only for candidates currently residing in the specified country. Your location may affect eligibility and rates. Please submit your resume in English and indicate your level of English proficiency.

At Mindrift , innovation meets opportunity. We believe in using the power of collective human intelligence to ethically shape the future of AI.

What we do

The Mindrift platform, launched and powered by Toloka , connects domain experts with cutting-edge AI projects from innovative tech clients. Our mission is to unlock the potential of GenAI by tapping into real-world expertise from across the globe.

Who we're looking for :

We’re looking for curious and intellectually proactive contributors, the kind of person who double-checks assumptions and plays devil’s advocate.

Are you comfortable with ambiguity and complexity? Does an async, remote, flexible opportunity sound exciting? Would you like to learn how modern AI systems are tested and evaluated?

This is a flexible, project-based opportunity well-suited for :

  • Analysts, researchers, or consultants with strong critical thinking skills.
  • Students (senior undergrads / grad students) looking for an intellectually interesting gig.
  • People open to a part-time and non-permanent opportunity.

About the project :

We’re on the hunt for QAs for autonomous AI agents for a new project focused on validating and improving complex task structures, policy logic, and agent evaluation frameworks. Throughout the project, you’ll have to balance quality assurance, research, and logical problem-solving. This project opportunity is ideal for people who enjoy looking at systems holistically and thinking through scenarios, implications, and edge cases.

You do not need a coding background, but you must be curious, intellectually rigorous, and capable of evaluating the soundness and consistency of complex setups. If you’ve ever excelled in things like consulting, CHGK, Olympiads, case solving, or systems thinking — you might be a great fit.

What you’ll be doing :

  • Reviewing evaluation tasks and scenarios for logic, completeness, and realism.
  • Identifying inconsistencies, missing assumptions, or unclear decision points.
  • Helping define clear expected behaviors (gold standards) for AI agents.
  • Annotating cause-effect relationships, reasoning paths, and plausible alternatives.
  • Thinking through complex systems and policies as a human would to ensure agents are tested properly.
  • Working closely with QA, writers, or developers to suggest refinements or edge case coverage.

    • How to get started :

    Apply to this post, qualify, and get the chance to contribute to a project aligned with your skills, on your own schedule. Shape the future of AI while building tools that benefit everyone.

    Requirements

  • Excellent analytical thinking : Can reason about complex systems, scenarios, and logical implications.
  • Strong attention to detail : Can spot contradictions, ambiguities, and vague requirements.
  • Familiarity with structured data formats : Can read, not necessarily write JSON / YAML.
  • Ability to assess scenarios holistically : What's missing, what’s unrealistic, what might break?
  • Good communication and clear writing (in English) to document your findings.

    • We also value applicants who have :

  • Experience with policy evaluation, logic puzzles, case studies, or structured scenario design.
  • Background in consulting, academia, olympiads (e.g. logic / math / informatics), or research.
  • Exposure to LLMs, prompt engineering, or AI-generated content.
  • Familiarity with QA or test-case thinking (edge cases, failure modes, “what could go wrong”).
  • Some understanding of how scoring or evaluation works in agent testing (precision, coverage, etc.).

    • Benefits

  • Get paid for your expertise, with  rates that can go up to $60 / hour  depending on your skills, experience, and project needs.
  • Take part in a flexible, remote, freelance project that fits around your primary professional or academic commitments.
  • Participate in an advanced AI project and gain valuable experience to enhance your portfolio.
  • Influence how future AI models understand and communicate in your field of expertise.
    • serp_jobs.job_alerts.create_a_job

    Ai Analyst • SC, US

    Job_description.internal_linking.related_jobs
    Statistical Analyst (Columbia)

    Statistical Analyst (Columbia)

    US Tech Solutions • Columbia, SC, US
    serp_jobs.job_card.part_time
    Partially onsite / M-F; 8 hours per day; 40 hours per week.Interview Process : 2 rounds First interview in person with manager. Second interview virtually with lead staff.Required Software : Microso...serp_jobs.internal_linking.show_more
    serp_jobs.last_updated.last_updated_variable_days • serp_jobs.job_card.promoted
    Senior Analyst, Capacity Planning (Remote)

    Senior Analyst, Capacity Planning (Remote)

    MAXIMUS • Columbia, SC, United States
    serp_jobs.filters.remote
    serp_jobs.job_card.full_time
    Senior Analyst, Capacity Planning.This role plays a key part in driving operational efficiency and workforce scalability in large, regulated environments. Reporting to the Senior Manager, Capacity P...serp_jobs.internal_linking.show_more
    serp_jobs.last_updated.last_updated_1_day • serp_jobs.job_card.promoted
    Mechanical Engineer ( Fuel Systems )

    Mechanical Engineer ( Fuel Systems )

    Career Transitions, LLC • Blythewood, SC, US
    serp_jobs.job_card.full_time
    serp_jobs.filters_job_card.quick_apply
    Mechanical Engineer ( Fuel Systems ).Meet your Talent Advisor Desmond Nugent.A global, leading tier 1 automotive parts manufacturer is seeking a Mechanical Engineer to support the re-engineering an...serp_jobs.internal_linking.show_more
    serp_jobs.last_updated.last_updated_30 • serp_jobs.job_card.promoted
    Statistical Analyst (Columbia)

    Statistical Analyst (Columbia)

    Meridian Technologies • Columbia, South Carolina Metropolitan Area, US
    serp_jobs.job_card.part_time +1
    Job Title : Statistical Programming Analyst.Contract duration : 6 month contract with potential for extension or conversion. We are seeking a Statistical Programming Analyst to join our Medicare Stati...serp_jobs.internal_linking.show_more
    serp_jobs.last_updated.last_updated_variable_days • serp_jobs.job_card.promoted
    Surrogates Start at $55K-$75K+ $1,200 Bonus + $11K Pre-Pregnancy Payout

    Surrogates Start at $55K-$75K+ $1,200 Bonus + $11K Pre-Pregnancy Payout

    Physician's Surrogacy • Blythewood, South Carolina, United States
    serp_jobs.job_card.full_time +2
    Earn up to $75,000 as a stay-at-home mom or as a second job.Sign up today and earn $1,200 bonus.What qualities make you a great Surrogate?. Experienced no complications during your own pregnancy.Hea...serp_jobs.internal_linking.show_more
    serp_jobs.last_updated.last_updated_variable_days • serp_jobs.job_card.promoted
    Become A Surrogate! Earn $75k+, plus all expenses. Women 21-43 with previous pregnancies only.

    Become A Surrogate! Earn $75k+, plus all expenses. Women 21-43 with previous pregnancies only.

    Simple Surrogacy • Gaston, SC, United States
    serp_jobs.job_card.full_time +1
    Become a surrogate and give the gift of family while helping your own family financially.Supplement your family’s income as stay-at-home mom or continue to work at your job, while.You should be in ...serp_jobs.internal_linking.show_more
    serp_jobs.last_updated.last_updated_variable_days • serp_jobs.job_card.promoted
    Analyst, Claims Research

    Analyst, Claims Research

    BlueCross BlueShield of South Carolina • Columbia, SC, US
    serp_jobs.job_card.full_time
    This position completes research efficiently and accurately to ensure the departmental goals to prevent aging of claims and erroneous disbursements or claims from processing incorrectly.This positi...serp_jobs.internal_linking.show_more
    serp_jobs.last_updated.last_updated_30 • serp_jobs.job_card.promoted
    Analyst, Tech Controls

    Analyst, Tech Controls

    Coinbase • Columbia, SC, United States
    serp_jobs.job_card.full_time
    Ready to be pushed beyond what you think you’re capable of?.At Coinbase, our mission is to increase economic freedom in the world. It’s a massive, ambitious opportunity that demands the best of us, ...serp_jobs.internal_linking.show_more
    serp_jobs.last_updated.last_updated_30 • serp_jobs.job_card.promoted
    Earn Extra Income Driving

    Earn Extra Income Driving

    Uber • Winnsboro, South Carolina, United States
    serp_jobs.job_card.full_time +1
    What is Uber? Driving with Uber is an alternative to a part-time or full-time job and can get you earning cash on the road quickly. Driving with Uber allows you to earn quick cash while maintaining ...serp_jobs.internal_linking.show_more
    serp_jobs.last_updated.last_updated_variable_days • serp_jobs.job_card.promoted
    Data Modeler

    Data Modeler

    SYSTEMTEC • Columbia, South Carolina Metropolitan Area, United States
    serp_jobs.job_card.full_time
    The ideal candidate will bring strong technical expertise, exceptional communication skills, and the ability to collaborate across business and technical teams to ensure high-quality, scalable data...serp_jobs.internal_linking.show_more
    serp_jobs.last_updated.last_updated_variable_days • serp_jobs.job_card.promoted
    Licensed Professional Counselor (LPC)

    Licensed Professional Counselor (LPC)

    LifeStance Health • Gaston, SC, US
    serp_jobs.job_card.full_time
    At LifeStance Health, we believe in a truly healthy society where mental and physical healthcare are unified to make lives better. Our mission is to help people lead healthier, more fulfilling lives...serp_jobs.internal_linking.show_more
    serp_jobs.last_updated.last_updated_variable_days • serp_jobs.job_card.promoted
    Remote Side Hustle Evaluator - Flexible Online Gig Work

    Remote Side Hustle Evaluator - Flexible Online Gig Work

    Finance Buzz • Ridgeway, South Carolina, US
    serp_jobs.filters.remote
    serp_jobs.job_card.temporary
    Are you looking to earn extra income from the comfort of your home? We're seeking motivated individuals to explore and test a variety of remote side hustle opportunities featured on FinanceBuzz.Thi...serp_jobs.internal_linking.show_more
    serp_jobs.last_updated.last_updated_30 • serp_jobs.job_card.promoted
    Travel - MRI Tech

    Travel - MRI Tech

    Titan Medical (Nebraska) • Winnsboro, SC, United States
    serp_jobs.job_card.full_time
    WHEN YOU WORK FOR US, WE WORK FOR YOU.With Titan Medical, you gain access to thousands of travel nursing and allied health jobs across the country. You also get unmatched service.From the moment you...serp_jobs.internal_linking.show_more
    serp_jobs.last_updated.last_updated_variable_days • serp_jobs.job_card.promoted
    Statistical Analyst

    Statistical Analyst

    US Tech Solutions • Columbia, SC, US
    serp_jobs.job_card.full_time
    Partially onsite / M-F; 8 hours per day; 40 hours per week.Interview Process : 2 rounds – First interview in person with manager. Second interview virtually with lead staff.Required Software : Micros...serp_jobs.internal_linking.show_more
    serp_jobs.last_updated.last_updated_variable_days • serp_jobs.job_card.promoted
    Engineering Technician

    Engineering Technician

    Career Transitions, LLC • Blythewood, US
    serp_jobs.job_card.full_time +1
    serp_jobs.filters_job_card.quick_apply
    A hands-on and detail-oriented Engineering Technician is needed to join their team a growing team.The Engineering Technician will support engineering and manufacturing teams in the design, testing,...serp_jobs.internal_linking.show_more
    serp_jobs.last_updated.last_updated_variable_days • serp_jobs.job_card.promoted
    Evaluation Scenario Writer - AI Agent Testing Specialist

    Evaluation Scenario Writer - AI Agent Testing Specialist

    Mindrift • SC, US
    serp_jobs.filters.remote
    serp_jobs.job_card.part_time
    serp_jobs.filters_job_card.quick_apply
    This opportunity is only for candidates currently residing in the specified country.Your location may affect eligibility and rates. Please submit your resume in English and indicate your level of En...serp_jobs.internal_linking.show_more
    serp_jobs.last_updated.last_updated_30
    Border Patrol Agent - Earn up to $30,000 in Recruitment Incentives

    Border Patrol Agent - Earn up to $30,000 in Recruitment Incentives

    U.S. Customs and Border Protection • Swansea, South Carolina, US
    serp_jobs.job_card.full_time +1
    Border Patrol Agent (BPA) Entry Level.Check out the role overview below If you are confident you have got the right skills and experience, apply today. NEW RECRUITMENT AND RETENTION INCENTIVES!.Duty...serp_jobs.internal_linking.show_more
    serp_jobs.last_updated.last_updated_30 • serp_jobs.job_card.promoted
    Enterprise Intelligence Developer, FT, Variable, - Remote

    Enterprise Intelligence Developer, FT, Variable, - Remote

    Prisma Health • Columbia, SC, United States
    serp_jobs.filters.remote
    serp_jobs.job_card.full_time
    Responsible for taking the vast array of data sources maintained by Prisma and turning them into actionable insights.Designs robust data models, analyzes data from diverse sources to identify metri...serp_jobs.internal_linking.show_more
    serp_jobs.last_updated.last_updated_variable_hours • serp_jobs.job_card.promoted • serp_jobs.job_card.new