Anyone Ai

Research Scientist (Remote/US/LATAM)

Mexico City, Mexico, Mexico · Remote (country restricted) · Contract

Apply on the employer’s site ↗See how well you match this job

You apply directly with the employer. ROLIVA does not submit applications from this page and is not the employer.

About the role

Description from the employer’s posting. Check the employer’s page for the current version.

EmployerAnyone Ai
LocationMexico City, Mexico, Mexico
Work styleRemote (country restricted)
Employment typeContract
Employer posted3 months ago

Research Scientist, LLM Evaluations & Benchmarking

Anyone AI Labs
Reports to: CEO · Remote / LatAm / US

The role Evaluation is one of the hardest open problems in AI: we still don't have reliable ways to measure what frontier models can and can't do, and the field mostly runs on benchmarks that are saturated, contaminated, or measuring the wrong thing. You'll own that problem at Anyone AI, measuring frontier model capability.

This is a research role at heart: you decide what a good evaluation is , design the benchmarks that prove it, and defend the methodology under lab scrutiny. You'll build frontier-grade evaluation packages across reasoning, coding, agents, tool use, and multi-modal — grounded in expert-verified truth, validated against multiple models, and QC'd to survive buyer-side review.

Responsibilities

● Evaluation research. Turn eval targets into original benchmark designs. Own the hard measurement questions: construct validity, item discrimination, headroom, reliability, contamination, and capability elicitation. Push toward evals that stay informative as models improve.

Show full description

● Benchmark development. Build evaluation packages with subject-matter experts, each with expert-verified ground truth, multi-model headroom results, and rigorous QC (calibration layers, severity-weighted rubrics, deterministic verifiers).

● Experts. Recruit, calibrate, and review a pool across coding, agentic/tool-use, and STEM/reasoning. Be the final arbiter of correctness and frontier difficulty.

● Lab relationships. Be a technical point of contact for labs, with CEO support. Understand what they're trying to measure and translate it into an evaluation design.

● Delivery & dissemination. Turn lab requests into winning sample packages and own pilots end to end. Where the work generalizes, help turn it into public benchmarks and papers: we support publishing at venues like NeurIPS Datasets & Benchmarks, ICLR, and ACL.

What we're looking for

● Research background in ML evaluation or benchmarking (a track record of published or open benchmarks, eval/measurement research, or equivalent hands-on work that labs have relied on).

● Deep LLM/frontier-model benchmarking expertise, with real strength in code-model and agentic evaluation.

● Fluency with the measurement problem itself: construct validity, psychometrics, rubrics, pass rates, headroom, contamination, and what makes a task genuinely discriminate a model.

● Interest in the safety side of evaluation, capability elicitation, robustness, and measuring the things that are hardest to measure honestly.

● Proven ability to hold a team or expert pool to a rigorous standard.

● Comfort with the full research loop: framing the question, running the study, and writing it up.

● Fluent English; Spanish a plus.

See how well you match this job Apply on the employer’s site ↗

Check your resume against this job Free, no sign-up. Nothing you paste is stored.

Staying safe while you apply

Never pay to apply, interview or accept an offer. ROLIVA never asks you for payment to apply to a job.
Apply only through the official “Apply on the employer’s site” button, on the employer’s own site, and check that a recruiter’s email uses that employer’s own domain rather than a free email address.
Be cautious of an offer made with no real interview, or pressure to decide quickly or share financial details early.

Does something look wrong with this listing? Use “Report broken link” in the role details, or read ROLIVA’s full job-safety guidance ↗.

Preparing for this role with ROLIVA

Check the fit.
Compare the requirements with the experience you have confirmed, including gaps.
Prepare honestly.
Build interview stories and application drafts from your own evidence.
Keep track.
Save the role and record your own application status and follow-ups.

Create a free workspace or log in.