olmo-eval: An analysis workbench for the mannequin growth loop

💻 Code: https://github.com/allenai/olmo-eval

When you’re constructing an LLM, you consider it again and again throughout many interventions. Each adjustment to its knowledge, structure, or hyperparameters — and each step up in scale — sends you again by the identical loop: including or reconfiguring benchmarks, re-running them on every new mannequin checkpoint, noting the outcomes, and checking whether or not one thing that helped in a small experiment nonetheless holds up on the total coaching run.

Most analysis instruments aren’t designed for this—they’re both constructed to run established benchmarks throughout completed fashions or run a mannequin by multi-step, tool-using issues in a sandbox. They don’t sustain with a mannequin that is consistently altering, nor do they replicate how a mannequin would possibly behave below particular real-world circumstances.

Our final venture to deal with this analysis problem was OLMES, the Open Language Mannequin Analysis Commonplace. Launched in 2024, it was meant to make LLM benchmark scores simpler to match throughout releases. The identical fashions have been being scored on the identical benchmarks in several methods — facets like immediate formatting and job formulation usually assorted from paper to paper — so claims about which fashions carried out finest usually weren’t reproducible. OLMES pinned benchmarking decisions down in an open, documented normal, and it turned the idea for evaluating our open fashions from Olmo to Tulu.

However a mannequin’s remaining rating is simply a part of the analysis course of—which is why we’re releasing olmo-eval, a brand new workbench that builds on OLMES and extends it throughout the remainder of LLM growth. In comparison with OLMES, olmo-eval cuts down the work of implementing new evaluations, affords extra flexibility in defining the place and the way they run, and makes it simpler to compose particular person parts into bigger workflows. Agentic and multi-turn analysis is supported as a first-class use case, and stronger evaluation instruments assist you decide whether or not an intervention really improved on the baseline or the distinction quantities to noise.

How olmo-eval differs from current instruments

Is a 2.4pp change in efficiency sufficient to make a name?

olmo-eval overlaps in some methods with Harbor, an open framework for evaluating AI brokers inside containerized, sandboxed environments. However the two instruments differ of their scope. Harbor is aimed primarily at operating and publishing agent benchmarks; olmo-eval was constructed for the on a regular basis work of growing a mannequin—including and configuring benchmarks, operating them throughout checkpoints, and analyzing the outcomes immediate by immediate as a substitute of as a single total rating.

Harbor runs the whole lot the identical means—inside sealed, reproducible containers. As a result of containers will be resource-intensive, olmo-eval allows you to select how every benchmark runs as a substitute. A benchmark that simply wants a mannequin to reply questions can run instantly, which is quicker and cheaper; a benchmark that wants a locked-down setting — say, one which runs code the mannequin wrote — will get an remoted container setup. The light-weight path is the default, and olmo-eval solely opts for the heavy setup when a benchmark really requires it.

Harbor’s course of for including a benchmark is constructed for evals you propose to publish and share publicly, with the additional verification steps that entails. olmo-eval is constructed for shifting rapidly whilst you develop, and the way you add a benchmark relies on what the benchmark wants: a brief definition for a primary eval, with choices to let a mannequin use instruments as it really works by a benchmark, or — for a benchmark that already has its personal code and process — a skinny wrapper so olmo-eval can run it as is and report the outcomes alongside different benchmark scores in the identical format.

Each Harbor and olmo-eval preserve benchmarks separate from the runtime coverage (how the mannequin is run to provide its solutions) so you’ll be able to change one with out rewriting the opposite, however olmo-eval is designed for higher modularity. In olmo-eval, the mannequin being evaluated, the instruments it may well use, the containerized setting, and any helper fashions – like an LLM-as-a-judge – are all swappable parts. You possibly can reuse a device throughout many harnesses, or plug a grading mannequin into one benchmark with out perturbing the others, and regulate small settings (e.g., the precise wording of the immediate) with out in depth effort.

Harbor stories an total rating for every mannequin. olmo-eval stories these scores too, every with a regular error and a minimal detectable impact (the smallest distinction that may be reliably distinguished from noise). However the extra helpful view traces the identical questions up throughout two mannequin checkpoints and compares them one after the other, with all else held fastened. This lets you see whether or not a tiny change in an total common would possibly point out an actual enchancment or just noise.

An built-in analysis stack

olmo-eval consists of 4 parts which might be helpful on their very own however designed to work collectively to tighten the experimental LLM growth loop:

A job/suite/harness abstraction that decouples benchmark logic from runtime coverage. A job is the way you outline a benchmark in olmo-eval—what’s being evaluated. A set teams duties right into a set you run collectively, and a harness controls how every job is run. This separation lets the identical job run as a regular baseline or with instruments and scaffolding, with out altering what it measures.

A sandbox and capability-routing layer, together with an asynchronous sandbox planner. This helps evaluations the place a mannequin’s response relies on the actions it takes utilizing instruments, like writing and operating code or shopping the net. The purpose is to guage the mannequin’s actual device use: when a benchmark requires instruments, olmo-eval runs these instruments and feeds the outcomes again to the mannequin.

A normalized experiment schema that data each run, its configuration, and the leads to the identical structured format. This makes it doable to group associated experiments, examine checkpoints over time, and keep away from the inconsistencies that always accumulate in long-running mannequin growth workflows.

A outcomes viewer for pairwise mannequin comparability: lining two fashions or checkpoints up query by query surfaces small however actual efficiency modifications that an total common can cover.

In most mannequin analysis setups, including a benchmark is a sizeable integration venture. In olmo-eval, all that’s wanted is a job—duties outline the benchmark dataset, how analysis requests are constructed, and the way mannequin solutions are scored (all code in Python):

from olmo_eval.widespread.formatters import ChatFormatter
from olmo_eval.widespread.metrics import AccuracyMetric
from olmo_eval.widespread.scorers import ExactMatchScorer
from olmo_eval.widespread.sorts import Occasion, SamplingParams
from olmo_eval.knowledge import DataLoader, DataSource
from olmo_eval.evals.duties.widespread import Job, register, register_variant

@register(“internal_freshqa”)
class InternalFreshQA(Job):
data_source = DataSource(path=“s3://evals/inside/freshqa.jsonl”, break up=“take a look at”)
formatter = ChatFormatter()
sampling_params = SamplingParams(temperature=0.0)
metrics = (AccuracyMetric(scorer=ExactMatchScorer),)

@property
def cases(self):
loader = DataLoader()
for idx, doc in enumerate(loader.load(self.config.get_data_source())):
yield Occasion(
query=doc[“question”],
gold_answer=doc[“answer”],
metadata={“id”: doc.get(“id”, f”freshqa_{idx}“)},
)

Variants specific modifications in analysis coverage with out duplicating the benchmark:

register_variant(“internal_freshqa”, “3shot”, num_fewshot=3, fewshot_seed=1234)
register_variant(“internal_freshqa”, “zero”, num_fewshot=0)

Suites group benchmarks into normal units you run collectively:

from olmo_eval.evals.suites import Suite, register

register(Suite(
title=“base_qa_few_shot”,
duties=(
“sciq:mc:3shot”,
“arc_challenge:mc:3shot”,
“internal_freshqa:mc:3shot”,
),
))

And since runtime coverage lives within the harness slightly than the duty definition, the identical benchmark will be simply rerun below completely different execution slightly than counting on whether or not a generated level observe merely appears believable.

# Baseline
olmo-eval run -m my-instruct-checkpoint -t internal_freshqa:zero

# Similar job, similar scoring, search/device runtime enabled
olmo-eval run -m my-instruct-checkpoint -t internal_freshqa:zero –harness search_agent

Reproducible analysis made open

Use olmo-eval when analysis is a part of ongoing mannequin growth slightly than a one-off run—when it’s essential to run the identical benchmarks repeatedly throughout checkpoints below reproducible circumstances and examine interventions at each the combination and per-question degree.

In case your recurring query is “How does this checkpoint differ from the final one, and the place precisely did it enhance or regress?”, that’s the workflow olmo-eval is constructed for.

Reproducible analysis ought to preserve tempo with how fashions are constructed—not solely how they’re scored as soon as they’re completed. olmo-eval carries the OLMES normal into lively mannequin growth, and we’re releasing it overtly so the group can construct on it.