Methodology
How Koliseum evaluates models in live environments, reducing contamination and reward hacking.
The model commits before the outcome exists.
Every Koliseum live environment presents a task whose outcome is not known when the model answers. The model researches, decides, and acts with only the information available at a documented cutoff. This keeps future outcomes out of the answer and reduces contamination.
At the moment it commits, the output, its stated confidence, the sources it used, the model and harness configuration, and the timestamp are recorded and sealed. Nothing that happens afterwards can reach back and change them.
The outcome comes from the world, not from a rubric.
The outcome arrives from outside the test. The environment waits for it, then reads it from a public or independently verifiable source. The transcript remains evidence; completion is read from the state the work left behind, reducing the room for reward hacking against a rubric.
A market moves
Sealed decisions are marked at the close.
A test suite runs
A repository repair passes or fails its tests.
A system state is verified
An order book, a ticket queue, a claim file, read after the work.
The score is measured against a disclosed baseline.
Every score is reported against a disclosed baseline: the market over the same window, the consensus at the same snapshot, or a stated statistical reference. A model that copies the baseline scores zero. Improvement and shortfall are measured from there, so the score rewards useful performance rather than satisfying a hidden rubric.
Forecast quality and action quality are reported separately because they answer different questions. A blended score can hide a failure mode; separate axes cannot.
Every run is recorded and can be inspected.
Each run keeps its configuration, actions, tool calls, cost, and elapsed time. Results can be inspected by task, model configuration, outcome, baseline, cost, and failure mode, and replayed step by step.
The environment, the grader, and the evaluation conditions are frozen between runs, so two checkpoints of the same model can be compared on the same terms. Only the declared model, prompt, or agent changes.
Build an environment with Koliseum.
Talk directly with the founders about the forecast or action you need to measure.