How it works — 7 of 7
Eval battery
Selected mechanisms described on the previous pages — including the arbiter's dice, committee behavior and escalation choices — have named evaluation tracks. Operators launch an allowlisted track through the task worker. Some tracks call language models, some analyze stored runs or registered forecast questions, and some run a purpose-built scenario; they are not one universal, entirely offline battery.
The organizing idea is to measure the ways the platform's own agents can go wrong, not just confirm the ways they go right. Does a committee's vote flip just because a question was reworded, with nothing substantive changed? Does swapping the underlying language model change the outcome of the same case? Does a lone dissenting position survive being outvoted and stay in the record, or does it quietly disappear into a unanimous-looking summary? When an actor is handed a face-saving way to step back from a stand-off, does it actually take that path, or does it escalate anyway because escalating is the more dramatic choice? None of these questions get answered by asking a model how it thinks it did — metrics are computed by track-specific code from the structured replies, votes, logs, registered questions or reference data that track actually uses. No universal “full transcript” exists across all tracks, and a track's scope matters as much as whether its result passed.
Each battery below reports one of the five evidence statuses from the previous page, and the mapping from a battery to its status lives in the same code that runs the check — it is not something chosen after seeing whether a result looks good.
| Battery | Evidence status |
|---|---|
| Arbiter Reliability | Synthetic |
| Committee Baselines | Synthetic |
| Forecasting (predict) | Forecast |
| Forecasting (score) | Forecast |
| Run Reports | Synthetic |
| Persona-drift curve | Synthetic |
| Synthetic trajectory regression (formerly "Retrodiction") | Synthetic |
| Provider smoke | Synthetic |
| Economic Connectivity (direction/sign/lag) | Calibrated |
| Population feedback (feasibility + ablation) | Calibrated |
| Demo run (golden scenario, Track D) | Scenario |
| Committee prompt-sensitivity | Synthetic |
| Committee provider-swap invariance | Synthetic |
| Committee rounds (diversity-collapse) | Synthetic |
| Committee walk-away rate | Synthetic |
| Escalation bench (unprovoked escalation / de-escalation uptake) | Synthetic |
| Committee pathology report (battery aggregator) | Synthetic |
| Deferred checks (backtest) | Retrodictive |
| Stylized-facts machinery (v6-track-ec-math-infra Шаг 3) | Synthetic |
flowchart LR
accTitle: Offline evaluation flow
accDescr: Selected mechanisms or registered questions enter an allowlisted evaluation job, a track-specific scorer computes metrics, and the system stores an evidence status and operator record.
A[Selected mechanism, stored run or registered question] --> B[Allowlisted worker job: track-specific inputs and scorer]
B --> C[Evidence status attached]
C --> D[Operator evaluation record]