How the index is built

Boards spell things differently. One model arrives as claude-opus-5-max, as Claude Opus 5 (High), as anthropic/claude-opus-5. Snapshot dates, hosting platforms and run settings are stripped; reasoning effort, thinking and preview labels fold into one model with the highest tier kept.

The exceptions are vendors who really did ship two sets of weights, such as Qwen3 2507 Instruct and Thinking, or Kimi K2 and K2 Thinking. Each needs evidence on the record before it is split. Merging two different models, or attributing one model’s score to another, is the worst thing this index can do, and it is treated that way.

An Elo of 1501 and a 57.9% pass rate cannot be averaged. Every measure is z-scored across the models its own source lists, then mapped to 50 ± 15 per standard deviation. 50 is that measure’s average, not a grade. Fewer than 3 models on a measure and it is not standardized at all, because there is no population to stand against. A figure a publisher prints as a risk or an error rate has its sign flipped; the raw stays exactly as printed.

Standardized distance is capped at ±2 standard deviations. Past that a board has stopped discriminating and started extrapolating, and one such figure could outweigh four boards. The cap changes no board’s own order.

Where a source publishes uncertainty, the figure counts for less, scaled by how wide the bar is relative to that board’s spread: ±3.8 is tight on a board spanning 40 points and useless on one spanning 4.

A figure with no published error bar counts in full. Absence of an error bar is not evidence of a wide one, and guessing would invent information.

A board earns per-measure weight by running a fixed set on everyone. A report ✱ does not: how many rows a launch post prints is a choice, not evidence. The measures one publication contributes to one scope are averaged and enter once.

An independent write-up carries 12 against a board measure’s 25; a figure the model’s own publisher printed carries 10. Reports never enter the Overall index, and in a category they decide nothing a board already measured.

Fixed, printed, and not adjustable by the reader. A reader cannot be expected to know what share a leaderboard deserves; that is an editorial call, and the only honest way to make one is to state it with the reason attached.

A board’s category figures shape their own columns at the same share, and never the Overall index, which would count that board twice.

A missing figure stays missing. Thin evidence is pulled toward 50 by a prior worth 25% of the weight available in that scope. With full coverage the pull is mild; with one small board it dominates, which is the intended answer to a model that topped a single leaderboard.

Boards from 2 independent publishers must have scored it, and 50% of the Overall weighting must actually have been measured.

The five boards reach very different distances, so most models are measured on part of the scheme and the prior fills the rest. Below half, the prior is deciding the placing, and a placing decided by the prior is an artefact of thin coverage rather than a finding about the model. The cost is visible on the page: 62 ranked out of 946.

A model with no Overall index is placed among models that have one, on a measure they share, and their index is read at that position. Outside the anchors’ range the nearest anchor is a bound, a ceiling or a floor, not a point. Safety and Decisions figures never anchor an estimate: a low risk rate is not evidence of capability, and hallucination rate and the Overall index correlate at −0.23.

• Run our own evaluations. There is nothing here to tune and no conflict of interest.
• Impute missing values.
• Let readers reweight. Adjustable weights mean everyone can produce the conclusion they came for.
• Use second-hand numbers. Every figure is read from the publisher’s own page and links back to it.

Stated here because a method page that only argues for itself is an advertisement.

• Standardizing assumes each board’s population is comparable. “Average” on a board of twenty frontier models is not “average” on a board of three hundred. Shrinkage softens this; it does not solve it.
• The weights are an editorial judgment. The shares above have no objective derivation. All we can do is print them with reasons and invite disagreement.
• The boards correlate. Artificial Analysis is itself a composite that overlaps others here, so there is partial double counting. The rationale says so; the arithmetic does not correct for it.
• Contamination is not something we can see. A board that publishes a sealed control set can show how much of a score survives unseen questions. Most do not, and we inherit whatever the published figure already includes.