Fleet Statistics: Cohorts, Percentiles and the n ≥ 30 Rule

A hotspot delta of 10.8 °C means little on its own. Is it good for this model? For this board? The number becomes useful when it is compared with other cards measured the same way. An Assurd certificate makes that comparison against one population only: the other cards Assurd has certified, which we call the fleet.

This post explains how that comparison works (SOP-GPU-001 v1.1, Step 4.2, and the Rules Engine that implements it). It covers where the fleet data comes from, which cards are compared with which, when a comparison is withheld, and how a percentile is computed. It also explains two metrics whose main purpose is fleet comparison: clock per watt and run-to-run variation.

Where the fleet data comes from

When a certificate is minted, its fleet metrics are copied into a fleet sample:

• hotspot delta, peak effective clock, peak VRAM and VRM temperatures;
• clock per watt;
• the mean consistency gap and stutter magnitude;
• the calibrated load Leq (Premium recordings that pass the validity checks only).

A metric the card did not measure is stored as null, never as 0, and it simply does not count towards that metric's statistics. Each gaming run is also stored separately, with its configuration, for the per-run comparison described below.

Each certified asset contributes one fleet sample, holding its latest certification. A Re-Certification of the same asset replaces that sample with the card's current condition instead of adding a second one, so a card that is certified twice is counted once, as it is now.

Once a night, an aggregation job recomputes each cohort's mean and standard deviation from the samples. A certificate minted today is compared against the statistics from the most recent run.

Cohorts

A comparison is only fair between cards that should behave alike. So every metric is ranked in up to three scopes, from the narrowest to the widest:

Scope	Who is in it
Exact silicon	Same PCI vendor, device and board (subsystem)
Weight class	Every SKU in the same weight class
Global fleet	Every certified card

The exact-silicon scope starts with the card's own board: vendor ID, device ID and subsystem ID, as the server reads them from the archived GPU-Z dump. If that board has fewer than 30 samples, it falls back to every board with the same vendor and device ID. The weight class is the SKU's performance class (1080p, 1440p, 4K or data centre), which is itself an engineering prior set per SKU, not derived from the fleet.

Gaming runs are compared per configuration. Each run is ranked only against runs with the same title, output resolution, settings preset, ray tracing on or off, upscaler, upscaler mode and frame generation. Upscaled and native runs are never compared, and nor are DLSS, FSR and XeSS runs with each other.

The n ≥ 30 rule

A percentile is published only when the cohort holds at least 30 samples of that metric. Below that, the certificate shows no rank, no cohort mean and no z-score. It shows the real cohort size instead, as Provisional · n = X. Nothing is estimated, borrowed from another cohort or filled in from a reference value.

Why withhold the rank instead of showing a rough one? Because the cohort mean is itself an estimate. With $n$ samples drawn from a cohort whose true spread is $\sigma$, the computed mean is typically off by one standard error:

$$\mathrm{SE} = \frac{\sigma}{\sqrt{n}} .$$

Take a card that sits exactly at the true cohort mean, so its true percentile is 50. If the computed mean is off by one standard error, the card's z-score is off by $1/\sqrt{n}$, and its percentile moves by this much:

Cohort size $n$	SE (σ)	Percentile error
5	0.447	17 points
10	0.316	12 points
30	0.183	7 points
100	0.100	4 points
300	0.058	2 points

At $n = 5$, "better than 50 %" could as easily have been 33 % or 67 %. At $n = 30$ the error is about 7 points, which we judged to be the smallest cohort worth publishing. The table only covers the error in the mean; the standard deviation is also an estimate, which makes small cohorts worse still.

The number 30 is an engineering prior pending fleet calibration (n ≈ 300), like every threshold in the SOP. It is a judgement about acceptable error, not a statistical law.

Fleet percentile

The Rules Engine does not sort the cohort and count. It models each cohort metric as a normal (Gaussian) distribution with the cohort's mean $\mu$ and standard deviation $\sigma$, and converts the card's value $x$ into a z-score:

$$z = \frac{x - \mu}{\sigma} .$$

The standard normal cumulative distribution $\Phi(z)$ gives the share of the modelled cohort below $x$. The engine stores a top-r % rank, rounded and limited to 1–99:

$$\begin{gathered} r = 100\,\bigl(1 - \Phi(z)\bigr)\ \text{(higher is better)} \\ r = 100\,\Phi(z)\ \text{(lower is better)} \end{gathered}$$

The certificate then shows $P = 100 - r$ as "better than P % of the cohort". The direction is already accounted for, so a higher $P$ is always better, whether the metric is a clock speed or a temperature.

Worked example 1, lower is better. A hypothetical exact-silicon cohort has a hotspot delta mean of 12.0 °C and a standard deviation of 2.5 °C, with 30 samples. A card measures 10.8 °C:

$$z = \frac{10.8 - 12.0}{2.5} = -0.48 .$$

$\Phi(-0.48) = 0.3156$, so $r = 32$ and the certificate reads better than 68 % of the cohort. With 29 samples instead of 30, the same card reads Provisional · n = 29.

Worked example 2, higher is better. A hypothetical cohort has a peak effective clock mean of 2700 MHz and a standard deviation of 50 MHz. A card peaks at 2758.1 MHz, so $z = 1.162$, $r = 12$, and it reads better than 88 %.

Three consequences of the method are worth knowing:

• It is a model percentile, not a head count. If a metric's real distribution is skewed, the modelled percentile and the share of actual cards below the value can differ, mostly in the tails.
• The scale stops at 99. A card more than about 2.2 standard deviations better than the mean reads "better than 99 %", however far out it is.
• Extreme samples are kept out of the baseline. Once a baseline holds 10 samples, a new sample more than 3 standard deviations from the running mean on any metric is left out of that cohort's statistics, so one faulty card cannot shift everyone else's percentile. The card is still certified and ranked like any other.

Certificate-level gaming percentiles combine the per-run ranks. Each run is ranked in its own configuration cohort, so there is no single cohort mean to show. The certificate reports the median of the runs' top-r % ranks that were ranked (rounded half up) and leaves the cohort mean blank. Runs with $r$ = 12, 20, 35 and 40 give a median of 27.5, stored as 28: better than 72 %.

Clock per watt

Clock per watt relates a card's peak effective clock to its peak board power, both taken from the 1800 s load trace:

$$\text{CPW} = \frac{f_{\text{peak}}}{P_{\text{peak}}}$$

with $f_{\text{peak}}$ in MHz and $P_{\text{peak}}$ in watts, so CPW is in MHz/W. It is a plain ratio: there are no reference points and no clamp, so cards keep their separation however efficient they are. It is null, never 0, when either reading is missing or the power reading is not above 0 W.

It is not performance per watt: a clock speed is not a measure of work done, and two architectures at the same clock do very different amounts of work. It is also a ratio of two peaks, which need not occur at the same moment of the trace. It says how far the card clocked relative to how much power it drew at most, not what it clocked at a given power.

• 2520 MHz at 450 W: $2520 / 450 = 5.60$ MHz/W.
• 2758.1 MHz at 221.6 W: $2758.1 / 221.6 = 12.45$ MHz/W.

Read it within the exact-silicon cohort, where the cards share a design, not as a ranking of architectures.

Clock per watt replaced an earlier, unitless clock-power index: the same ratio divided by 10 and capped at 1.00, so every card at 10 MHz/W or more read exactly 1.00. That index is no longer shown or ranked. In the fleet statistics, an old sample's index below the cap is read as clock per watt (×10); a capped one says only "10 MHz/W or more" and is left out.

Run-to-run variation

Premium certifications run Stable Diffusion XL for 20 consecutive measured iterations (after 2 unrecorded warm-up iterations). The benchmark script reports the mean iteration speed $\mu$ and its standard deviation $\sigma$, both in iterations per second, and the technician records them. Run-to-run variation is their coefficient of variation:

$$\mathrm{CoV} = 100\,\frac{\sigma}{\mu}\ \%.$$

The sample certificate's 4.62 ± 0.07 it/s gives $100 \times 0.07 / 4.62 = 1.5$ %. A mean of 0 gives no CoV (null), not an infinite one.

The CoV is shown, not ranked in the fleet. Its job is in the tensor-efficiency Soft Limit (Step 4.3): a measured speed below 85 % of the SKU reference and a standard deviation above 10 % of the mean, which is a CoV above 10 %, together point to memory or memory-controller instability and require a justification. A low speed with a steady CoV, or an unsteady CoV at full speed, does not trigger it.

Calibration status

The 30-sample minimum, the 10-sample start of outlier rejection, the 3-standard-deviation cut-off, the 85 % and 10 % in the tensor Soft Limit and the weight classes are all engineering priors pending fleet calibration (n ≈ 300). None of them was fitted to data. We do not describe them as empirical or statistically validated.

The Gaussian model is also an assumption. Once cohorts hold enough samples, it can be checked against the actual distributions, and a metric that is clearly not normal can be ranked differently. Any change will be versioned in the SOP.

See it on a certificate

Every ranked metric appears in the certificate's Fleet intelligence tile, with its observed value, the cohort mean and the rank in each scope. The sample certificate shows the layout. It is illustrative: no card was measured, and since the sample belongs to no fleet, every rank on it reads Provisional · n = 0. Its clock per watt of 12.45 MHz/W is the second example above.

The metrics being ranked are explained in Frame Pacing from First Principles, Hotspot Delta and TIM Pump-Out, The Thermal Time Constant and the 1800 s Trace and Acoustic Leq and A-Weighting.

Notation

Symbol	Meaning	Unit
$n$	Samples of a metric in a cohort	—
$\mu$, $\sigma$	Cohort mean and standard deviation	—
$x$, $z$	Card's value and its z-score	—
$\Phi$	Standard normal cumulative distribution	—
$r$, $P$	Top-r % rank; "better than P %" ($100 - r$)	%
$f_{\text{peak}}$	Peak effective clock over the trace	MHz
$P_{\text{peak}}$	Peak board power over the trace	W
CPW	Clock per watt	MHz/W