SpinProof: measured prefill and evaluated quality
Qwen2.5-0.5B on NVIDIA GH200: 38.69 to 33.72 ms median prefill with Transformers 4.56.2, a 1.1475× speedup. Evaluated quality outputs were bit-identical in that run. A retained Transformers 5.19.0 run regressed to 0.7899×.
| Transformers | Baseline (ms) | SpinProof (ms) | Ratio |
|---|---|---|---|
| 4.56.2 | 38.691839 | 33.718063 | 1.147511× |
| 5.19.0 | 27.252048 | 34.502640 | 0.789854× |
- The Transformers 4.56.2 result is prefill latency for a batch of 8 sequences, each 1,024 tokens long. It is not milliseconds per generated token.
- The retained Transformers 5.19.0 result is 26.6% higher latency than its own baseline. Performance is sensitive to the software version.
Measurement conditions
Both runs used Qwen2.5-0.5B in FP16 on NVIDIA GH200 480GB, with PyTorch 2.7.0, CUDA 12.8, Python 3.10.12 and NVIDIA driver 570.148.08. Each configuration used 5 warmups followed by 30 timed measurements, batch size 8 and 1,024-token prefill. The reported statistic is the median.
The Transformers 4.56.2 run was measured on 8 October 2026; the retained Transformers 5.19.0 run on 9 October 2026. Each ratio compares the measured configuration with its baseline in that run.
Both measurements flagged possible GPU contention after timing, although no other GPU applications were recorded. The timings should not be interpreted as independently confirmed uncontended measurements.
Evaluated quality scope: Transformers 4.56.2
Evaluated outputs were bit-identical across 292 non-overlapping 1,024-token WikiText-2 raw test windows, 5,153 LAMBADA test examples and 32 fresh 1,024-token WikiText-2 probe windows.
WikiText-2 test perplexity was 14.651782 for both configurations. LAMBADA last-word accuracy was 52.513099% for both. The bit-identical statement is limited to the evaluated outputs in this suite.
Free-running greedy generation was not evaluated, and incremental decode timing was not reported. These results do not establish equivalence for arbitrary inputs or generated answers.
Version sensitivity: Transformers 5.19.0
The retained run with Transformers 5.19.0 measured 27.252048 ms baseline and 34.502640 ms with SpinProof: 0.789854×, rounded to 0.7899×. This is a regression, not a speedup.
Its evaluated quality checks passed the registered bounds but were not bit-identical. The bit-identical result from Transformers 4.56.2 must not be carried over to this run. Free-running greedy generation was not evaluated in this run either.
Scope and limitations
These observations cover the stated FP16 model, hardware, software versions and evaluation suite. Quality agreement on evaluated records is not a universal quality guarantee. Prefill timing does not establish incremental decode performance, production serving throughput or performance across other models and software environments.