SpinCache: measured decode at 128K context
Llama-3.1-8B-Instruct on an RTX PRO 6000 Blackwell Server Edition: 49.24 to 18.84 ms/token, a 2.6141× speedup with a four-bit cache. Fixed random inputs; prefill excluded. Capacity and quality were measured separately.
| Cache configuration | ms/token | Tokens/s |
|---|---|---|
| FP16 baseline | 49.240969 | 20.308292 |
| SpinCache four-bit | 18.836422 | 53.088639 |
- The speedup is baseline latency divided by measured latency: 2.614136× before rounding. These are full-model decode timings, not an isolated cache-operation benchmark.
- The timing workload uses fixed random input tokens. It is not a natural-language quality evaluation.
Measurement conditions
Model: Llama-3.1-8B-Instruct (NousResearch/Meta-Llama-3.1-8B-Instruct). Hardware: NVIDIA RTX PRO 6000 Blackwell Server Edition. Batch size 1; starting context 131,072 tokens. The same Transformers engine was used for both configurations.
Each configuration performed prefill first, then 8 warmup steps and 64 timed full-model greedy decode steps. Prefill time is excluded. GPU work was synchronized at the timing boundaries.
Recorded software: PyTorch 2.13.0+cu130, CUDA 13.0, Python 3.10.12 and NVIDIA driver 580.126.09. A Transformers version was not recorded.
Separate capacity measurement
With the same Llama-3.1-8B-Instruct model weights resident, complete sequence caches were allocated at 131,072-token context. The four-bit configuration allocated 15 caches versus 4 for FP16, a 3.75× allocation-capacity ratio.
This test measured storage allocation. It did not execute concurrent serving or establish serving throughput. FP8 is a capacity baseline here; no FP8 full-model decode timing is claimed. The retained full-model batch-8, 128K test did not fit GPU memory.
| Cache configuration | Allocated sequence caches |
|---|---|
| FP16 | 4 |
| FP8 | 9 |
| SpinCache four-bit | 15 |
Separate quality evaluation: Llama-2-7B
A separate WikiText-2 raw test evaluation used Llama-2-7B (NousResearch/Llama-2-7b-hf), not the Llama-3.1-8B-Instruct model used for timing. It evaluated 166 non-overlapping 2,048-token windows and 339,802 scored tokens on the same hardware and recorded software environment.
FP16 perplexity was 5.472060530; four-bit perplexity was 5.483590895. The relative increase was 0.210713%, rounded to +0.211%. Lower perplexity is better.
This result does not establish Llama-3.1-8B quality, 128K-context quality, generated-answer equivalence or quality parity for the timing workload.
Scope and limitations
These results describe the stated model, hardware, inputs and configurations. They do not establish production request latency, concurrent serving performance or a universal speedup. The latency, allocation-capacity and quality measurements answer different questions and must be interpreted separately.