---
title: SpinCache: measured decode at 128K context | SpinForge
url: https://spin-forge.com/proof/spincache
description: 2.6141×: decode speedup, four-bit cache vs FP16. Llama-3.1-8B-Instruct on an RTX PRO 6000 Blackwell Server Edition: 49.24 to 18.84 ms/token, a 2.6141×…
---

AI INFRASTRUCTURE · MEASURED BENCHMARK

# SpinCache: measured decode at 128K context

Llama-3.1-8B-Instruct on an RTX PRO 6000 Blackwell Server Edition: 49.24 to 18.84 ms/token, a 2.6141× speedup with a four-bit cache. Fixed random inputs; prefill excluded. Capacity and quality were measured separately.

2.6141× decode speedup, four-bit cache vs FP16

49.24  18.84 ms/token at batch 1, 131,072-token context

64 steps timed full-model decode after 8 warmups

Fixed inputs random input tokens; prefill excluded

Full-model decode latency, measured 8 October 2026. Lower latency is better.

| Cache configuration | ms/token | Tokens/s |

| FP16 baseline | 49.240969 | 20.308292 |

| SpinCache four-bit | 18.836422 | 53.088639 |

- The speedup is baseline latency divided by measured latency: 2.614136× before rounding. These are full-model decode timings, not an isolated cache-operation benchmark.
- The timing workload uses fixed random input tokens. It is not a natural-language quality evaluation.

## Measurement conditions

Model: Llama-3.1-8B-Instruct (NousResearch/Meta-Llama-3.1-8B-Instruct). Hardware: NVIDIA RTX PRO 6000 Blackwell Server Edition. Batch size 1; starting context 131,072 tokens. The same Transformers engine was used for both configurations.

Each configuration performed prefill first, then 8 warmup steps and 64 timed full-model greedy decode steps. Prefill time is excluded. GPU work was synchronized at the timing boundaries.

Recorded software: PyTorch 2.13.0+cu130, CUDA 13.0, Python 3.10.12 and NVIDIA driver 580.126.09. A Transformers version was not recorded.

## Separate capacity measurement

With the same Llama-3.1-8B-Instruct model weights resident, complete sequence caches were allocated at 131,072-token context. The four-bit configuration allocated 15 caches versus 4 for FP16, a 3.75× allocation-capacity ratio.

This test measured storage allocation. It did not execute concurrent serving or establish serving throughput. FP8 is a capacity baseline here; no FP8 full-model decode timing is claimed. The retained full-model batch-8, 128K test did not fit GPU memory.

Complete sequence caches allocated at 131,072-token context; not concurrent requests served.

| Cache configuration | Allocated sequence caches |

| FP16 | 4 |

| FP8 | 9 |

| SpinCache four-bit | 15 |

## Separate quality evaluation: Llama-2-7B

A separate WikiText-2 raw test evaluation used Llama-2-7B (NousResearch/Llama-2-7b-hf), not the Llama-3.1-8B-Instruct model used for timing. It evaluated 166 non-overlapping 2,048-token windows and 339,802 scored tokens on the same hardware and recorded software environment.

FP16 perplexity was 5.472060530; four-bit perplexity was 5.483590895. The relative increase was 0.210713%, rounded to +0.211%. Lower perplexity is better.

This result does not establish Llama-3.1-8B quality, 128K-context quality, generated-answer equivalence or quality parity for the timing workload.

## Scope and limitations

These results describe the stated model, hardware, inputs and configurations. They do not establish production request latency, concurrent serving performance or a universal speedup. The latency, allocation-capacity and quality measurements answer different questions and must be interpreted separately.

[Explore SpinCache](https://spin-forge.com/spincache)

## More ai infrastructure benchmarks

- [SpinProof: measured prefill and evaluated quality](https://spin-forge.com/proof/spinproof)

[All results, charts and papers](https://spin-forge.com/proof)

Results describe the stated inputs, hardware and conditions and are not a guarantee of results in your environment. See the [Terms of Use](https://spin-forge.com/terms).