We describe Spindrift, A streaming post-processing layer that collapses short-horizon token repetitions introduced by nondeterministic decoding paths.. Inference stacks that batch requests or run across multi-GPU splits are not bit-exact across replicas.
We specify a pre-registered protocol for Given the same open-weights model, the same prompt, and temperature=0 settings, do three widely-used inference stacks (vLLM, llama.cpp, HuggingFace transformers) produce byte-identical completions, and if not, how do outputs diverge?
We present an independent replication of TurboQuant (Zandieh and Mirrokni, ICLR 2026), a two-stage KV cache quantization method for large language model inference combining Lloyd-Max optimal scalar quantization with random orthogonal rotation and 1-bit Quantized Johnson-Lindenstrauss residual correction. We implement the full algorithm from scratch in PyTorch and integrate it into the Llama-3.