Contrary to common belief, nondeterminism is not set in stone for LLM inference. Horace He with Thinking Machines Lab published a great piece WITH a demonstration of an truly deterministic LLM.
TL;DR:
-
Even when setting temperature to 0 (greedy decoding == always choosing the highest-probability token), most LLM systems still produce non-deterministic outputs.
-
The commonly-cited culprit is floating-point non-associativity: the fact that, especially on GPUs, the order of operations (due to parallelism and atomic reduction) can affect the result (as in A+B+C ≠ C+A+B ≠ A+C+B).
-
But the most significant culprit is the lack of batch invariance in inference kernels: when serving multiple concurrent requests, the batch size and position can change for each inference step, which alters the outputs (even if the inputs and model weights are identical).
-
The authors defeated nondeterminism by rewriting and patching inference kernels to be batch-invariant. This enables truly deterministic LLM inference, with vLLM in their demonstration, even in a multi-user setup, at about a 50% slowdown in performance.
-
Having truly deterministic LLMs could unlock a range of applications, from reproducible research and consistent debugging to auditable production systems.
👉 Read the great post here: https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/