Why Your LLM is a Stochastic Process (And Why Temperature=0 Doesn't Save You)

It is one of the most common assumptions in AI engineering: if you set the temperature to exactly zero, your Large Language Model (LLM) becomes a deterministic function. You feed it the exact same prompt, and it yields the exact same output. You write test suites verifying JSON structures, build regression tests for code-generation agents, and benchmark retrieval-augmented generation (RAG) pipelines, all under the comfort of this absolute reproducibility.

It is a lie.

In production, you will eventually see a prompt at temperature=0 return a different JSON schema, fail a test it has passed a hundred times, or output a slightly mutated sentence. Your first instinct will be to blame network latency, model updates, or prompt injection. But the root cause is far more fundamental.

Your LLM is a stochastic process. It is built on mathematics that govern random physical dynamics, and it runs on hardware that is physically incapable of guarantees.

This post is a deep dive into the mechanics of LLM stochasticity. We will unpack the formal mathematics of next-token generation, show how GPU thread scheduling and parallel optimization algorithms break determinism at a hardware level, connect modern speculative decoding to textbook Monte Carlo rejection sampling, and explore why multi-agent pipelines behave like controlled stochastic systems.

Contents

1. The Mathematical Frame: Auto-Regression as a Markov Chain

To understand why LLMs behave randomly, we must first look at the mathematical model of language generation. An auto-regressive language model is a discrete-time, discrete-state stochastic process $(X_t{)}_{t\ge 1}$ moving through a state space defined by the model's vocabulary $V=\{v_1,v_2,\dots ,v_N\}$.

When an LLM generates text, it does not output a whole document at once. It samples one token at a time, appending that token to its context, and using the expanded context to calculate the probability distribution of the next token.

Transition Dynamics

The probability of selecting token $v_j$ at step $t+1$ is conditioned on the entire history of generated tokens:

where $f_{\theta }$ represents the transformer forward pass parameterized by weights $\theta$. The output vector ${\mathbf{p}}_t=f_{\theta }(x_1,\dots ,x_t)$ lies on the $(N-1)$-dimensional probability simplex:

Each forward pass of the transformer projects the history of the conversation to a single point on this simplex. The sampling strategy (greedy, top-p, or top-k) then picks a vertex representing the next token.

Because the transition probabilities are conditional on the history, any microscopic deviation in the output probability simplex at step $t$ can shift the selection of token $X_{t+1}$. Once a different token is appended to the context, the history is altered. The process branches onto a completely different path, leading to radically different final text outputs.

The Simplex and the Softmax Bottleneck

The vocabulary of a modern LLM is massive—often between 50,000 and 256,000 tokens. The probability simplex ${\Delta }^{N-1}$ is therefore a high-dimensional space. However, the model represents tokens internally in a much lower-dimensional hidden space $d_{model}$ (usually between 2,048 and 12,288 dimensions).

To output probabilities, the final hidden state ${\mathbf{h}}_t\in {\mathbb{R}}^{d_{model}}$ is multiplied by the unembedding matrix $W_U\in {\mathbb{R}}^{N\times d_{model}}$ to produce logits ${\mathbf{z}}_t$, which are then passed through the softmax function:

This projection introduces the Softmax Bottleneck (Yang et al., 2018). Because the rank of the logit matrix is bounded by $d_{model}$, the model cannot represent arbitrary probability distributions over the vocabulary. The model's stochastic process is restricted to a lower-dimensional statistical manifold. This rank limitation means that the model's estimate of uncertainty (represented by the entropy of the output distribution) is inherently constrained, making it highly sensitive to minor numerical perturbations.

2. The GPU Layer: How Parallelism Breaks Determinism

If you set the temperature $T=0$, the model performs greedy decoding. It bypasses random sampling libraries entirely and selects the token with the highest logit:

In theory, this argmax operator should make the generation process deterministic. If the inputs are identical, the forward pass should yield the exact same logits, and the argmax should pick the exact same token.

To see why this fails, we have to look at the silicon under the hood.

Floating-Point Non-Associativity

Modern GPUs are massively parallel computing engines. They achieve high throughput by distributing matrix additions, multiplications, and reductions across thousands of arithmetic units (tensor cores and CUDA threads) executing concurrently.

These computations are performed using floating-point arithmetic (typically float16, bfloat16, or FP8 to optimize memory bandwidth). In mathematics, addition is associative: $(a+b)+c=a+(b+c)$. In computer science, floating-point addition is non-associative:

Because floating-point numbers have a fixed number of significand bits, rounding errors occur with every operation. The magnitude of the rounding error depends on the relative scale of the numbers being added. Consequently, the order in which a set of numbers is summed changes the final result.

Consider a simple reduction: summing a list of attention weights or logit values. In a parallel GPU kernel, the order of summation is determined by dynamic thread scheduling. Whichever warp finishes its computation first writes its partial sum to shared memory first. Because thread scheduling is subject to microscopic variations in hardware state, temperature, and memory bus congestion, the summation order varies between runs.

Run 1 Summation Path:  (Logit_A + Logit_B) + Logit_C  -->  z_i = 12.450001
Run 2 Summation Path:  Logit_A + (Logit_B + Logit_C)  -->  z_i = 12.450000

A difference of $10^{-6}$ in a logit seems trivial. However, if two tokens have nearly identical logits at the top of the distribution, this tiny arithmetic drift can flip the argmax, changing the token selected at step $t$ and derailing the determinism of all subsequent steps.

FlashAttention and Block Reductions

This non-determinism is exacerbated by modern memory optimizations. Standard attention computation scales quadratically with sequence length, requiring large memory transfers between high-bandwidth memory (HBM) and SRAM.

FlashAttention (Dao et al.) solves this by computing attention incrementally. It loads blocks of Query, Key, and Value matrices into SRAM, computes intermediate softmax scalings, and writes the results back to HBM without materializing the full attention matrix.

The algorithm tracks running partition statistics to compute softmax block-by-block:

Because the GPU partitions the work into threads that execute in a non-deterministic order depending on block-level thread scheduling, the sequence of scaling factors and accumulated sums shifts dynamically. The final attention activations carry subtle, non-deterministic numeric noise.

Dynamic Routing and API Complexity

If you run models via cloud APIs (such as OpenAI, Anthropic, or Azure), you are interacting with complex system-level architectures that add more layers of randomness:

3. Speculative Decoding as Monte Carlo Rejection Sampling

To combat the latency of auto-regressive generation, modern inference engines use speculative decoding (Leviathan et al., 2023). This technique is a beautiful bridge back to classical numerical methods: it is a direct application of rejection sampling, a fundamental Monte Carlo method.

The problem with large LLMs is that computing a forward pass to generate a single token is memory-bound. Speculative decoding bypasses this by pairing a large target model $p(x)$ with a small, fast draft model $q(x)$.

The Rejection Sampling Mechanics

The speculative decoding loop works as follows:

1. Draft Generation: The draft model $q(x)$ quickly generates $k$ candidate tokens $x_1,x_2,\dots ,x_k$ auto-regressively.
2. Parallel Evaluation: The target model $p(x)$ evaluates the joint log-probabilities of these $k$ tokens in a single parallel forward pass. This is fast because the weights only need to be loaded into memory once for all $k$ tokens.
3. Verification Step: For each candidate token $x_i$ (from $i=1$ to $k$ ), we compute the acceptance probability:

We then sample a random number $U\sim \text{Uniform}(0,1)$. If $U\le \alpha$, we accept the token and proceed to verify $x_{i+1}$. If $U>\alpha$, we reject the token, discard all subsequent candidate tokens, and sample a replacement token from the target model's residual distribution:

graph TD
    A["Draft Model q(x) generates k tokens"] --> B["Target Model p(x) evaluates tokens in parallel"]
    B --> C{"Check alpha = p(x_i)/q(x_i)"}
    C -- "Accept (U <= alpha)" --> D["Verify next token x_i+1"]
    C -- "Reject (U > alpha)" --> E["Discard remaining tokens"]
    E --> F["Sample new token from residual distribution"]

Proof of Lossless Recovery

Despite relying on a smaller, less capable draft model to generate candidates, the output of speculative decoding is mathematically guaranteed to follow the exact probability distribution of the large target model $p(x)$.

Let's prove this for a single token step. The probability of outputting a token $x$ through acceptance is the probability that the draft model generates it, multiplied by the probability that the target model accepts it:

If $p(x)\ge q(x)$, this is:

If $p(x)<q(x)$, this is:

Thus, the probability of outputting $x$ via acceptance is $\min (q(x),p(x))$.

The probability of outputting a token $x$ via rejection is the total probability of rejecting any token, multiplied by the probability of sampling $x$ from the residual distribution $p^{\prime }(x)$:

Since ${\sum }_y\max (0,p(y)-q(y))=1-{\sum }_y\min (q(y),p(y))$, the normalization terms cancel out perfectly, leaving:

Adding the acceptance and rejection paths together:

The math is elegant and clean. However, notice the dependency: speculative decoding relies on sampling a random variable $U$. Even if both $p(x)$ and $q(x)$ are configured to run with greedy settings, the engine's internal speculative verification loops introduce explicit stochastic checks, adding another vector of randomness to the system.

4. Agent Pipelines: MCP and Controlled Stochastic Processes

When we transition from a single LLM call to an agent pipeline—where the model loops through actions, calls tools via the Model Context Protocol (MCP), and updates its context based on external feedback—we are looking at a controlled stochastic process.

The State Space of an Agent

Let the state of the agent system at step $t$ be represented by the tuple $S_t=(X_t,Y_t)$, where:

• $X_t\in V^{\ast }$ is the text state (the chat history and prompt context).
• $Y_t\in \mathcal{E}$ is the external environment state (database rows, file systems, API states).

The transition dynamics occur in two phases:

1. Agent Action Selection: The model samples a tool call action $A_t$ from its output distribution:

2. Environment Transition: The tool execution mutates the external environment state and returns an observation $O_t$:

3. State Update: The conversation state appends the action and the observation:

Noise Decomposition in Agent Runs

In a deterministic software system, the variance of the output state is zero. In an agent pipeline, the total variance of the system outcome ${\sigma }_{\text{system}}^2$ can be decomposed into three distinct components:

1. Model Noise ( ${\sigma }_{\text{model}}^2$ ): The variance introduced by token sampling (if $T>0$ ) or GPU floating-point variations in logit projections (if $T=0$ ).
2. Tool Noise ( ${\sigma }_{\text{tool}}^2$ ): External stochasticity in the tools themselves (network latency variations, database state shifts, web scraping failures).
3. Interaction Noise ( ${\sigma }_{\text{interaction}}^2$ ): The compounding effect where a small logit variation shifts a tool parameter slightly (e.g., changing a search query), causing the tool to return a radically different payload, which then redirects the agent's entire trajectory.

Because of ${\sigma }_{\text{interaction}}^2$, agent pipelines are highly unstable. A tiny floating-point drift in self-attention on layer 32 can scale into a failed transaction, a database write error, or a loop of useless API retries.

5. Engineering with Randomness: Practical Mitigation

If you are building production systems on top of LLMs and MCP servers, you cannot debug them under the assumption of determinism. You must treat them as stochastic systems to be monitored, calibrated, and stabilized.

Here is a practical framework to build resilience against LLM noise:

1. Track Entropy and Perplexity in Real-Time

Do not just log the generated text; log the token-level metadata.

• Sudden Entropy Drops: If the Shannon entropy of the next-token probability distribution drops sharply mid-generation, the model has committed to a specific path. Use this as a confidence signal.
• Perplexity Spikes: If the perplexity of the generated tokens spikes on RAG search results, it indicates that the retrieved context is surprising to the model. Use this to flag retrieval failures.

2. Isolate Noise via Mocking and Frozen Seeds

When writing unit and integration tests for agents:

• Mock the Environment: Fix the tool transition probability $P(Y_{t+1}\mid Y_t,A_t)$ by mocking all MCP server returns.
• Seed the Engine: Set explicit seeds in your model configurations. While this does not guarantee absolute determinism due to hardware factors, it drastically reduces warp scheduling variance when executing on the same GPU cluster.

Here is how you configure seeds across the major LLM provider SDKs, along with their respective caveats:

OpenAI (Python SDK)

OpenAI supports a seed parameter. When configured, the API attempts to return deterministic completions. It also returns a system_fingerprint in the response header. If the fingerprint changes, it indicates that OpenAI changed the underlying hardware configuration, breaking determinism.

from openai import OpenAI

client = OpenAI()

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Explain the Softmax Bottleneck"}],
    temperature=0.0, # Must be 0 for seed to have max effect
    seed=42          # The frozen seed
)

print(f"Output: {response.choices[0].message.content}")
print(f"System Fingerprint: {response.system_fingerprint}")

Google Gemini (GenAI SDK)

In the Google Gemini API, you pass the seed parameter inside the configuration block.

from google import genai
from google.genai import types

client = genai.Client()

response = client.models.generate_content(
    model='gemini-2.5-flash',
    contents='Explain the Softmax Bottleneck',
    config=types.GenerateContentConfig(
        temperature=0.0,
        seed=42 # Frozen seed
    )
)

print(response.text)

Anthropic Claude (API Caveat)

As of mid-2026, Anthropic's Claude API does not support a seed parameter. Due to Anthropic's distributed load-balancing system and speculative execution optimizations, you cannot guarantee path-level reproducibility.

To minimize variance with Claude, you must set temperature=0.0 to force greedy decoding (mode-seeking), but you must write your agent parser to handle spelling or syntax variations resiliently.

Local Models (vLLM / Transformers)

If you run models locally or on private clusters, you can enforce determinism at the inference engine level. In vLLM, you set the seed in the sampling parameters:

from vllm import LLM, SamplingParams

llm = LLM(model="meta-llama/Meta-Llama-3-8B-Instruct")
sampling_params = SamplingParams(
    temperature=0.0,
    seed=42 # Set the seed on the sampler
)

outputs = llm.generate(["Explain the Softmax Bottleneck"], sampling_params)

3. Move from Point Metrics to Distributional Evals

Evaluating a RAG pipeline on a single generation per test query is a statistical trap. It conflates generation variance with retrieval quality.

• Pass@k Metrics: Generate $k$ independent completions for each test prompt. Compute the fraction of paths that successfully return the correct answer.
• Variance Decomposition: Run your evaluations multiple times under identical contexts to measure ${\sigma }_{\text{model}}^2$ independently from ${\sigma }_{\text{tool}}^2$. If model variance dominates, adjust your sampling hyperparameters (e.g., lowering temperature or restricting nucleus top-p thresholds).

Conclusion: Respecting the Jagged Path

Just as physical systems are governed by the continuous, jagged trajectories of Brownian motion, modern AI systems are shaped by the discrete, jagged paths of token-level stochastic processes.

Kiyosi Itô did not try to smooth out Brownian motion; he built a calculus that embraced its roughness. As software engineers and data architects, our job is not to force LLMs into deterministic boxes. Our job is to build systems, test suites, and databases that respect their statistical nature.

Bibliography

• Dao, T., Fu, D. Y., Ermon, S., Rudra, A., & Ré, C. (2022). FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness. Advances in Neural Information Processing Systems (NeurIPS), 35, 16344-16359.
• Leviathan, Y., Kalman, T., & Matias, Y. (2023). Fast Inference from Transformers via Speculative Decoding. International Conference on Machine Learning (ICML), PMLR, 19274-19286.
• Yang, Z., Dai, Z., Salakhutdinov, R., & Cohen, W. W. (2018). Breaking the Softmax Bottleneck: A High-Rank RNN Language Model. International Conference on Learning Representations (ICLR).
• Belrose, N., Furman, Z., Smith, L., Halawi, D., Ostrovsky, I., McKinney, L., Biderman, S., & Steinhardt, J. (2023). Eliciting Latent Predictions from Transformers with the Tuned Lens. arXiv:2303.08112.

Related Posts

Interested in more technical deep dives? Check out these related articles:

• Truth is Cold: LLM Temperature and Data-Driven Decision Making - Understanding precision vs. creativity in computational language systems
• The Quiet Genius Who Made Randomness Calculable - Explore the history of Kiyosi Itô and the birth of stochastic calculus
• When Randomness Met Calculus: A Practitioner's Guide to Stochastic Differential Equations - Hands-on Python guide to simulating physical and financial SDEs

Get in Touch

Need help implementing numerical methods or debugging non-determinism in your LLM agent pipelines?

Connect with me:

• 📧 Email: [email protected]
• 🐦 Twitter/X: @TheDataGuyPro
• 💼 LinkedIn: Muhammad Afzaal
• 💻 GitHub: @mafzaal
• 🎥 YouTube: @TheDataGuyPro
• 🎧 Podcast: TheDataGuy Show

Whether you're looking for consulting services, code optimization, or want to discuss AI systems architectures, I'd love to hear from you!