Introducing Background Temperature to Characterise Hidden Randomness in Large Language Models

Introducing Background Temperature to Characterise Hidden Randomness in Large Language Models Alberto Messina1and Stefano Scotta1 1RAI - Radiotelevisione Italiana, Centre for Research, Technological Innovation and Experimentation (CRITS) October 6, 2025 Abstract Even when decoding with temperature T = 0, large language models (LLMs) can produce divergent outputs for identical inputs. Recent work by Thinking Machines Lab highlights implementation-level sources of nondeterminism, including batch-size variation, kernel noninvariance, and floating-point non-associativity. In this short note we formalize this behavior by introducing the notion of background temperature Tbg , the effective temperature induced by an implementation-dependent perturbation process observed even when nominal T = 0. We provide clean definitions, show how Tbg relates to a stochastic perturbation governed by the inference environment I , and propose an empirical protocol to estimate Tbg via the equivalent temperature Tn ( I ) of an ideal reference system. We conclude with a set of pilot experiments run on a representative pool from the major LLM providers that demonstrate the idea and outline implications for reproducibility, evaluation, and deployment. 1 Introduction A common assumption in LLM deployment is that setting the decoding temperature to T = 0 (greedy decoding) ensures determinism. However, empirical evidence shows output variability persists under nominally deterministic settings. The recent work in [ 3 ] argues that nondeterminism in LLM inference often arises from practical systems issues such as varying batch sizes and the lack of batch-invariant kernels, along with floating-point non-associativity and reduction-order effects. This paper proposes a rigorous framing of such effects via the notion of abackground temperature. Contributions. (i) A concise formal model that addresses the phenomenon of nondeterminism as a stochastic effect on the output probability; (ii) a formal definition of background temperature Tbg ; (iii) the outline of a practical protocol to estimate Tbg ; (iv) a set of pilot studies illustrating the concept. 2 Related Work The recent work by Thinking Machines Lab provides a systems-first analysis of LLM nondeterminism, emphasizing batch-size variation and batch-invariant kernels for inference; they also explain how floating-point non-associativity and reduction ordering contribute to variability. [ 3 ]. In addition to this work, several recent studies have quantified non-determinism in large language model outputs even under settings intended to be deterministic (e.g. temperature T= 0, fixed seeds). For example: 1 • Atil et al. (2025) [ 1 ] systematically evaluate multiple LLMs configured under deterministic settings across zero-shot and few-shot tasks. They observe large accuracy variations (up to 15%) across runs with the same input, and show that even the string outputs are often not identical. • Song et al. (2024) [ 10 ] explore how evaluation practices often ignore variability arising from different decoding configurations (greedy vs sampling). They show that even for greedy decoding, evaluation metrics vary, and that alignment methods can help reduce sampling variance. • Ouyang et al. (2023) [ 6 ] analyze code generation benchmarks and show that many coding tasks produce different code outputs across repeated prompt invocations, even when using T = 0. This confirms that deterministic temperature settings do not guarantee output consistency. These works align closely with observations from Thinking Machines Lab’s blog [ 3 ] about system-level implementation factors (batch size, kernel non-invariance, floating point nonassociativity, etc.) causing output variation even under nominally deterministic decoding. While prior work largely documents the existence and magnitude of non-determinism, there remains a gap in formalizing this behavior in terms of an equivalent temperature transformation functional and in proposing standard protocols to measure the effective background randomness. Our work addresses this by introducing the notion of an equivalent temperature Tn ( I ) and its expectation Tbg . In the next sections, we transition from formal definitions to a concrete empirical protocol aimed at estimating an equivalent temperature Tn ( I ) induced by implementation noise, and ultimately the background temperature. To give a concrete description of what an overall measurement protocol for Tbg would look like, we first describe general criteria for selecting prompts and datasets that are sensitive to small perturbations in model behaviour (including general, task-oriented, and adversarial/synthetic prompts). We then introduce the actual measurement protocol, made up of reference runs under known nonzero temperature settings to calibrate output variability. Following this, based on a suite of quantitative metrics - such as exact-match frequency, first-divergence token index, edit-distance or string similarity, distributional divergence (e.g. JS or KL) over next-token / top-k probabilistic outputs, and entropy/confidence measures - we finally outline a fitting procedure to infer Tn ( I ) by minimizing divergence between outputs under noisy T = 0 runs and reference nonzeroT runs, and describe how to aggregate over I to compute Tbg with statistical confidence. 3 Preliminaries and Notation Let D denote the token vocabulary with size |D| . At generation step i , the model produces logits z∈R|D| and associated probabilities P ( t ) ∈ [0 , 1] that the i -th token in the sequence is the t−th token in D, such that P|D| t=1 P(t) = 1 via softmax: P(t) = P(τt|τ<i) = exp(zt) Ps∈Dexp(zs)for t= 1,...,|D|,(1) where τt denotes the t -th token in D and P ( τt|τ<i ) is the probability of generating τt given the sequence of tokens generated up to the i -th token. At T = 0, the conventional model is greedy decoding by argmax: τi= arg max τ∈DP(τ|τ<i).(2) Decoding at temperature T > 0 is equivalent to do the same operation but with modified logits ˆz∈R|D|: ˆ PT(τi|τ<i) = expˆzi Ps∈Dexpˆzs.(3) 2 Then the i -th token is distributed as some Categorical random variable depending on the probability distribution above, i.e. τi∼Categorical(ˆ P(τ|τ<i)).(4) Logits are modified through the randomization effects that are included in the decoding process by the specific LLM implementation. In standard autoregressive language models, the decoding temperature parameter modifies the randomness of next-token selection by adjusting the probability distribution derived from logits. Typically, one scales or transforms the raw (pre-softmax) logits via a temperature parameter and then passes them through softmax to obtain the final distribution for sampling or greedy selection. In general, but as a sufficient assumption for the sake of this work, lower temperatures concentrate probability mass on the most likely tokens, making output more deterministic, while higher temperatures flatten the distribution and increase variability. Equivalently, this can be seen as the result of the application of an opportune temperature transformation functional FT: FT:R|D|→R|D|,ˆ P=FT(P),(5) with the ideal identity limit F0 ( P ) = P . Many implementations use temperature T so that the model effectively computes something like FT ( P ), a functional transformation of the original token probability vector P , where T = 0 corresponds (ideally) to purely greedy decoding, and T > 0 allows stochastic sampling. 4 Modelling Intrinsic Nondeterminism at T= 0 As noted by authors in [ 3 ], real systems exhibit implementation-dependent perturbations even under T = 0. Let I∈ I denote the inference environment (batch size and composition, concurrency/load, hardware/backends, kernel choices, numeric precision, reduction ordering, etc.) and F′ T the temperature transformation functional of the real system. We model a perturbation ϵI , mapping probability distribution over the set D to probability distribution over the same set, that alters the effective distribution as: F′ 0(P) = ϵI(F0(P)) ≈ϵI(P).(6) While ϵI may differ only slightly from F0 ( P ), in regions where multiple tokens have similar probability mass, even slight changes can flip the argmax in (2) and thus the emitted token sequence. 5 Equivalent Temperature and Background Temperature We posit that the perturbation in (6) behaves as if decoding were performed by an inference environment - free (ideal) system at a nonzero equivalent temperature Tn(I) : F′ 0(P)≈ϵI(P)≈FTn(I)(P).(7) This motivates the following definition. Definition (Background temperature). The background temperature of an LLM implementation is the expected equivalent temperature induced by the inference environment under nominal T= 0: Tbg ≜EI∈I[Tn(I)] .(8) Intuitively, Tbg captures the implicit randomness in a deployment stack even when the user selects T= 0. 3 6 Estimating Tn(I)and Tbg Empirically The problem with the definition given in (8) is that the inference environment-free (ideal) system may be not at hand. In fact, the key challenge in estimating Tn ( I ) is that it requires comparing to a perfect, deterministic reference - which may not exist in practice. To make Tn ( I ) calibration feasible without an unattainable ideal, one can first identify a quasi-ideal environment: for example, by using inference pipelines with batch-invariant kernels (in normalization, matrix multiplication, attention), fixed numeric precision, minimal or single-request concurrency, and deterministic configuration flags. Thinking Machines Lab demonstrates that replacing standard kernels with batch-invariant ones drastically reduces output divergence under zero temperature [ 3 ] Similarly, [ 9 ] show that floating-point non-associativity and asynchronous parallel reductions are major sources of run-to-run variability, and that enforcing deterministic alternatives significantly stabilizes inference and scientific computing pipelines. Based on this evidence, one can anchor measurement of Tn ( I ) relative to such quasi-ideal baselines, or employ multiple such baselines (differing in hardware, kernel implementation, or precision) to absorb uncertainty. Further, measuring various output statistical distributions (rather than only output strings) allows matching of environments I to baselines via statistical divergence metrics, reducing sensitivity to rare argmax flips. Reporting Tn together with such baseline variances yields operationally meaningful estimates even in the absence of a perfect oracle. Another practical way to assess the background temperature of an online model (e.g. ChatGPT) is to use a local installation of another model (e.g. Llama) as a benchmark reference. The local model must be configured to be as deterministic and stable as possible—fixed precision, consistent batch sizes, kernel implementations that do not alter behavior when batch shape changes, deterministic reduction orders, disabled non-deterministic/autotuned operations. This reference becomes a baseline environment that approximates “ideal behavior”. Then, by comparing output distributions from the online model versus those from the stable local model, one can compute how far the online model’s behavior diverges, for example via measures like Jensen-Shannon divergence or KL divergence. By finding what temperature setting of the local model would make its distribution match the diverged distribution of the online model, it is possible to infer an equivalent temperature for the online model in that environment. Repeated across many prompts and local configurations, this yields an estimate of the online model’s background temperature, together with uncertainty bounds. This method avoids relying on an unattainable perfect system, by using the best stable reference you can build. With these considerations in mind, we can outline a practical protocol to estimate Tn ( I ) and Tbg which is pictorially described in Figure 1. 6.1 Prompt Sets and Datasets The first element of the protocol is constituted by a relevant prompt set Π, an element of the theoretical set of all the possible combinations of prompts P . To explore the full range of behavior of the system under test, the suggestion is to use a diverse evaluation suite, e.g.: •General generation prompts (short/long, common/rare vocab). • Task benchmarks: QA (e.g., SQuAD[ 7 ]/TriviaQA[ 4 ]), summarization, close, and shortformat classification. Code-generation prompts if applicable. • Edge/adversarial prompts: long contexts, rare tokens, near-ties among topk token probabilities. •Synthetic prompts engineered to create finely balanced next-token choices. 4 Figure 1: Measuring protocol. 6.2 Controlling the Inference Environment I Run repeated inference (e.g., M≥ 50 per prompt) at T = 0 while varying I along axes known to influence nondeterminism: • Batch structure: batch size, e.g. ∈ { 1 , 2 , 4 , 8 , 16 , 32 ,...} ; co-batching with other prompts vs. serial. •Concurrency/load: single request vs. many simultaneous requests. • Hardware/backends: GPU types, CPU vs. GPU, precision (fp16/bf16/fp32), kernel implementations (batch-invariant vs. standard). •Numerics: reduction order, deterministic flags in frameworks, fused vs. unfused kernels. For remote systems, for which it may be impossible or impractical to govern the inference environment, one can assume that prolonged and repeated operation is a good way to sample the inference environment statistical distribution. 6.3 Reference Runs at Known Temperatures Under a stable environment Istable , e.g. a local anchor system, run the same prompts e.g. at a grid of T∈ { 0 ,..., 1 , . . .} to build a mapping between T and output-variability statistics. As noted earlier, this stable environment can either be a specific configuration of the system under test or another anchor used as reference. Given that the anchor configuration is supposed to be stable for what concerns the inference environment, a lower number K of runs for each prompt in the prompt set should suffice. 6.4 Variability Metrics The key element of the protocol is the set of metrics used to obtain the association between the sought-for background temperature parameter for the system under test and the reference measurements on the anchor system. Since th Tbg is thought as a genereic high level account of the system’s nondeterminism, metrics should be content-agnostic. Furthermore, since different systems are trained independently, it is practically certain that the same prompt would produce 5 different outputs even under strict deterministic configurations. For example, for each prompt, and across the M(or K) runs of Figure 1, compute process parameters like e.g.: •Exact-match rate: fraction of runs producing identical strings for the same prompt. •First-divergence index: position of first token mismatch across pairs of runs. •Edit distance first order and second order statistics between different outputs. • Distributional divergence: e.g., symmetrized KL or JS divergence between empirical next-token distributions (top-k) across runs. •Entropy of next-token distributions. Then, for each variability metric computed across the runs, construct a multidimensional distribution f that captures the values of the variability metrics for the system considered. In particular, we’ll denote by fT ( Istable ) and g ( I ) respectively the distribution of the variability metrics for the reference system when the temperature is T and for the system under test set at temperature 0. Note that these distributions depend on multiple factors, including the specific LLMs used; for notational simplicity, we omit these dependencies. 6.5 Estimators for Tnand Tbg As explained in Section 6 the ideal reference system does not exist. However, it is possible to estimate Tn using some reference model running in an environment Istable as stable as possible. In particular, for a reference LLM ℓ , it is possible to compute an estimator ˆ Tℓ n = ˆ Tℓ n ( I, Π) of Tn , for each I in the set of environments considered ˜ I ⊆ I and each Π in the set of all the collections of prompts considered ˜ P ⊆ P, as ˆ Tℓ n= arg min T≥0DfT(Istable), g(I),(9) where D is a chosen divergence (e.g., JS or KL divergence, or a weighted combination) applied to the variability distributions g ( I ) and fT ( Istable ), corresponding respectively to the system under test and to the reference system based on ℓ (see Section 6.4). Therefore, it is possible to compute an estimate ˆ Tbg =ˆ Tbg(ℓ) of Tbg, for each ˆ Tn, as ˆ Tbg(ℓ) = 1 |˜ I| 1 |˜ P| X I∈˜ I X Π∈˜ P ˆ Tℓ n(I, Π),(10) where |˜ I| and |˜ P| denote, respectively, the number of all the I and Π considered. To further improve robustness, we repeat the same process across a set of different reference LLMs L and take the average1 Tbg =1 |L| X ℓ∈L ˆ Tbg(ℓ),(11) where |L| denotes the number of different ℓ (LLMs) used. Theoretically, as the set of reference LLMs L , prompts, environments, and variability metrics grows, we can expect Tbg to converge to the true Tbg, as defined in (8). 1 Beyond the average estimate, the availability of multiple reference LLMs and configurations also allows the computation of higher-order moments and confidence intervals, providing a more precise characterization of the uncertainty associated with this kind of estimate. 6 6.6 Engineering to Reduce Tbg Once for a certain system the background temperature Tbg is available, several mechanisms can be put in place to mitigate its effect. For example, empirical and systems work suggests several interventions: • Batch-invariant kernels for core ops (matmul, attention, RMSNorm) to prevent batchshape–dependent numerics [3]. •Deterministic reductions and stable accumulation orders where feasible [9]. • Consistent pipelines: fix kernel configs across shapes; avoid opportunistic algorithm switching that alters reduction paths [8]. •Deterministic flags in frameworks and careful precision selection [9]. • Operational controls: cap concurrency or shape buckets to reduce co-batching variability [3]. Ablation studies can further determine what intervention is impacting the most on the background temperature. This transforms the outlined protocol into an iterative practice aimed at controlling the nondeterministic characteristics of the system in use, as opposed to a mere observation of an empirical phenomenon. 7 Pilot Experiments In this section, we present some experiments to validate the theory presented in this work. In particular, in Section 7.1, we present a simple pipeline for estimating the background temperature for a given model. After that, we present additional experiments that could clarify and add elements to analyze the background temperature. 7.1 Basic pipeline for estimating Tbg Here we perform a pilot experiment to estimate Tbg for the OpenAI model gpt-4.1-nano accessed via the Microsoft Azure AI services with temperature T = 0 (i.e., considering it as System B in Figure 1). Note that, being the model used a via third part service, we can not control the inference environment I but only the temperature. The prompt set Π used is composed of the first 200 questions of the dataset truthful qa 2(see [5]). The reference LLM ℓ , playing the role of System A in Figure 1, is Hugging-Face LLM SmolLM3-3B 3 (see [ 2 ]). As outlined in previous sections, we selected representative temperature values Θ sampled in increments of 0 . 01 from 0 to 0 . 2, in increments of 0 . 05 from 0 . 2 to 0 . 5 and in increments of 0.1 from 0.5 to 1, i.e. Θ = {0,0.01,...,0.19,0.2,0.25,...,0.45,0.5,0.6,...,0.9,1}. For each T∈ Θ, we generated K = 32 responses, limited to 32 tokens, with the reference LLM for each of the 200 prompts in Π. As variability metric (see Section 6.4), we used the exact-match fraction, i.e. for each temperature considered and each prompt in Π, we computed the maximum fraction of identical answers among the 32 generated. In this way, for each T∈ Θ we obtained 200 values in the interval [1 / 32 , 1], which constitute the discrete distribution fT of the exact-match fraction for that temperature in the answers given by the reference LLM. In Figure 2, these distributions are graphically represented, showing how the density estimate shifts from a delta concentrated at 1 when the temperature is 0 - indicating that all answers are 2https://huggingface.co/datasets/truthfulqa/truthful_qa 3https://huggingface.co/HuggingFaceTB/SmolLM3-3B 7 Figure 2: Distribution of exact-match fractions obtained from the reference LLM answers. Top row (from left to right): histograms representing the distributions f0 , f0.2 and f1 . Bottom row: kernel density estimates of the exact-match fraction for all sampled temperatures in Θ. Note that for T = 0, the density is represented as a vertical line because all answers are identical, so the density is entirely concentrated at 1, forming a Dirac delta. 8 identical - to a distribution with most of its mass near 0, indicating that the answers tend to be unique. After computing the reference distributions fT for T∈ Θ of the chosen variability measure, we computed the same for the model for which we want to estimate Tbg , i.e., gpt-4.1-nano, accessed via the Microsoft Azure AI services. To do this, we prompted the model 100 times for each of the 200 prompts in Π, but this time setting the temperature at T= 0 and limiting the answers to 32 tokens, as done for the reference system. Then, analogously to the procedure for the reference system, for each prompt in Π we computed the maximum fraction of identical answers provided by gpt-4.1-nano. These 200 values, in [1 / 100 , 1], form the discrete distribution g (see Figure 3) that we need to compare with the reference distributions computed in system A (see (9)). Figure 3: Discrete distribution g of the fraction of identical answers given by the LLM under test, gpt-4.1-nano, to the prompts in Π. The distribution is shown both as histograms (with the y -axis on the left) and as a kernel density estimate (with the y-axis on the right). In order to compare the discrete distributions of observations, fT for T∈ Θ and g , we chose to use the Kolmogorov–Smirnov (K-S) distance, which is equal to 0 for identical distributions and 1 for completely different ones. The computed values of K-S distance are reported in Figure 4. From the values in Table 4 (b), we can conclude that the estimator of Tbg found in this experiment is ˆ Tbg ( ℓ ) = 0 . 05 (which, in this simple case, coincides with Tbg ), as this is the case where fT is closest to g , considering only the reference distributions computed from T∈ Θ. Figure 5 shows the two matching histograms. Ideally, this experiment should be repeated using a wider range of T values - especially lower ones - more prompts, fewer token limits, and different variability metrics (see Sections 6.4 and 6.5). However, the purpose of this pilot experiment was simply to demonstrate the full procedure to estimate Tbg. 7.2 Extending the reference model set L One of the possibilities for making the estimate of the Tbg more robust is to add reference models, i.e. extend the set L introduced in Section 6.5. In particular, we used the LLM Llama-3.2-3B-Instruct 4 and made it answer 32 times to the same 200 prompts (the same set 4https://huggingface.co/meta-llama/Llama-3.2-3B-Instruct 9