Attention is Holographic: A Functional Characterization — Detokenizing Transformers I: Theory

Attention is Holographic: A Functional Characterization Detokenizing Transformers I — Theory Russell Waller October 17, 2025 Abstract Transformers are widely viewed as discrete, token-based architectures, yet their selfattention mechanism already defines a normalized kernel operator analyzable in the continuous limit. We show that attention belongs to a natural class C of positive, row-stochastic interference operators admitting a constructive two-plate factorization (Ψ V, Ψ 1 )under mild kernel and feature assumptions. Within this class, robustness and invariance are characterized by three axes—scale ( s ), normalization quotient ( u ), and phase ( c )—whose joint constraints define a measurable “holographic set” of lawful behavior. Under shared positional structure and fair feature thinning (randomly dropping features with unbiased reweighting), we derive expectation bounds showing (i) monotone growth of point-spread width, (ii) first-order separability of numerator (content) versus denominator (normalization) perturbations, and (iii) phase neutrality when the feature law is rotationinvariant. We also formalize a discrete–continuous correspondence: softmax attention is a Riemann-sum approximation to a normalized kernel integral under standard regularity assumptions. Thus, Transformers approximate continuous interference operators whose behavior can be analyzed via first-order, diffusion-like trends in expectation and convex functional differentials, rather than relying solely on discrete heuristics. Empirical diagnostics and “scratch tests” validating these properties are presented in a companion paper. 1 Introduction Over the past several years, the Transformer architecture has become the dominant model for large-scale language, vision, and multimodal learning. Despite its empirical success, the mechanism at its core—self-attention—remains largely treated as a computational heuristic: a dot-product similarity normalized by softmax and applied over a discrete sequence of tokens. This paper asks a more fundamental question: What continuous operator does attention approximate? 1.1 From heuristics to operators We reinterpret self-attention as a member of a well-defined class of normalized, positive kernel operators. When expressed in Plate form—two compact summary states (Ψ V, Ψ 1 )accumulating numerator and denominator mass—the operator becomes an explicit quotient of interference integrals. This form allows direct functional-analytic treatment without assuming a discrete token basis, providing expectation-level results on robustness and invariance under mild regularity assumptions. 1.2 The three holographic axes We introduce three intrinsic coordinates that describe attention’s lawful behavior—lawful in the sense of exhibiting consistent, measurable trends in expectation under isotropy and fair thinning assumptions—along scale (s), normalization quotient (u), and phase (c). 1 •s -axis (scale): captures monotone broadening of point-spread width under fair thinning of feature channels (randomly dropping features with unbiased reweighting; see §3.1); •u -axis (use / normalization): exposes the quotient geometry, distinguishing content from normalization effects through orthogonal first-order responses; •c -axis (phase / frequency): characterizes invariance in expectation under rotational (RoPE-like) phase shifts of the embeddings. These axes arise from distinct analytical structures: blur-time expectation (scale), convex quotient geometry (use), and rotational symmetry (phase). Together they define a measurable region of graceful degradation—the holographic set. 1.3 Main results The paper establishes the following: 1. Monotone blur under fair thinning. Under shared positional structure, the expected point-spread width of attention grows monotonically with thinning rate p. 2. Quotient separability. Independent numerator and denominator perturbations produce orthogonal first-order effects on contrast and entropy, explaining dual robustness modes. 3. Phase neutrality. For isotropic feature laws, rotation of queries and keys commutes with thinning in expectation, yielding a flat phase-response profile. 4. Discrete–continuous correspondence. Transformers are Riemann-sum approximations of continuous normalized interference operators, establishing a correspondence under Lipschitz and positive-definite kernel assumptions. 1.4 Broader significance This framework replaces the token-by-token picture with a continuous, field-based model of inference. It defines a theoretical baseline—a reference manifold—against which the behavior of specific, trained models can be empirically measured and understood. It unifies discrete attention, kernelized variants, and continuous-time analogs within a single normalized interference formalism, yielding explicit expectation-level laws of robustness, compositionality, and invariance. It also motivates detokenized architectures, continuous-time Plate Machines, and a principled theory of holographic computation. Disclaimer and companion work. This paper develops the theoretical structure and expectation results only. All empirical evaluations—numerical “scratch tests,” simulated thinning experiments, and quantitative measurements of blur, entropy, and phase response—are presented in the companion paper (Detokenizing Transformers II: Holographic Signatures and Empirical Validation). 2 Attention as a Positive Kernel Operator This section defines the mathematical object of interest—the class of normalized positive kernel operators that includes standard softmax attention—and shows how the Plates representation arises directly from this form under explicit regularity assumptions. 2 Weights vs. readout. We separate the row-stochastic weight operator from the value readout: W(Q, K)∈RT×T,AQ,K(V):=W(Q, K)V. All structural properties (positivity, row-stochasticity, continuity) are stated for W ; the readout AQ,K is linear in V . We also assume a stabilizer δ > 0ensures a well-posed denominator (see §2.4), i.e. infqD(q)≥δ. Notation: ⟨a, b⟩ denotes the Euclidean inner product; ⊙ is the Hadamard (elementwise) product; expectations are taken over feature and thinning randomness unless otherwise stated; δ > 0is a small numerical stabilizer. 2.1 From discrete attention to normalized interference Let Q, K, V be matrices of queries, keys, and values, each with dimensions T×dq , T×dk , and T×dv, respectively. The standard single-head attention output is Y= softmax QK⊤ √dk!V. For a single query vector q, the output y(q)can be written componentwise as y(q) = Pjexpq·kj/√dkvj Pjexpq·kj/√dk, κ(q, k) = expq·k/pdk. This is a normalized positive kernel operator: y(q) = N(q) D(q), N(q) = X j κ(q, kj)vj, D(q) = X j κ(q, kj). Normalization ensures y(q)is bounded and invariant to uniform scaling of κ. Observation. The kernel κ is positive and smooth. Each attention head therefore implements a continuous positive map that combines global interference (via κ ) and local normalization (via D). 2.2 The admissible operator class C We define the class Cof operators to which all subsequent results apply. Class C(weights). A row-stochastic weight operator W : ( Q, K ) 7→ W ( Q, K ) ∈RT×T belongs to Cif: (i) Positivity: Wij ≥0for all i, j; (ii) Row-stochasticity: PjWij = 1 for each row i; (iii) Continuity: small perturbations in (Q, K)yield small changes in W(Q, K); (iv) Shared positional structure: queries and keys use the same positional map (e.g., sinusoidal or RoPE embeddings), ensuring approximate translation equivariance and enabling the circulant approximation (see §3.1); (v) Fair thinning: random removal of feature channels with unbiased reweighting leaves E[W]invariant. Intuitively, C is the family of normalized positive kernel weight smoothers that preserve probability mass and behave consistently under thinning. Standard softmax attention and its random-feature variants (e.g., Performer, Plates) satisfy these properties at the level of W ; the readout is AQ,K(V)=W(Q, K)V. 3 2.3 Assumptions on feature maps We explicitly list the assumptions under which the Plate factorization and all subsequent results hold. A1 (Positive-definite kernel). κis continuous, bounded, and positive definite. A2 (Feature realization). There exists m and a (possibly learned) feature map ϕ : Rdk→Rm such that κ(q, k)≈ϕ(q)⊤ϕ(k). A2 ′ (Signed or nonnegative features). When ϕ is nonnegative, all results below apply directly. When ϕ is signed, one can use a two-plate decomposition with positive and negative parts (Ψ± V,Ψ± 1); all results apply to the signed quotient N/D provided D > 0. A3 (Denominator stability). There exists δ > 0such that D ( q ) = ⟨ϕ ( q ) , Ψ 1⟩ + δ≥δ for all q , and under infinitesimal plate perturbations the relative fluctuation of D ( q )is O ( ε ). We use A3 in §4.2 to justify first-order expansions of the quotient. Remark. Exact nonnegative factorization does not exist for all kernels; the approximation error scales with m and the chosen construction. Representative references: Rahimi and Recht (2007), Cho and Saul (2009), Choromanski et al. (2020), and Bach (2017). Scope of later results: All subsequent expectation and concentration results assume the nonnegative feature case of (A2 ′ ), or the bounded-signed case with D ( q ) > 0ensuring a well-posed quotient. Results for general signed features would require separate variance control and are left for future analysis. 2.4 Plate factorization and interpretation Whenever a (possibly nonnegative or signed) feature map ϕ exists, any A ∈ C admits a constructive Plate factorization: ΨV=X t ϕ(kt)v⊤ t,Ψ1=X t ϕ(kt). Then the induced row-stochastic weights are Wij =ϕ(qi)⊤ϕ(kj) Pj′ϕ(qi)⊤ϕ(kj′)+δ,AQ,K(V)=W(Q, K)V. Equivalently, in per-query quotient form, y(q) = ⟨ϕ(q),ΨV⟩ ⟨ϕ(q),Ψ1⟩+δ, where we ensure infq⟨ϕ ( q ) , Ψ 1⟩ + δ≥δ > 0by construction (stabilizer). Plates are thus a constructive factorization whose accuracy depends on the chosen ϕ and width m . They summarize all key–value pairs into two fixed-size tensors (Ψ V, Ψ 1 )independent of sequence length. 4 2.5 Representation closure (expectation form) Let A∈C admit a Plate factorization with feature map ϕ and accumulators (Ψ V, Ψ 1 ). The induced Plate operator Pϕ(q) = ⟨ϕ(q),ΨV⟩ ⟨ϕ(q),Ψ1⟩+δ remains in Cand preserves three measurable structures in expectation under A2–A3: (i) Stationarity: if A is approximately translation-equivariant, E [ Pϕ ]is too, with blur bounds adding under composition; (ii) Quotient separability: numerator and denominator perturbations yield orthogonal first-order effects under isotropic ϕ; (iii) Phase neutrality: if the feature law is rotation-invariant, E [ Pϕ ]commutes in expectation with phase rotations. 3 The s-axis: Monotone Blur under Fair Thinning With shared positional structure, the attention weight kernel W ( Q, K )is well-approximated by a circulant operator along the sequence dimension (convolutional to first order), so each row acts as a point-spread function (PSF). Fair thinning induces a predictable broadening of these PSFs in expectation. 3.1 Fair thinning Each feature dimension f of the feature map ϕ ( k )is dropped independently with probability p (i.i.d. Bernoulli thinning) and reweighted by 1 / (1 −p )when retained. Let mf∈ { 0 , 1 / (1 −p ) } be the random mask variable with E[mf]=1. The thinned kernel becomes ˜κ(q, k) = ϕ(q)⊤(m⊙ϕ(k))=κ(q, k) + ξ(q, k),E[ξ] = 0,Var[ξ]∝p 1−p. Blur functionals. We call W ablur functional if it is (i) invariant under circular shifts, (ii) monotone under radial spectral attenuation of the kernel spectrum, and (iii) continuous. This definition isolates the class of observables for which monotone blur can be proved; it does not assert that all blur measures behave monotonically. Examples include the baseline-subtracted full width at half maximum (FWHM) and the second moment (variance) of the PSF. 3.2 Theorem: Monotone blur under fair thinning, in expectation Assume approximate translation equivariance. Let ˜κp be the thinned kernel. For any blur functional Wcomputed from the expected PSF of the row-stochastic weight operator, E W(PSF(˜κp))−W(PSF(κ))≤Cqp m. 3.3 Proposition: First-order diffusion-like behavior for small p For small thinning rates p, the expected spectral multiplier satisfies E[˜ λp(ω)] = λ0(ω)1−αpρ(ω)+o(p), ρ(ω)≥0, so that the corresponding blur functional expands as W(p) = W(0) + α′p+o(p). 5 4 The u-axis: Quotient Asymmetry and Dual Robustness The second holographic axis, u , captures the internal quotient geometry of attention—how robustness differs when we perturb its two components: the numerator (content) and the denominator (normalization). This asymmetry appears naturally once attention is expressed in Plate form: y(q) = ⟨ϕ(q),ΨV⟩ ⟨ϕ(q),Ψ1⟩+δ. 4.1 Differential quotient form Let N ( q ) = ⟨ϕ ( q ) , Ψ V⟩ and D ( q ) = ⟨ϕ ( q ) , Ψ 1⟩ + δ . Then y ( q ) = N/D , and the differential of the quotient is dy(q) = dN(q) D(q)−N(q)dD(q) D(q)2. 4.2 Orthogonality in expectation Assumptions. We assume A2–A3 from §2.3, including denominator stability. Proposition (Orthogonality in expectation). Assume ϕ ( q )has zero-mean fluctuations and covariance Eq [ ϕ ( q ) ϕ ( q ) ⊤ ]=Σ ϕ , and is independent of perturbations (∆ V, ∆ 1 ). Consider infinitesimal perturbations Ψ V→ Ψ V + ε ∆ V and Ψ 1→ Ψ 1 + ε ∆ 1 . If ϕ ( q )is isotropic and independent of (∆V,∆1), then the first-order variations are orthogonal in expectation: Eq[dyVdy1]=0. Sketch. Expanding dy ( q )gives two linear forms in ϕ ( q )multiplied by D ( q ) −1 and N ( q ) /D ( q ) 2 . Under A3, D ( q )is bounded away from zero and its relative variation is O ( ε ), so N ( q ) /D ( q ) 3 = ¯α + O ( ε )with ¯α = Eq [ N ( q ) /D ( q ) 3 ]nearly constant. Since Eq [ ⟨ϕ ( q ) , ∆ V⟩⟨ϕ ( q ) , ∆ 1⟩ ] = 0 under isotropy, the cross term is O ( ε ). Hence orthogonality holds in expectation up to first order in the denominator fluctuation amplitude. Heuristic interpretation: contrast vs. normalization mass. In practice, perturbations to the content plate Ψ V chiefly affect output contrast (local variance), whereas perturbations to the normalization plate Ψ 1 alter the overall output mass (entropy). This split provides an operational diagnostic rather than a formal theorem of convex analysis. 5 The c-axis: Phase Response and Rotational Neutrality 5.1 Setup and definitions We consider rotations R ( θ ) ∈U (1) acting on paired positional coordinates (e.g., RoPE). Such rotations act on queries and keys as Q(θ) = R(θ)Q, K(θ) = R(θ)K, inducing a phase shift in the feature representations. We are interested in whether attention responses remain invariant, in expectation, under such rotations of the feature space or positional phase. 6 5.2 Theorem: Phase neutrality in expectation If the feature law is rotation-invariant and thinning is independent of phase, then E[Ap(Q(θ), K(θ))] = E[Ap(Q, K)] for all θ, so the phase-response curve is flat up to sampling error. 6 The Holographic Set of Operators The three axes—scale ( s ), quotient ( u ), and phase ( c )—jointly describe measurable behaviors of normalized positive kernel operators. 6.1 Definition of the holographic set Definition (Holographic set H). H ⊂ C consists of operators that satisfy the following measurable properties in expectation: (i) Monotone blur: exhibit nondecreasing blur functionals under fair thinning (see §3); (ii) First-order quotient separability: numerator and denominator perturbations are orthogonal in expectation (see §4); (iii) Phase neutrality: commute in expectation with U (1) rotations under isotropic feature laws (see §5). We emphasize that H is defined in expectation: membership signifies adherence to these measurable laws on average over feature and thinning randomness, not in every realization. Almost-sure and finite-sample analogues are left for future work. 6.2 Conditional stability under composition [Conditional stability under composition] If A1, A2∈ H share the same positional frame and their kernels are approximately circulant with independent thinning noise, then A2◦A1∈ H in expectation. The Holographic Set as a Reference Manifold The holographic set H is not proposed as a predictive class for individual realizations, but as a calibration baseline describing how normalized positive kernel operators behave in expectation under isotropic and fair-thinning assumptions. Real attention layers in trained Transformers correspond to single realizations whose behavior can deviate from this baseline in measurable ways (e.g., anisotropic features, correlated dropout, non-circulant weights). In this sense, H defines a reference manifold—a low-dimensional coordinate surface describing expectation-level regularities of attention—that serves as a baseline for interpreting empirical diagnostics rather than a performance guarantee. 7 Discrete–Continuous Correspondence This section establishes that discrete attention approximates a continuous normalized kernel integral. 7.1 Continuous normalized kernel operator Let q(t), k(τ), v(τ)be Lipschitz maps. Define the continuous operator A(t) = RT 0κ(q(t), k(τ)) v(τ)dτ RT 0κ(q(t), k(τ)) dτ . 7 7.2 Discrete Riemann approximation The discrete attention operator with step size ∆τis Y∆τ(ti) = Pjκ(Qi, Kj)Vj∆τ Pjκ(Qi, Kj) ∆τ. This recovers the softmax form under uniform quadrature: identical ∆ τ factors cancel in the normalization. 7.3 Theorem: Discrete–continuous correspondence [Discrete–continuous correspondence] Assume q, k, v are L -Lipschitz, κ is bounded and positive definite, and ϕis an m-feature approximation of κwith error εϕ(m). Then sup t∈[0,T ] Y∆τ(t)−A(t) 2=O(∆τLκ+εFFN +εϕ(m))(1 + ∥A∥∞), where the implicit constant depends on δ−1via infqD(q)≥δ. Remark. Because Y∆τ and A are normalized quotients, numerator and denominator errors couple multiplicatively; the bound above absorbs these interactions into the single factor (1+∥A∥∞). 8 Discussion and Outlook 8.1 The manifold of holographic operators All admissible operators in C admit smooth expectation-level coordinates ( s, u, c )corresponding to blur scale, normalization quotient, and phase. These coordinates behave locally like charts on a manifold of normalized kernel operators, though we leave the formal construction of that manifold to future work. 8.2 Directional sensitivities as Gâteaux derivatives in probability We define the directional sensitivities of an operator Y along the three holographic axes as first-order Gâteaux derivatives in probability, with limits taken in L2under §2.3: ∂sY:= lim ε→0 Yp+ε−Yp ε(sensitivity to feature thinning), ∂uY:= d dαY(N+α∆V, D +α∆1)α=0 (sensitivity to numerator–denominator balance), ∂cY:= d dθY(Q(θ), K(θ))θ=0 (sensitivity to rotational phase).(1) 8.3 Functional interpretation of robustness The holographic framework explains common empirical trends: •Monotone blur corresponds to graceful degradation under thinning (§3); •Quotient separability accounts for resilience to normalization drift (§4); •Phase neutrality captures invariance to global rotational shifts (§5). 8 On nonnegativity. The restriction ϕ≥ 0isolates the regime where attention acts as a normalized positive operator, making monotone-blur and separability proofs tractable. Empirically, signed-feature models (e.g., Performer with random Fourier features) exhibit similar behavior in expectation, suggesting that these results approximate a broader phenomenon. Empirical diagnostics in Paper II show the same monotone trends for signed features with D > 0. A rigorous extension to general signed features remains an open direction. 9 Conclusion We have presented a unified framework for analyzing attention as a normalized positive kernel operator defined over three measurable axes—blur ( s ), quotient ( u ), and phase ( c ). Each axis captures a distinct and empirically observable property of attention: monotone blur growth under fair thinning, orthogonal first-order effects of content and normalization perturbations, and phase neutrality under rotation-invariant feature laws. Together, these behaviors define the holographic set H : the family of operators that preserve measurable regularities in expectation. The discrete–continuous correspondence establishes that softmax attention is a Riemann approximation to a normalized kernel integral under standard regularity assumptions. Within this setting, the directional sensitivities ( ∂s, ∂u, ∂c )provide a principled calculus for local analysis of robustness and invariance. This formulation reframes attention not as a stochastic diffusion or metric contraction, but as a measurable mapping governed by convex observables. The holographic framework thus provides more than a bridge between theory and practice: it establishes a principled reference manifold. This baseline of lawful, expected behavior offers researchers a coordinate system for measuring, interpreting, and ultimately controlling the complex dynamics of real-world attention mechanisms. References [1] A. Rahimi and B. Recht. Random features for large-scale kernel machines. In Advances in Neural Information Processing Systems (NeurIPS), 2007. [2] Y. Cho and L. K. Saul. Kernel methods for deep learning. In Advances in Neural Information Processing Systems (NeurIPS), 2009. [3] K. Choromanski, V. Likhosherstov, D. Dohan, X. Song, A. Gane, T. Sarlos, P. Hawkins, J. Davis, A. Mohamed, D. Kaiser, and J. Kislyuk. Rethinking attention with performers. In International Conference on Learning Representations (ICLR), 2020. [4] F. Bach. On the equivalence between kernel quadrature rules and random feature expansions. Journal of Machine Learning Research, 18(21):1–38, 2017. 9