Papers
arxiv:2611.26450

A CPU Inside an LLM: Zero-Gradient Compilation of Exact Circuits into Pretrained Weights

Published on Nov 19
· Submitted by
Tanmay Sah
on Nov 20
Authors:

Abstract

Pretrained language models (LMs) remain unreliable on exact multi-digit arithmetic without gradient fine-tuning or external tool interpreters. We introduce closed-form, zero-gradient circuit-grafting primitives that compile deterministic computation directly into pretrained LM weights using zero training examples and zero runtime hooks, while bounding per-layer normalization drift and preserving natural-language (NL) competence. Our construction combines three analytical mechanisms: (1) two-layer Mellin-GLU quadrature over bias-free SwiGLU/GeGLU layers, turning unknown per-token RMSNorm scaling into translation along a log-spaced gate grid (>250times dynamic range) with IEEE 754 error bounds; (2) cross-token scale cancellation via signed Chebyshev Indefinite-RoPE self-sink attention and memoryless SiLU-antisymmetric GatedDeltaNet shifts; and (3) axis-aligned width extension (d to d+K) with d/(d+K) norm compensation, Base-1000 canonical BPE emission, and differential (+254,-256) tied unembeddings. An automated 5-stage compiler synthesizes 4-8 digit signed {+,-,times} circuits in constant TC^0 depth alongside 3-literal Boolean (100%, N=1{,}728; plus 42-layer Gemma 4 E4B audits) and 5-bit Majority/Parity (100%) circuits. On 6-digit arithmetic (N=1{,}200, BF16), IBM Granite 4.2 3B (1.07M-1.33M active non-zeros, le 0.036%) and Alibaba Qwen 3.5 4B (2.25M-2.39M non-zeros, le 0.057%) reach 99.17% and 100.00% ALU-prefix accuracy (100% with an FP32 sidecar) and 98%-100% mid-sentence resumption, outperforming 12.8K-sample LoRA (42.2% / 61.7%, 0% inline continuation). A factorized block-sparse sidecar holds A100 memory at 7.89 / 10.65 GiB (4.60 / 8.14 GiB under INT8/INT4 + FP32 sidecar) while passing all 14/14 pre-registered >2K-token Tango splits (N=4{,}392; 0/11,134 OOD triggers).

Community

Paper author Paper submitter

Why teach an LLM arithmetic when you can compile arithmetic directly into its weights?

TL;DR: From learning weights to programming weights. We introduce closed-form, zero-gradient compilation primitives that embed deterministic computation directly into pretrained LLMs, demonstrating signed multi-digit arithmetic (+, -, ×) and automated compilation of Boolean, Majority, and Parity circuits across Qwen 3.5 4B, IBM Granite 4.2 3B, and architectural audits on Gemma 4 E4B.

  • Zero gradient updates. Zero training examples. Zero external tools. Computation is encoded directly into model weights and executed through unmodified model.generate().
  • Exact logic across >250× RMSNorm swings. Two-layer Mellin-GLU quadrature transforms unknown per-token normalization scaling into translation along a log-spaced gate grid, with a calibration-free variant.
  • Attention-free circuit routing on Qwen 3.5. Zero of eight full-attention layers modified. Cross-token operands are routed entirely through GatedDeltaNet blocks.
  • Less than 0.06% additional active weights. Compatible with INT4/INT8 quantized backbones and a compact FP32 circuit sidecar.

Key results:

  • 100% evaluated ALU-prefix accuracy with FP32 circuit execution.
  • 100% evaluated Boolean, Majority, and Parity accuracy.
  • 98–100% fluent inline continuation, versus 0% for 12.8K-example LoRA fine-tuning.
  • Zero false circuit triggers across 11,134 out-of-domain inputs.
  • Natural-language noninferiority across all 14 preregistered evaluation splits.
  • INT4 deployment with just 4.6 GiB / 8.1 GiB VRAM for Granite / Qwen.

What if adding new capabilities to neural networks didn't always require training them?

This is an automated message from the Librarian Bot. I found the following papers similar to this paper.

The following papers were recommended by the Semantic Scholar API

Please give a thumbs up to this comment if you found it helpful!

If you want recommendations for any Paper on Hugging Face checkout this Space

You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: @librarian-bot recommend

Sign up or log in to comment

Get this paper in your agent:

hf papers read 2611.26450
Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2611.26450 in a model README.md to link it from this page.

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2611.26450 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2611.26450 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.