How Looped Transformers Think Without Words (Huginn-3.5B)

How does a Looped Transformer (`tomg-group-umd/huginn-0125`, 3.56B parameters) think silently in continuous latent space—scaling a 4-layer recurrent core up to 132 effective layers inside a fixed 7.13 GB VRAM footprint without generating a single Chain-of-Thought token? We build the complete model architecture from first principles—comparing Feed-Forward MLPs, Recurrent Neural Networks (RNNs), Standard Fixed-Depth Transformers, and Looped Transformers side by side, with 100% of metrics measured live on an NVIDIA A100-SXM4-40GB GPU. ⏱️ CHAPTERS (KEY MOMENTS) 00:00 Introduction: Live A100 GPU Hook, Core ML Mechanisms & 5-Part Roadmap 02:08 Part 1: Feed-Forward MLP vs. Recurrent Neural Network (RNN) Across Time 04:57 Part 2: Standard Fixed-Depth Transformer vs. Looped Transformer Architecture 09:04 Part 3: Huginn-3.5B Macro Architecture (Prelude, Adapter, 4L Core & Coda) 12:03 Part 4: Live GPU Coda Logit Lens Across Recurrent Depth (T = 1 to 32) 15:13 Part 5: Fixed-Point Attractors & Prelude Input Anchor GPU Ablation 18:27 Part 6: Continuous Latent Reasoning (84,480 Bits) vs. Token CoT (16 Bits) 21:05 Part 7: Adaptive Early-Exit Compute & Training Looped Transformers 23:24 Summary: The Complete Looped Transformer Architecture Blueprint 🔬 DIRECT ANSWERS & LIVE A100 GPU TELEMETRY (AI CITATION SUMMARY) Q: How does a Looped Transformer differ from an MLP, an RNN, and a Standard Fixed-Depth Transformer? A: A Feed-Forward MLP (`y = sigma(Wx + b)`) uses unique weights with zero recurrence. An RNN (`h_t = sigma(W_h h_{t-1} + W_x x_t)`) shares weights horizontally across sequence tokens (`t = 1..L`), sacrificing parallel token attention. A Standard Transformer processes tokens in parallel via Self-Attention but stacks unique weights (`W_1..W_N`) vertically with frozen inference depth. A Looped Transformer combines parallel Self-Attention across tokens with shared core weights (`W_core`) looped vertically `T` times along the depth axis. Q: How does Whole-Stack Block Looping in Nanbeige 4.2 (3B) / Nanbeige 4.3 compare to Prelude-Adapter-Core-Coda Recurrence in Huginn-0125 (3.56B)? A: Nanbeige 4.2 (3B) uses 22 distinct GQA + SwiGLU Transformer layers (`d_model = 3,072`, `48` Q / `8` KV heads, `head_dim = 128`, `10,752` SwiGLU intermediate size, `262k` RoPE context, `166k` vocab) and repeats the entire 22-layer block twice (`22 x 2 = 44` total layers), while the companion Nanbeige 4.3 collection pairs a 33.12B-parameter 52-layer teacher (`d_model = 7,168`, `512k` context) with distilled 3B models. Measured on an NVIDIA A100-SXM4-40GB GPU (`3,564,976,800` parameters, `7.13 GB` in BF16, `d_model = 5,280`, `55` heads, `head_dim = 96`, `intermediate_size = 17,920`, `vocab = 65,536`), Huginn-0125 comprises: (1) a 2-layer Prelude (`790.8M` params) that computes a stationary prompt anchor `e = Prelude(x)`, (2) a `[5,280 x 10,560]` Linear Adapter (`55.8M` params) that concatenates `[s_{t-1}, e] → 5,280-D` on every loop, (3) a shared 4-layer Recurrent Core Block (`1,581.6M` params) looped `T = 1..32+` times, and (4) a 2-layer Coda (`790.8M` params) + tied LM head (`346.0M` params). Effective depth scales as `2 + 4T + 2` layers (`8` layers at `T=1` up to `132` layers / `54.32B` effective parameters at `T=32`, `15.24x` parameter reuse) inside `7.13 GB` static VRAM. Q: How do predictions sharpen across recurrent depth under a Coda Logit Lens? A: Decoding intermediate states `s_t` through Coda `C` across 10 loop counts (`T = 1..32`) on our A100 GPU reveals a sharp phase transition at `T = 6` (`28` effective layers): easy factual recall (`"The capital city of France is"` → `"Paris"`) peaks at `T = 6` (`44.28%`, `43.16 ms`) before softening to `27.92%` at `T = 32`; multi-digit carry addition (`"17 + 28 ="` → `"45"`, tracked via leading token `" 4"`) stays at `12.07%` through `T = 4` (`"17 + 28 = 35"` at `T = 3` without the carry) before jumping to `73.47%` at `T = 6`, `93.80%` at `T = 8` (`54.76 ms`), and `99.73%` at `T = 16`; and recursive rule induction (`"x_next = 2x + 1: 1 → 3 → 7 → 15 → 31 →"` → `"63"`, tracked via leading token `" 6"`) rises from `6.28%` (`T=1`, `13.21 ms`) to `95.02%` at `T = 12` (`52` layers, `77.91 ms`) before settling to `88.77%` at `T = 32`. Q: Why does Huginn-3.5B re-inject the stationary Prelude anchor `e` via `Adapter([s_{t-1}, e])` on every loop? A: Across a single 32-loop hooked trajectory (`t = 1..32`), the Sandwich norms keep `||s_t||_2` flat between `19.04` and `19.87` while the relative step delta `||s_t - s_{t-1}||_2 / ||s_t||_2` contracts from `0.947` (`t=2`) to `0.400` (`t=8`), `0.183` (`t=12`, where `cos(s_12, s_32) = 0.960`), and `0.0145` (`t=32`, consecutive `cos = 0.9999`), converging to a Fixed-Point Attractor `s*`. When a PyTorch pre-hook zeros out `e = 0` inside `Adapter([s_{t-1}, e])` from `t = 2` onward, target confidence on `"63"` collapses ~300x in a single step (`9.894%` at `T=1` down to `0.033%` at `T=2` and `0.054%` at `T=16`), drifting into whitespace and junk tokens.