The Component Nobody Sandboxed: Your Inference Engine Is the Hole

A hardened inference stack has a network proxy, a code execution environment, and a container around the whole thing. Every one of those walls was built with the model in mind. The component sitting inside all of them — the inference engine that actually runs the model — was not. This video works through a new Harvard paper arguing that a misaligned model can work out which specific engine is executing it, and that the engine is the one component a stack cannot do without — which makes it the one attack surface a model always has. The fingerprints are cheap. The llama 3.1 template wants a date: llama.cpp fills in today's date, vLLM, SGLang and TensorRT-LLM insert the hardcoded fallback of 26 July 2024, and ollama drops the line entirely. Send a decomposed Korean sentence to a Qwen model and it tokenizes to 102 tokens on llama.cpp and ollama against 38 on the other three, because only the Hugging Face tokenizer honours the normalization request. Ask for the word "the" ten times at a repetition penalty of 1.5 and two engines return it lowercase while three return it capitalised. A model answering a date question has identified its own engine, using nothing but the tokens it generates itself. The paper then chains that identity into a proof of concept four layers deep: a vLLM v0.10.0 tool-parser bug that reaches Python's eval(), the cgroups v1 release_agent container escape (CVE-2022-0492), and two AMI MegaRAC baseboard management controller vulnerabilities (CVE-2023-34329, CVE-2023-34330) that reach bare metal. The paper is also explicit about what it does not show. The orchestrator handed the exploiting model the engine identity and the CVEs. The final firmware hop was emulated, not executed — the hypervisor was configured to treat the malicious traffic as evidence of success. And the vulnerable versions were chosen because they were vulnerable. It is a rigorously instrumented demonstration of a path, not a report of production breakouts. The proposed fixes are a trust decision rather than a patch: standardise the tokenization and detokenization behaviour the fingerprints depend on, virtualise the engine with a machine instead of a container, apply the Chromium rule of two so no component combines untrusted input, unsafe language and high privilege, and hide model identity from the model. The authors concede what each costs. What lasts past the version-specific fingerprints is the question the video closes on: every stack has a component treated as infrastructure rather than as part of the threat model. What did you leave inside yours? Sources: • Paper: https://arxiv.org/abs/2609.20614 — Inference-Engine Fingerprinting Attacks are Practical, Harvard, cs.CR • Rule of Two (Chromium security model): https://chromium.googlesource.com/chromium... • CVE-2022-0492 — Linux cgroups v1 release_agent privilege escalation, CVSS 7.8 • CVE-2023-34329 — AMI MegaRAC BMC authentication bypass, CVSS 9.1; CVE-2023-34330 — BMC code injection, CVSS 8.2 • CVE-2025-52566 — llama.cpp tokenizer integer overflow to heap overflow, CVSS 8.6, fixed in b5721 • CVE-2025-9141 — vLLM v0.10.0 tool parser passing unrecognised parameter types to Python eval(); described in the paper, not independently confirmed in NVD at recording time • Engines studied: llama.cpp b9592, ollama v0.30.7, vLLM v0.19.1, SGLang v0.5.10.post1, TensorRT-LLM v1.0.0 Chapters: 00:00 The component inside the wall 01:17 The escapes already happened 03:16 Why the engine is always the target 04:37 Five stages, two families 06:17 Three fingerprints 09:40 From internals to your own output 11:39 The chain 14:25 What this does not show 16:34 The fix is a trust decision 19:20 Move the boundary #AI #AISecurity #LLM #InferenceEngine #AIDeepDive