Why Cloudflare Kept Failing and Recovering

On 18 November 2025, Cloudflare's network started failing, then recovered on its own, then failed again. The team first suspected a hyper-scale DDoS attack. The cause was a change Cloudflare had made itself that morning, and a feature file rebuilt every five minutes. This video follows Cloudflare's own postmortem: a database permissions change, a query on system.columns with no filter on the database name, a feature file with more than twice as many rows, a 200-feature limit, and an unwrap() in the Rust code of the new FL2 proxy. The old FL proxy showed no errors but gave every request a bot score of zero. The video also reproduces the query change on a one-node ClickHouse (commands below). Chapters 0:00 Fail, recover, fail again 0:36 Everything behind one proxy 1:18 A file rebuilt every five minutes 2:29 A change meant to make things safer 4:03 Two hundred slots 5:24 Two proxies, two kinds of wrong 7:22 Trusting your own config Details the video keeps short Start time: the postmortem's overview says the failures began at 11:20 UTC. Its timeline table says 11:28, when the deployment reached customer environments and the first errors on customer HTTP traffic were observed. The video uses 11:28, the first errors on customer traffic. The ~60 vs more than 200 gap: normal use was about 60 features, the limit is 200, and the bad file had more than 200. The postmortem describes the rows as more than doubling, but does not say what took the count past 200. Cloudflare's follow-up list on the day also included more global kill switches for features, and stopping core dumps and other error reports from overwhelming system resources. Common claims, checked "A missing WHERE clause." The query has a WHERE clause, on the table name. What it lacks is a filter on the database name. Cloudflare: "Note how the query does not filter for the database name." "It crashed every five minutes." The file was rebuilt every five minutes, and each rebuild could come out good or bad. In the end every ClickHouse node made the bad file, and the network stayed in the failing state. "The old proxy was fine." FL returned no errors, but every request got a bot score of zero, so customers with rules to block bots saw large numbers of false positives. "It was an attack." Cloudflare: "The issue was not caused, directly or indirectly, by a cyber attack or malicious activity of any kind." Reproduce the query change (one ClickHouse node in Docker, clickhouse-server 26.8.36.3; table and column names are invented, not Cloudflare's) Setup: r0.http_requests_features is a MergeTree table with 7 columns. default.http_requests_features is a Distributed table over it. User feature_generator has SELECT on the default table only, and prefer_localhost_replica = 0, so its Distributed reads run as the cluster's connection user, standing in for Cloudflare's shared system account. 1. As feature_generator, run Cloudflare's query: SELECT name, type FROM system.columns WHERE table = 'http_requests_features' order by name; Result: 7 rows. 2. As admin: GRANT SELECT ON r0.* TO feature_generator; The same query now returns 14 rows, every column name twice (7 from default, 7 from r0). 3. Add AND database = 'default' to the query: 7 rows again. Not reproduced: Cloudflare's cluster, the gradual rollout across nodes, the feature counts, the 200 limit and the Rust code. Sources (search by title; links will be added once the channel can post them) Primary (Cloudflare): Cloudflare outage on November 18, 2025 (Matthew Prince, 18 Nov 2025) Cloudflare outage on December 5, 2025 (Dane Knecht, 5 Dec 2025) Code Orange: Fail Small, our resilience plan following recent incidents (Dane Knecht, 19 Dec 2025) Code Orange: Fail Small is complete. The result is a stronger Cloudflare network (1 May 2026) Cloudflare just got faster and more secure, powered by Rust (26 Sep 2025) The FL2 Rust code shown in the video is re-typeset from the image in the 18 November postmortem (lines 71 to 82). Reproduction: Undertrace, minimal ClickHouse reproduction of the system.columns change (run locally 29 Oct 2026, clickhouse-server 26.8.36.3, one node). The commands are above as text, not a link. Commentary: Hackaday, How One Uncaught Rust Exception Took Out Cloudflare (Maya Posch, 20 Nov 2025), the headline shown in the video Ruohang Feng (Vonng), Cloudflare's Nov 18 Outage, Translated and Dissected (19 Nov 2025) Music and sound effects: YouTube Audio Library. Tracks: Oumuamua (Alex Hamlin), Phillip Glass Elevator (Ezra Lipp), Rapid Unscheduled Disassembly, Event Horizon and Kayak (The Grey Room / Density & Time), Polymetric Juggling (DivKid), Decision (The Tower of Light), Curse Of Old and Under The Rug (Density & Time), When the Ash Settles (The Westerlies). Narrated with an AI clone of my own voice. Spotted a mistake? Tell me in the comments, with a source if you have one. Corrections get pinned. #cloudflare #postmortem #softwareengineering