You Rescued Terabytes of Data. Now What Is In It?

You rescued terabytes of scraped government pages and dataset dumps before they disappeared, and now nobody knows what is in them. This walkthrough runs a local LLM over samples of every collection to describe it, classify it, and flag likely duplicates into a searchable SQLite catalog. The archive stays read-only the whole time: the model catalogs, and a human decides. What's covered: • When you need a model at all, and when inventory, hashes and DuckDB are enough • Seven passes, and why only three of them use a language model • Mounting the archive read-only and keeping a sidecar catalog • Exact duplicates by shared-hash fraction (both over 0.95 is the same collection twice) • Stratified sampling: 20 to 60 samples per typical collection • A JSON schema with closed enums and UNKNOWN escape hatches, at temperature zero with thinking off • Why the model cannot count, so SQL does the counting • Map-reduce for files bigger than the context window, and what survives digestion • Near-duplicate collections with embeddings, and calibrating the threshold on your own pairs • The cost math for an illustrative 12 TB rescue (about 90,000 calls): hosted versus a used 12 GB card • Why a description is not provenance ⏱️ Chapters 00:00 Triage a Rescued Archive 01:11 Do You Need an LLM? 02:38 Inventory Without a Model 04:37 Classify Every Sample 05:59 Let Code Do the Counting 07:29 Near-Duplicates and Review 08:42 The Cost Math 📖 The full guide (inventory script, sampling query, classification schema, map-reduce, embedding code, verification and rollback): → LLM Triage for Bulk Archives: Classifying Scraped Datasets Locally: https://www.bigiron.cc/guides/llm-tri... 🛠️ Gear (affiliate — same price to you, supports the channel): • 12 GB graphics card for an 8B classifier → https://www.amazon.com/s?k=rtx+3060+1... • NAS hard drives for the archive itself → https://www.amazon.com/s?k=nas+hard+d... • Multi-bay NAS enclosure → https://www.amazon.com/s?k=4+bay+nas+... As an Amazon Associate we earn from qualifying purchases. Affiliate links — same price to you, supports the channel. 👉 Subscribe for self-hosted AI + homelab, with new videos every day. #DataHoarding #Archiving #LocalAI #LLM #Ollama #DuckDB #SQLite #WARC #SelfHosted #Homelab