Multi-Shard Qdrant Edge - Partitioning One Dataset Across Local Shards + Federated Query Layer
HiDevs
0:00 / 0:00
Multi-Shard Qdrant Edge - Partitioning One Dataset Across Local Shards + Federated Query Layer
2 058 просмотров · 8 дней назад
HiDevs
617 подписчиков
2 058 просмотров · 8 дней назад
One dataset, split across multiple local Qdrant Edge shards - by tenant, by time window, or by modality - with a custom federated query layer that fans out and merges results across independent local databases. No server, no cloud, just folders on disk. This video breaks the "one shard" assumption almost every Qdrant Edge tutorial makes, and shows exactly when local sharding is worth the complexity - and when it isn't.
In this video you'll learn how to:
🔹 Run multiple independent Qdrant Edge clients locally, one per shard folder
🔹 Build a FederatedQdrantEdge manager that routes inserts and auto-discovers shards on disk
🔹 Decide when sharding actually helps: hard isolation, RAM-constrained selective loading, avoiding one giant memmap file, and per-modality vector configs
🔹 Decide when it's just added complexity: small datasets, cross-tenant-by-default queries, metadata overhead
🔹 Run single-shard queries for strict isolation vs. fan-out federated queries merged by score across all shards
🔹 Physically delete a tenant's data with rmtree() instead of a DELETE WHERE- verified by rerunning the query after deletion
🔹 Simulate a real multi-tenant edge agent (3 tenants, 3 isolated physical databases)
NOTEBOOK
Get the notebook used in this video: https://colab.research.google.com/dri...
QDRANT DOCS REFERENCED IN THIS VIDEO
Local / Embedded Quickstart (running Qdrant Edge with no server):
https://qdrant.tech/documentation/qui...
Qdrant Edge overview (single Edge Shard concept):
https://qdrant.tech/documentation/edg...
Collections (creating a collection per shard):
https://qdrant.tech/documentation/con...
Filtering (the payload-filter alternative to physical sharding):
https://qdrant.tech/documentation/con...
MORE QDRANT
Qdrant homepage:
https://qdrant.tech/?utm_source=hidev...
Qdrant GitHub:
https://github.com/qdrant/qdrant?utm_...
Qdrant Python Client GitHub (local mode reference):
https://github.com/qdrant/qdrant-clie...
#Qdrant #VectorDatabase #EdgeAI #LocalAI #MultiTenant #RAG #AIAgents #MachineLearning