RAG实战：何时Plain FAISS不再够用及工程取舍

There's a constant in r/Rag: someone asks "do I need Pinecone/Weaviate/pgvector for my

documents?" and gets told yes, and someone else ships a working product on a flat index.

I maintain a RAG scaffold that uses plain faiss.IndexFlatL2 and I want to lay out exactly

where that's the right call and where it becomes a lie.

What "one command" gives you here:

chimerai add rag

• ingest: RecursiveCharacterTextSplitter, chunk_size=1000 (CHARACTERS, ~250 tokens), overlap 200
• store: faiss.IndexFlatL2, 1536-dim (text-embedding-ada-002), persisted to data/faiss_index
• retrieve: similarity_search(query, k) -> stuff context into the system prompt -> LLM
• response carries a rag_metadata block with the retrieved chunks + scores, so you can

debug "why did it answer that" without a second search call

The important line in the whole pipeline isn't the vector math, it's the prompt:

"If the context doesn't contain relevant information, say so and provide a

general answer."

Without that sentence the model answers from parametric memory and your retrieval eval is

fiction. I've seen more RAG "quality" problems trace to a missing instruction like this

than to the embedding model.

Where the flat index genuinely stops being enough — I'd like to compare notes:

• Scale. Flat L2 is exact brute force, perfect recall, scans everything. Comfortable to
• tens of thousands of chunks. Past that, p95 latency is the wall, not correctness.
• Multi-tenancy. One global index, no namespaces. Per-user/per-tenant retrieval needs
• metadata filtering you write by hand — or a DB that does it natively. In a real SaaS this
• bites BEFORE scale does. This is the one that pushed me to consider pgvector.
• Scaling out. On-disk single-process, no locking. Two instances = two divergent indexes.
• Horizontal scale is simply out of scope.
• Retrieval quality. k is a fixed int. No hybrid BM25+dense, no re-ranker, no MMR
• diversification. When quality plateaus, those are the levers and they're all absent.

So my honest rule of thumb: flat FAISS is the correct start for anything single-tenant and

under ~50k chunks, and the moment you have tenants OR need hybrid search you should move —

not because FAISS is bad, but because you're about to hand-roll the two things a vector DB

gives you (metadata filtering + hybrid).

The FAISS import is guarded (try/except -> FAISS_AVAILABLE) so a broken faiss/numpy on

Python 3.13 degrades only retrieval, not the whole service boot — small but it saved people

a "my app won't start" panic.

Question for the sub: at what chunk count did YOUR flat index actually get slow in real

traffic (not benchmarks)? And for the multi-tenant crowd — did you end up filtering inside

FAISS with metadata, or move to a DB with native per-namespace search? Genuinely trying to

calibrate when "just use FAISS" stops being the lazy correct answer.