Artificial Intelligence
Retrieval-Augmented Generation, One Year In: What Actually Held Up
RAG was supposed to solve hallucination. A year of production traffic told a more specific story.
Retrieval-augmented generation shipped everywhere in the last two years, and the pattern is now mature enough to have real failure data behind it, not just launch-day demos.
What actually broke in production
The most common failure was not retrieval accuracy — it was staleness. Indexes that update on a nightly batch job quietly serve outdated answers for anything that changed that day, and users notice a wrong answer far more than they notice a slow one.
- Chunking strategy mattered more than embedding model choice for most document types.
- Re-ranking after retrieval improved perceived quality more than expanding top-k.
- Incremental index updates, not full nightly rebuilds, fixed the staleness complaints.
The takeaway
Treat your retrieval index like any other production data store: monitor freshness, alert on drift, and budget engineering time for it the same way you would for a cache invalidation strategy.