Proposal: add a small “RAG failure modes & debug checklist” doc (docs only) #223

Hi authors, thanks a lot for releasing FlashRAG — it is extremely helpful as a unified RAG research toolkit.

I am working on failure-mode taxonomies for RAG systems and recently contributed a small robustness entry to Harvard MIMS Lab’s ToolUniverse. Given that FlashRAG already ships 30+ benchmark datasets and 20+ RAG methods, a lot of users will inevitably run into subtle failure patterns that are not simple “bugs”, but configuration or evaluation issues.

I’d like to propose a very small, docs-only addition:

Scope (docs only, no code changes)

• Add one markdown page under docs/ (or any location you prefer), tentatively named:
  • rag_failure_modes_and_debug_checklist.md
• Optionally, link it once from the README or from the “Supporting Methods / FAQs” section.

Suggested contents (kept short and practical)

1. Typical failure modes in RAG experiments:
   • retrieval looks fine but answers drift from ground truth
   • embedding / index mismatch or stale collections
   • overconfident generation when retrieval silently fails
   • evaluation mismatch between retrieval metrics and final answer quality
2. For each failure mode:
   • symptom → likely cause → minimal debug steps
3. A simple “what to include in bug reports” checklist:
   • dataset id, method name, config snippet, log snippet, and seed

Motivation

• Many users are reimplementing or extending RAG methods on top of FlashRAG.
• A small, central checklist could reduce repeated support questions and give new users a more systematic way to debug.
• This is documentation only and should be very safe to review, merge, or revise.

If this sounds in-scope, I’m happy to open a PR and keep the page concise and aligned with your documentation style. Please let me know your preferred location and naming conventions.

Thanks again for the great toolkit.