Skip to content

perf(knowledge): batch document topic counts - #520

Open
l1amw wants to merge 1 commit into
EverMind-AI:mainfrom
l1amw:perf/knowledge-batch-topic-counts
Open

l1amw wants to merge 1 commit into
EverMind-AI:mainfrom
l1amw:perf/knowledge-batch-topic-counts

Conversation

@l1amw

@l1amw l1amw commented Oct 20, 2026

Copy link
Copy Markdown

Summary

Listing N documents currently performs N separate topic-count queries, in addition to the total-count and page queries. A 100-document page executes 102 SELECTs and fans out database sessions.

Add a grouped topic count for the page's document IDs and call it once from list_documents. Nonempty pages now execute 3 SELECTs; empty pages execute 2. Preserve zero counts, scope/category filters, ordering, pagination metadata, and the existing single-document count method.

Area

  • Architecture method
  • Benchmark
  • Use case
  • Documentation
  • Developer experience
  • CI, build, or release

Verification

Real SQLite regressions reproduced 22 and 102 SELECTs for 20- and 100-document pages before the fix, plus 5 SELECTs for a filtered 3-document page. After the fix, all 26 targeted tests passed on Python 3.12.13 / macOS.

uv run pytest tests/integration/test_knowledge_topic_counts.py tests/unit/test_service/test_knowledge_crud.py -q --tb=short

The full repository checks passed: lint and architecture gates; 2751 unit tests passed (4 skipped); 193 integration tests passed (9 skipped, 7 deselected by the default test configuration); sdist/wheel build and wheel-import smoke test passed.

The local file-size gate uses the target upstream/main because this fork's origin/main has no merge base with the target. The invocation below runs the existing ci target with only that gate's base argument adjusted; no Makefile change is included:

make -f <(sed 's@python scripts/check_file_sizes.py$@python scripts/check_file_sizes.py --base upstream/main@' Makefile) ci

The three independent patches were also applied together in a temporary worktree: 60 relevant compatibility tests passed.

Checklist

  • I kept the change scoped to the relevant area.
  • I am opening this from a separate branch, not pushing directly to main.
  • Docs, examples, and setup notes do not require changes for this fix.
  • I added or updated tests when the change affects behavior.
  • I did not commit secrets, .env files, dependency folders, or generated output.
  • Active relative links in Markdown files resolve.

Notes for Reviewers

The query-budget tests observe actual SQLite statements, including empty pages, zero-topic documents, duplicate/missing IDs, and filtered pagination. No wall-clock latency improvement is claimed. No schema, HTTP contract, or dependency changes are needed.

By submitting this pull request, I agree that my contribution is licensed under
the Apache License 2.0.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant