Skip to content

feat(cascade): report duplicate index primary keys - #535

Open
chuiwenwei wants to merge 1 commit into
EverMind-AI:mainfrom
chuiwenwei:feat/cascade-status-duplicates
Open

chuiwenwei wants to merge 1 commit into
EverMind-AI:mainfrom
chuiwenwei:feat/cascade-status-duplicates

Conversation

@chuiwenwei

Copy link
Copy Markdown

Summary

LanceDB has no unique constraint, so a write path that bypasses the merge
helper can leave two rows for one (app_id, project_id, id). The next
cascade write may pick one arbitrarily and search silently loses the
other. Add an opt-in everos cascade status --duplicates scan that counts
extra rows per logical key in every business table.

The scan is O(N) per table, so it stays off the default status path.

Closes #479.

Area

  • Architecture method
  • Benchmark
  • Use case
  • Documentation
  • Developer experience
  • CI, build, or release

Verification

make lint        passed
pytest tests/unit/test_infra/test_index_diagnostics.py \
       tests/unit/test_entrypoints/test_cli/test_cascade_command.py   9 passed

The full unit suite will run in CI; the local run was stopped under host
load after the targeted tests passed.

Checklist

  • I kept the change scoped to the relevant area.
  • I am opening this from a separate branch, not pushing directly to main.
  • I updated docs, examples, or setup notes when behavior changed.
  • I added or updated tests when the change affects behavior.
  • I did not commit secrets, .env files, dependency folders, or generated output.
  • Active relative links in Markdown files resolve.

Notes for Reviewers

  • The flag is opt-in so cascade status stays cheap for normal use.
  • The scan is backend-neutral; Milvus enforces its own primary key and
    should report zero.
  • docs/cli.md and docs/cascade_runbook.md describe the new flag.

By submitting this pull request, I agree that my contribution is licensed under
the Apache License 2.0.

LanceDB has no unique constraint, so a write path that bypasses the
merge helper can leave two rows for one (app_id, project_id, id). The
next write may pick one arbitrarily and search silently loses the other.

Add an opt-in `everos cascade status --duplicates` scan: it counts extra
rows per logical key in every business table, backend-neutrally, and
prints them beside the existing live row counts. The scan is O(N) per
table, so it stays off the default status path.

Refs EverMind-AI#479.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

cascade status: report duplicate primary keys in the derived index

1 participant