Repository navigation
DDIR: integer bounds per chunk, so readers skip chunks that can't hold their keys - #1071
Open
frankmcsherry wants to merge 1 commit into
Open
frankmcsherry wants to merge 1 commit into
frankmcsherry wants to merge 1 commit into
Conversation
…d their keys Each `CorgiChunk` now keeps the first and last identifiers of its rows (the integer key, or the hash lane of a compound key) beside its columns. Readers use them through three methods: `find` searches only the needles inside a chunk's bounds, `rows_before` answers a seek without a binary search when the target is past either end, and `may_hold` rules out a chunk whose bounds miss a range. The reduce's key search, the join's seeks, block ends and probes, and the half join's chunk filter go through them; their results are unchanged. The bounds are what a page index and Bloom filters would sit beside, and they stay readable once a chunk's key column is not. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H6wB4H99TLw4tqu4x48pXL
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Each
CorgiChunkkeeps the first and last identifiers of its rows, so a reader can rule a chunk out without reading its key column.The reduce's key search, the join's seeks, block ends and probes, and the half join's chunk filter use them; results are unchanged.
On today's workloads this measures as neutral; it is groundwork for chunks whose key columns are compressed or paged out, and it bounds the work a reader spends on chunks that can't hold its keys.
What changes
Innergainsbounds: Option<(i64, i64)>: the ends of the identifier lane (the integer key, or the hash lane of a compound key), set when the chunk is built.Nonefor an empty chunk, or one with noi64identifier lane.find(needles, ..):matching_rangesover only the needles inside the bounds, with indices into the full needle list;rows_before(id): where a seek toidlands, without a search whenidis past either end;may_hold(first, last): false only when the bounds rule the range out.searchusesfindon every input and output chunk;Probe::newusesfind;leaf_probebuilds no probe for a chunk whose bounds miss the block's keys;advance_leaf's seek andblock_endsuserows_before;may_holdinstead of reading the lane's ends.searchdoc mentions the bounds, and the seek comment inadvance_leafsays the bounds answer before the binary search.Why
Every keyed reader touched every chunk: a binary search per needle per chunk, or a merge with the whole lane, even for chunks whose identifiers cannot overlap the needles.
With the bounds that is two binary searches over the needles, and nothing touches the chunk's columns.
A page index and Bloom filters would sit beside these bounds, and they stay readable once a chunk's key column is compressed or not resident.
Measurements
One run each, mini-02, one worker, diagnostics on, against stored runs of
a6b7922a.All within noise.
Answer and physical digests are identical on all 808 LDBC ticks, and SCC's initial and final peeks are identical.
Tests
bounds_agree_with_the_identifier_lanechecksfind,rows_beforeandmay_holdagainst searching the whole lane, for integer and hashed keys, with needles below, inside, between and above a chunk's identifiers, and on an empty chunk.cargo test -p interactive: 295 passed, 0 failed, 9 ignored; no warnings.It merges cleanly with #1069.
Implemented and measured by Claude Code.
🤖 Generated with Claude Code
https://claude.ai/code/session_01H6wB4H99TLw4tqu4x48pXL