Skip to content

DDIR: integer bounds per chunk, so readers skip chunks that can't hold their keys - #1071

Open
frankmcsherry wants to merge 1 commit into
TimelyDataflow:master-nextfrom
frankmcsherry:ddir-chunk-bounds
Open

frankmcsherry wants to merge 1 commit into
TimelyDataflow:master-nextfrom
frankmcsherry:ddir-chunk-bounds

Conversation

@frankmcsherry

Copy link
Copy Markdown
Member

Each CorgiChunk keeps the first and last identifiers of its rows, so a reader can rule a chunk out without reading its key column.
The reduce's key search, the join's seeks, block ends and probes, and the half join's chunk filter use them; results are unchanged.
On today's workloads this measures as neutral; it is groundwork for chunks whose key columns are compressed or paged out, and it bounds the work a reader spends on chunks that can't hold its keys.

What changes

  • Inner gains bounds: Option<(i64, i64)>: the ends of the identifier lane (the integer key, or the hash lane of a compound key), set when the chunk is built. None for an empty chunk, or one with no i64 identifier lane.
  • Three methods read them:
    • find(needles, ..): matching_ranges over only the needles inside the bounds, with indices into the full needle list;
    • rows_before(id): where a seek to id lands, without a search when id is past either end;
    • may_hold(first, last): false only when the bounds rule the range out.
  • Callers:
    • the reduce's search uses find on every input and output chunk;
    • the join's Probe::new uses find; leaf_probe builds no probe for a chunk whose bounds miss the block's keys; advance_leaf's seek and block_ends use rows_before;
    • the half join's filter uses may_hold instead of reading the lane's ends.
  • Two comments now say what the code does: the reduce search doc mentions the bounds, and the seek comment in advance_leaf says the bounds answer before the binary search.

Why

Every keyed reader touched every chunk: a binary search per needle per chunk, or a merge with the whole lane, even for chunks whose identifiers cannot overlap the needles.
With the bounds that is two binary searches over the needles, and nothing touches the chunk's columns.
A page index and Bloom filters would sit beside these bounds, and they stay readable once a chunk's key column is compressed or not resident.

Measurements

One run each, mini-02, one worker, diagnostics on, against stored runs of a6b7922a.

base this
LDBC SF1 one-off, 808 ticks: instructions 2,126.8 G 2,122.4 G
LDBC: server CPU 120.17 s 120.65 s
SCC (LiveJournal 250k) first tick: instructions / CPU 58.87 G / 3.30 s 58.92 G / 3.25 s
SCC, 50 churn ticks: instructions / CPU 34.53 G / 2.08 s 34.49 G / 2.05 s

All within noise.
Answer and physical digests are identical on all 808 LDBC ticks, and SCC's initial and final peeks are identical.

Tests

  • New: bounds_agree_with_the_identifier_lane checks find, rows_before and may_hold against searching the whole lane, for integer and hashed keys, with needles below, inside, between and above a chunk's identifiers, and on an empty chunk.
  • cargo test -p interactive: 295 passed, 0 failed, 9 ignored; no warnings.

It merges cleanly with #1069.

Implemented and measured by Claude Code.

🤖 Generated with Claude Code

https://claude.ai/code/session_01H6wB4H99TLw4tqu4x48pXL

…d their keys

Each `CorgiChunk` now keeps the first and last identifiers of its rows (the integer key, or the hash lane of a compound key) beside its columns.
Readers use them through three methods: `find` searches only the needles inside a chunk's bounds, `rows_before` answers a seek without a binary search when the target is past either end, and `may_hold` rules out a chunk whose bounds miss a range.
The reduce's key search, the join's seeks, block ends and probes, and the half join's chunk filter go through them; their results are unchanged.
The bounds are what a page index and Bloom filters would sit beside, and they stay readable once a chunk's key column is not.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H6wB4H99TLw4tqu4x48pXL

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant