Repository navigation
DDIR: sort arrange ingest by (key, val, time) at once, and concatenate by ranges - #1064
Merged
frankmcsherry merged 2 commits intoOct 31, 2026
Conversation
… run labels The chunker's flush concatenated its input blocks through a per-row block number and offset, sorted by (key, val), and then walked every (key, val) run: it copied the run, sorted it by time with a Rust comparison sort, and compared neighbouring times to consolidate. Two changes remove those passes. - Concatenation copies each block a range at a time. Each block goes to corgi's `gather_lanes` as a one-row list holding the whole block, so corgi copies ranges at every level it can (leaves, products, lists, references into one arena) and falls back to row positions only for sums, as before. No per-row index is built. - The time is part of the one corgi sort. The time lanes become trailing integer fields: lanes constant within the flush are dropped, the rest are taken as offsets from their least value and packed into one u64 wherever their spreads fit (in practice always), each word offset by 2^63 so that corgi's signed order is the lanes' unsigned order. The sort's run labels then number (key, val, time) classes, and consolidation is one pass that sums diffs within runs of equal labels, with no comparisons, into vectors sized from the number of runs. Output is identical: the sort is stable, so each class's representative row and the order its diffs are summed in are unchanged. On captured flush inputs the stage goes from 21.5 to 14.5 ns per input row on SCC and from 39.8 to 34.3 on LDBC. End to end, one worker, three alternating trials: SCC initial computation -13.1% CPU, reachability initial -14.2%, churn ticks unchanged, LDBC SF1 one-off replay -4.0%; answers identical. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019XsakwFGwtESwZ1gSYwG4o
`CorgiChunk::concat`, which `advance` and `settle` use to join a run of chunks, built a block number and an offset for every row and copied the key and value columns row by row through them. It now uses `concat_blocks`, which the ingest stage already uses: each chunk's columns are copied as one range. End to end, one worker, three alternating trials: instructions fall in every phase (SCC -1.6%, reachability churn -2.0%, LDBC -0.3%); CPU SCC churn -1.1%, reachability churn -1.6%, LDBC -0.3%; answers identical. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019XsakwFGwtESwZ1gSYwG4o
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The arrange ingest stage (the chunker's flush) now sorts by
(key, val, time)in one corgi sort and consolidates by the sort's run labels, and it concatenates its input blocks a range at a time. Two commits, both ininteractive/src/runtime/chunk.rs(+106 −46).gather_lanesas a one-row list holding the whole block, so corgi copies ranges, instead of a block number and offset per row.u64where their spreads fit (in practice always).(key, val, time)classes, so consolidation is one pass summing diffs over equal labels. The per-run Rust time sort and its comparisons are gone.advanceandsettle:CorgiChunk::concatuses the same block copy.Output is identical: the sort is stable, so each class's representative row and the order its diffs are summed in are unchanged.
Measured on one worker, three alternating trials, answers byte-identical:
Peak memory also falls (SCC −2.4%, reachability −14.3%).
sort_consolidate. DDIR: chunk times and diffs in adaptive columns #1059 adds a fast path for flushes whose times are all equal, which this PR's dropping of constant lanes covers. Whichever merges second needstime_wordsrewritten against the other'sColTimes.Checks:
cargo test -p differential-dataflow -p differential-dogs3 -p interactive -p ddir-server(485 passed) and the 1,000-case release difftest.Implemented and measured by a Claude subagent.
🤖 Generated with Claude Code
https://claude.ai/code/session_019XsakwFGwtESwZ1gSYwG4o