Skip to content

misc: add out-of-core hydration experiment harness - #40515

Draft
antiguru wants to merge 11 commits into
MaterializeInc:mainfrom
antiguru:ooc-nonupsert-measure
Draft

antiguru wants to merge 11 commits into
MaterializeInc:mainfrom
antiguru:ooc-nonupsert-measure

Conversation

@antiguru

@antiguru antiguru commented Nov 5, 2026

Copy link
Copy Markdown
Member

Adds the harness behind the October buffer-pool hydration measurements on staging. run.py measures TPC-H index and materialized-view hydration per arm, where each arm is a set of LaunchDarkly rules keyed on the replica's cluster name. summarize.py reports the hydration time, the speedup against columnation, and the peak RSS and swap per phase and arm.

The README lists the arms, the commands for the fits-in-memory (SF30) and memory-pressure (SF150) configurations, the cleanup each run needs, and where the pool's cost shows up in CPU profiles.

Posted by Claude Code.

🤖 Generated with Claude Code

antiguru and others added 11 commits November 5, 2026 17:46
Measure index hydration, MV first write and MV second write across
buffer-pool arms (columnation, chunked resident, pool without lz4, pool
with lz4 at 5% and 1% budget) on a staging region. Each arm owns one
single-replica cluster per workload, with flags set by LaunchDarkly
rules on the replica context's cluster name, so all arms run
concurrently.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`--parallel-trials` runs every trial of a phase at once, trial n on its own
clusters suffixed `_t<n>`. Sequential runs keep using the bare arm clusters.
The docs now point memory analysis at mz_cluster_replica_metrics_history and
at extent bytes for telling the raw arm apart.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Also applies bin/fmt to run.py.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add summarize.py, which reports hydration time, speedup against columnation,
and peak RSS and swap per phase and arm. Document the commands for the
measured configurations, the run hygiene they needed, and where the pool's
cost shows up in CPU profiles.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
List the flag, value, and arms of every rule, and give step-by-step
instructions for setting the rules up for another staging environment.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant