[antiguru]

The process-wide pool now takes its extent backend from a closure. apply_pool_config evaluates the closure exactly once, when it installs the pool.

Compute chooses the backend as follows:

• It uses the file store in the replica's scratch directory when the new enable_column_paged_batcher_file_extents dyncfg is on.
• It uses swap otherwise.
• If the file store fails to open, it falls back to swap with a warning.

The backend is fixed for the life of the process, so changing the flag later has no effect.

The flag defaults to off, and mzcompose turns it on so CI exercises the file store. New metrics export the file store's counters, its read latency, and which backend is installed.

🤖 Generated with Claude Code

[def-]

QA LLM Review

1. MEDIUM -- File store sizes itself to the whole shared scratch volume and keeps its peak, starving lgalloc and RocksDB

src/compute/src/compute_state.rs:559

capacity_bytes: None lets the pool's file store claim nearly all of /scratch, and it keeps every block it ever allocated. The same volume backs lgalloc's arenas and the upsert RocksDB state. After a demotion peak, lgalloc page faults hit ENOSPC and clusterd dies with SIGBUS, and RocksDB writes fail, which puts upsert sources into an error state.

Details

• None makes FileStore::open take the volume's free space minus max(1 GiB, 2%) (src/ore/src/pool/file.rs:220). That size is measured once, when the pool installs at instance creation, before lgalloc or RocksDB have grown. After that, capacity only ever goes down, after the store itself hits ENOSPC.
• FileStore::free leaves a freed slot's blocks allocated (src/ore/src/pool/file.rs:348). The store punches holes only inside alloc, to make room for its own allocations. So the pool's disk footprint stays at its demotion high-water mark for the life of the process, even after its chunks are gone.
• The same directory is lgalloc's path (compute_state.rs:415). enable_lgalloc and enable_columnation_lgalloc both default to true. lgalloc ftruncates sparse files and maps them MAP_SHARED, so every page it touches for the first time needs a free block. When none is left, the fault is SIGBUS. src/orchestrator-kubernetes/src/lib.rs:1362 already records exit 135 as "lgalloc runs out of disk". Upsert RocksDB lives under scratch/storage/upsert (src/storage/src/upsert.rs:588).
• Here is a concrete path on a replica with a scratch disk and no swap, with spill and this flag on. An index hydration demotes the batcher's spilled input to the file store. The batches seal and their chunks are freed, but the blocks stay allocated. The arrangement built from those batches is then allocated through lgalloc on a volume with only the 1 GiB / 2% headroom left.

Before this change the pool never wrote to scratch, so this contention is new. Possible fixes: pass an explicit bound, for example a dyncfg fraction of the volume as capacity_bytes: Some(..). Or let the store punch warm slots above a small cap when they are freed, so it hands space back once its data is gone. Or do both.

[bosconi]

The selection and fallback logic looks right, but the points from the previous round are still open. Details inline.

• Add the new flag to KNOWN_MISSING_FROM_LD in test/launchdarkly-flag-consistency/mzcompose.py next to its siblings, or the nightly launchdarkly-flag-consistency step fails unless an LD flag exists first.
• Nothing confirms CI runs on the file store. If CI's /scratch were memory-backed or the open failed, every replica would fall back to swap with CI still green. An mzcompose or testdrive check that a replica with spill on reports mz_column_pool_backend{backend=~"file_.*"} = 1 and mz_column_pool_backend_fallback = 0 would close that.
• def-'s comment about capacity_bytes: None sizing against the whole shared scratch volume still applies.

[bosconi]

When the volume has 1 GiB or less available, FileStore::open derives a capacity of 0 and still succeeds, so build_pool installs a store that can never take an extent, extents stay in RAM, and mz_column_pool_backend_fallback reads 0. An explicit bound would cover this and def-'s case: his dyncfg fraction, or a minimum capacity below which build_pool takes the swap path.

[bosconi]

This only covers installation. After a non-ENOSPC write error the store disables writes for its lifetime with no route back to swap, so say so here and in the mz_column_pool_backend_fallback help text.

Suggested change

	/// Whether [`apply_pool_config`] was asked for a file backend and installed
	/// the pool on swap instead.
	/// Whether [`apply_pool_config`] was asked for a file backend and installed
	/// the pool on swap instead. Covers installation only: a store that later
	/// disables writes on an I/O error reports through
	/// `mz_column_pool_extent_file_write_errors_total`.

[def-]

QA LLM Review

1. MEDIUM -- File store can take the shared scratch volume from lgalloc, the column pager and RocksDB

src/compute/src/compute_state.rs:567

Compute opens the file store with capacity_bytes: None, so the store's capacity is the free space on the replica's scratch volume at install time, minus max(1 GiB, 2%). The same volume holds lgalloc's mmapped files, the column pager's spill files and upsert RocksDB state, and none of them handle ENOSPC gracefully. Freed file slots keep their blocks, so a single spill peak can leave those writers without space for the rest of the process's life.

Details

Capacity is computed once, at src/ore/src/pool/file.rs:220, when the first config apply with a spill gate on installs the pool. That is near boot, before the other scratch users have grown. Once they grow by more than the 1 GiB or 2% headroom, the pool's fallocate keeps succeeding until the filesystem is actually full. The pool handles its own ENOSPC by lowering its capacity. The next writer to hit ENOSPC is one of the following, and it fails:

• lgalloc gets the scratch path in this same function (compute_state.rs:420). It grows sparse files with ftruncate and maps them, so a write that faults on a hole in a full filesystem raises SIGBUS.
• The column pager uses the File backend whenever scratch exists (compute_state.rs:505), and its pager::pageout_with panics on a write error (src/timely-util/src/column_pager.rs:441, :496).
• Upsert keeps RocksDB under scratch/storage/upsert (src/storage/src/upsert.rs:589), where a write error makes the source report an error.

FileStore::free puts a slot on the warm list and leaves its blocks allocated. punch_for releases them only when the pool's own allocation would exceed its capacity, so mz_column_pool_extent_file_allocated_bytes is a high-water mark the other writers never get back. Swap replicas get no scratch disk (src/controller/src/clusters.rs:735), so the flag only matters on disk replicas, which run lgalloc and RocksDB. The comment that the subdirectory prevents collisions covers file names, not space.

A fix is to pass an explicit capacity_bytes, for example a dyncfg fraction of the volume, so the pool's share of the scratch disk is planned alongside the other users.

[antiguru]

Posted by Codex.