Skip to content

build: Limit unnecessary build-script reruns - #40442

Merged
def- merged 2 commits into
MaterializeInc:mainfrom
def-:pr-track
Nov 3, 2026
Merged

def- merged 2 commits into
MaterializeInc:mainfrom
def-:pr-track

Conversation

@def-

@def- def- commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Declare explicit inputs for AST and timezone generation, catalog schema processing, native source compilation, and build metadata so unrelated package changes avoid unnecessary build-script runs.

This avoids rerunning build scripts after unrelated package files change.
Track asset directories so additions and removals still trigger rebuilds.
@def-
def- requested a review from antiguru November 3, 2026 08:55
@def-
def- requested review from a team, DAlperin and SangJunBak as code owners November 3, 2026 08:55
@def-

def- commented Nov 3, 2026

Copy link
Copy Markdown
Contributor Author

QA LLM Review

1. MEDIUM -- Watching the npm static dirs makes every CI build recompile environmentd, clusterd and all binaries a second time

src/npm/src/lib.rs:202

ensure() now watches src/http/static and src/http/static-dev, but the same build script downloads the npm packages into those trees ({css,js}/vendor). On a checkout without vendor files, which is every CI job, the next cargo invocation finds the watched dirs newer than the build-script run. It then reruns the mz-environmentd and mz-prof-http build scripts and rebuilds everything downstream, including mz-alloc and every binary that links it. This PR's own CI shows the effect: back-to-back builds that used to be no-ops now recompile.

Details

Buildkite checkouts run git clean -ffxdq -e target-xcompile -e target ..., which deletes the gitignored vendor files but keeps the target dir. Both mzbuild (misc/python/materialize/mzbuild.py:848-858, a second --message-format=json pass that its comment calls "nearly instantaneous") and cargo-test (nextest run --no-run, then nextest run) invoke cargo twice with identical args. On this PR's build:

  • build-x86_64: the second pass recompiles mz-prof-http, mz-environmentd, mz-alloc, mz-clusterd, mz-materialized and takes 13.27s.
  • cargo-test: the second build recompiles 18 workspace crates and takes 28.56s. The same step took 1.15s on a recent build without this change.

Both of those runs use incremental profiles. Release builds (tags, nightly, release-qualification) use --release, which is thin LTO and non-incremental, so the second pass there repeats the environmentd/clusterd codegen and the LTO link of every binary. I did not measure an LTO run. In a recent nightly, that work falls inside the final ~16 minutes of a 23m37s build.

The rerun comes from directory mtimes. The unpacked files keep npm's 1985 timestamps, but create_dir_all gives the new vendor directories the current time, and cargo's directory watch covers subdirectories. Leaving the vendor subtrees out of the watch (listing committed assets instead of their parent dirs), or downloading outside the watched tree, avoids it.

@def-
def- enabled auto-merge (squash) November 3, 2026 10:26
@def-
def- merged commit 4f62eec into MaterializeInc:main Nov 3, 2026
84 checks passed
@def-
def- deleted the pr-track branch November 3, 2026 10:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants