Repository navigation
Shrink the C macro documentation snapshot and name generated support items - #2391
Draft
eeeebbbbrrrr wants to merge 31 commits into
Draft
eeeebbbbrrrr wants to merge 31 commits into
eeeebbbbrrrr wants to merge 31 commits into
Conversation
Contributor
Author
|
The committed snapshot came from a macOS arm64 host with local
It should be regenerated on the usual release host before release: PGRX_PG_SYS_GENERATE_BINDINGS_FOR_RELEASE=1 cargo build -p pgrx-pg-sys --no-default-features --features pg18I kept the existing Linux Checks I ran:
Sizes: a Why there are version variants: 645 of 2,109 macros have more than one. About 300 come from changed |
This was referenced Oct 5, 2026
eeeebbbbrrrr
marked this pull request as draft
October 5, 2026 02:54
eeeebbbbrrrr
force-pushed
the
macro_rules-doc-snapshots
branch
4 times, most recently
from
October 29, 2026 08:13
b3eb299 to
366b011
Compare
eeeebbbbrrrr
force-pushed
the
macro_rules-doc-snapshots
branch
from
October 29, 2026 17:05
366b011 to
e9dcb62
Compare
Inline call adapters, function address getters, bitfield places, callback signatures and enum identities were named with hashes of compiler profile facts. The names were unreadable and changed whenever any hashed fact changed. They now come from the C function, record and field, or type spelling they adapt, such as `Inline_MemoryContextSwitchTo`, `Bitfield_ItemIdData__lp_off` and `EnumIdentity_enum__NodeTag`. Native helper symbols put their role before the C name, and multi-part names join plain identifiers with `__` while hex-encoding any other part, so each name identifies exactly one declaration. Anonymous type locations drop installation directories so names do not depend on where headers are installed.
The release snapshot stored each PostgreSQL version's full generated macro tree and native support, 179 MB that put the packaged pgrx-pg-sys at about 9.8 MB against crates.io's 10 MiB limit. docs.rs only needs what rustdoc shows, so release generation now writes one snapshot of documentation shells: each macro keeps its generated documentation and accepted invocation forms, and every arm expands to `unimplemented!()`. Shells identical across versions are written once behind `cfg(any(feature = ...))`, changed definitions sit next to each other in their header's module, and selecting each version from the merged tree must reproduce exactly that version's output. The snapshot is 5.6 MB, about 300 KB compressed, and the package estimate drops to 2.2 MB. Crates documented on docs.rs can still call these macros because rustdoc does not type-check function bodies, but a crate that docs.rs compiles as a dependency cannot use a shell's result. Header modules whose stem collides now use the whole filename, such as `port_h`, before falling back to hex.
Every shell arm repeated the same `unimplemented!()` call and message. The snapshot root now defines one hidden `__pgrx_c_documentation_shell!` macro, supplied by `pgrx-c-macros` next to the shell renderer, and each arm expands to it, so the message appears once and the snapshot drops from 5.6 MB to 5.4 MB.
Every documentation shell repeated the same `# Safety` contract, and many repeated other fixed paragraphs such as the inline adapter and caller-return notes. Those paragraphs are now named constants in the emitter, and a shell line that matches one uses `#[doc = crate::__pgrx_c_doc_safety!()]` and similar hidden macros instead of the text. Rustdoc renders the same documentation, and expanding the references reproduces the previous snapshot exactly. The snapshot drops from 5.4 MB to 3.5 MB, about 250 KB compressed.
Documentation shells need to name a generated macro's result type without its body, and today they expand to `unimplemented!()`, whose type `!` rejects `.get()` and `.into_value()`. `type_witness::<M>()` names `M`'s C expression value type and panics if called. A dead generic function compiled in every build proves that each helper the generator may replace with a witness returns exactly that type: casts and implicit conversions return `M::Value`, comparisons and logical operators return C int, the sizeof, alignof and offsetof family returns size_t, and branch hints return C long. Changing one of those signatures now breaks the build instead of silently mistyping documentation.
…support docs.rs builds pgrx-pg-sys from a snapshot whose macro shells expanded to `unimplemented!()`, so a crate compiled against it could not call `.get()` or `.into_value()` on a result, and the snapshot lacked generated support such as NodeTag's NativeType, the integer bridges and the metadata getters. Each shell arm now has its real macro's result type: an expression macro renders its value arm with every helper call whose signature fixes the result replaced by `type_witness`, a statement macro yields `()`, and every arm still panics if run. As the real argument classifier does, extra arms read a `*pointer` or `(*pointer)` operand that the witness reads as a place through the pointer, keeping its `*mut` qualification. The snapshot also ships the exact Rust support each version's ordinary build includes, merged across versions per module by item text, and verification requires that each version's macros and support items are reproduced and that every support item a shell names exists in each version it is gated to. Release generation must now run on the single docs.rs target named in pgrx-pg-sys's Cargo.toml, inspects each version's headers rather than importing a target bundle, appends a SHA-256 generation id to each pgNN.rs that the snapshot asserts, and writes nothing to src/include until the merged snapshot verifies. A run with only some supported versions still publishes, with a compile error for each missing version in docs builds, and reports the snapshot size and per-version result counts. The committed bindings and snapshot must be regenerated together on x86_64 Linux.
The existing docs.rs check documents pgrx with rustdoc, which does not type-check function bodies, and it checks one PostgreSQL version. `tools/docsrs-typecheck.sh` runs `cargo check` of pgrx-pg-sys's library and its generated C macro test with DOCS_RS=1 on the docs.rs target for each version in DOCSRS_PG_VERSIONS, 15 through 19 by default, against the committed bindings and snapshot. It then requires the packaged pgrx-pg-sys crate to stay within 5 MiB. publish.sh runs it after the existing check.
The architecture notes, the pgrx-c-macros README and the extension guide described shells that only show invocation forms and said dependents cannot call `.get()` under docs.rs. They now describe typed shells and their witnesses, the generated support the snapshot ships, how release generation keeps bindings and snapshot coherent, and the cases in which a shell's types differ from its real macro's.
Witness rendering resolved every parameter of a direct native call before choosing between the witness and the real arm, in source order. The real arm used to resolve them last parameter first, so a call with several rejected parameters reported a different skip reason than before. Resolution now runs in the original order, and generated macros and their reports are unchanged.
The testing section said the documentation snapshot needs its own validation without naming it. It now points to the documentation oracle, which compiles real macros beside their shells, and to the per-version docs.rs type check. The architecture notes and extension guide also said a shell never rejects what its real macro accepts and then listed cases where it does; they now say shells match their real macros except in those listed cases.
The documented list of invocations that shells reject was inaccurate. It left out nested generated macros passed where the C macro needs its operand's place, which fail with E0745. It named dereferences that shells read through the pointer like their real macros, and it listed large and i64-suffixed literals, which the real macros and shells reject alike. The remaining dereference cases are other spellings, such as `((*pointer))`, and macros whose witness reads several operands as places. The documentation oracle now compares shells over place, read-place, `*pointer` and `(*pointer)` operands, compound and postfix updates, a non-collapsed null constant, a bitfield and an integer-storage typedef. It also requires the real macros to accept each listed divergence and the shells to reject it, so the list cannot drift silently.
Text after a version's last root support item became a gated snapshot item. When that text was only a comment, its `cfg` attribute applied to the next merged item, so an item later versions add could vanish from every build while verification, which compares only the in-memory merge, still passed. Support splitting now returns that trailing text separately, and the snapshot drops it. After rendering, the merged support must also split back into exactly its entries, each preceded by its own gate.
The id each pgNN.rs records is a hash of its bindings, so the snapshot's assertion rejects a snapshot generated with different bindings. It cannot tell two runs apart when they produce the same bindings and different macros, yet the architecture notes and the assertion message said it detects any snapshot and bindings from different runs.
The native metadata getters' link names hash the compilation profile, which records absolute header and checkout paths, HOME, PGRX_HOME and other environment values. Release generation copied those names into the committed snapshot, so regenerating from another checkout or machine rewrote it even with identical headers. docs.rs never links these getters, so the snapshot now names them with a fixed suffix; ordinary builds keep their hashed names.
Release generation configures versions from PGRX_HOME without checking the active pgNN feature's version, and it formatted each pgNN.rs in place after writing the snapshot. So a run missing the active version, or one without rustfmt, failed after rewriting src/include, which the docs said could not happen. It now requires the active version first and formats the snapshot and bindings in a staging directory before changing src/include. The release scripts run cargo with -q, which hid the warning about a partial snapshot, so prepare-release.sh sets PGRX_PG_SYS_RELEASE_REQUIRE_ALL_VERSIONS=1 to make a run that does not configure every supported version fail, and the snapshot root records each version's shell counts.
docsrs-typecheck.sh packaged pgrx-pg-sys alone, so on every release Cargo tried to resolve the new, unpublished pgrx-macros, pgrx-sql-entity-graph and pgrx-bindgen from crates.io and publish.sh stopped. It now packages those internal dependencies together. docsrs-check.sh documented the host target off x86_64 Linux, which the committed snapshot's target guard now rejects, so both scripts build for pgrx-pg-sys's docs.rs target and fail early with instructions when the host lacks that target's standard library or a C compiler for it. The release docs state these host requirements.
Two tests in generated_c_macros.rs were excluded from docs builds because the documentation shells used to have type `!`. With typed shells they compile there, and excluding them kept tools/docsrs-typecheck.sh from checking the snapshot's member projection, enum field and binding-constant shells, such as nodeTag! and BUFFERALIGN!. Docs builds still only type-check these tests.
prepare-release.sh committed with `git commit -a`, which skips untracked files. A regeneration that adds snapshot files, such as a new shared support chunk, would produce a release commit whose snapshot module names files it does not contain.
A release run whose version was selected through PGRX_TARGET_INFO_PATH_PGNN inspected every version and then failed, because a target bundle has no C macro documentation to publish. The error also named the unsuffixed variable even when the target-suffixed spelling selected the bundle. Release generation now checks every configured version before generating and names the spelling that was set. Unit and release harness tests cover the refusal.
publish.sh checked pgrx's docs build only for pg15, and the pgrx-pg-sys check looped over a fixed version list. docs.rs builds pgrx against the shells of an extension's chosen version, so a divergence in pg16 to pg19 could pass every release check. tools/docsrs-typecheck.sh now also checks pgrx for each version, and takes the versions from pgrx-pg-sys's pgNN features.
ReleaseVersion stored the PostgreSQL major version beside the snapshot, which already records it, and nothing kept the two equal. A release built another way could publish pgNN.rs files that do not match the snapshot versions checked against them. publish_release now reads the major version from the snapshot.
The export test still wrote per-version macro trees under src/include/cmacros, a layout the single documentation snapshot replaced. Those trees served only as copy sources for OUT_DIR. The test also compiled and checked the same docs.rs library twice per version, once for each support mode that docs.rs builds ignore. It now writes the generated trees straight to OUT_DIR and checks each version's docs.rs build once.
The documentation oracle pinned only the assignment and member-read cases of a nested generated macro passed where the C macro needs its operand's place. The documented increment and address-of cases were unchecked, and no compared operand was a *const pointer, so the new dereference arms' qualification for *const rested on manual probes. The oracle now requires shells to reject the increment and address-of cases, and compares every dereference spelling of a *const pointer with the real macros.
… shells Documentation shells matched every operand except a lone type or identifier as an expression. Real macros take raw tokens for an unused or only stringified argument, and an identifier path for an offsetof member. A docs build therefore rejected calls such as Assert!() on a non-assert build, or a member named by a Rust keyword, that ordinary builds accept. Shells now use the real normalizer's fragments, and a last unparsed argument takes any tokens. Such an argument before another must still be an expression, which the guide and the oracle record.
Analysis admits operand stringification only as an argument of a direct void native call. A witness collapses such a call without rendering its arguments, or drops it as a comma's left operand, so the witness branch that wrapped a stringified operand and the shell's matching rewrite never ran. Both are gone, and a witness that reaches stringification now fails, which leaves its shell untyped. The oracle's typed DOC_REPORT and DOC_SPELL_NAME witnesses cover this.
A shell's witness collapses native calls, comparisons and casts, drops a comma's left operand, and a statement shell renders no body. When those were the real macro's only unsafe operations, the shell compiles without an unsafe block, and an unsafe block around it warns as unused in docs builds. The design notes and the guide did not list this among the ways a shell accepts more than its real macro. They do now, and the oracle pins a native call, a comparison through a pointer and a statement body.
The committed snapshot still held untyped shells from before typed documentation shells, generated support and the generation check, so the docs typecheck that publish.sh runs failed with 17 errors for pg15. Release generation now ran on x86_64-unknown-linux-gnu with PGDG PostgreSQL 15.19, 16.15, 17.11, 18.6 and 19rc1 configured and every version required. Every shell is typed, and the bindings differ from #2389's regeneration only by their generation ids.
tools/docsrs-typecheck.sh checks that pgrx-pg-sys and pgrx compile as docs.rs builds them, against the committed bindings and C macro snapshot, for each supported PostgreSQL version. Only release scripts ran it, so a change that broke the documentation build of a version not covered by the pg15 package test went unnoticed until publishing. A new package workflow job now runs it on every pull request.
The documentation snapshot rendered its support include paths from a PathBuf, so on Windows its index named `__pgrx_c_support\\shared_0000.rs` and the snapshot unit test failed. Support fragments now return their `include!` items written with `/`, and both the snapshot and the per-version trees use them. Snapshot generation also fails when two of its files differ only in case, because merging versions could otherwise commit a tree that case-insensitive filesystems cannot check out. The committed snapshot is unchanged.
Generated support names encode anonymous C types by their location, keeping only the path below PostgreSQL's server include directory. That directory was found only in paths separated by `/`, so a Windows path fell back to the bare filename and gave a different name than Linux for the same header. A directory such as `Program Files (x86)` also ended the location at its parenthesis. Either separator now ends a directory, the location ends at the parenthesis after its line and column, and the kept path is written with `/`.
The rustdoc for documentation_shell_support linked to the private SHARED_PARAGRAPHS constant, so `cargo doc -p pgrx-c-macros` failed when warnings were denied. The comment now describes the shared paragraphs without naming the constant.
eeeebbbbrrrr
force-pushed
the
macro_rules-doc-snapshots
branch
from
October 29, 2026 20:35
e9dcb62 to
07dec93
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #2389. Its release snapshot stored every PostgreSQL version's full generated macro tree and native support, 179 MB that put the packaged pgrx-pg-sys at about 9.8 MB against crates.io's 10 MiB limit. Release generation now writes one snapshot for all versions, with a documentation shell per macro and the generated support items that dependents compile against, shared across versions where identical, which brings the package to 2.9 MB. A shell keeps the macro's documentation and invocation forms but evaluates to a never-run value of the real macro's result type, so docs.rs can compile pgrx and extensions that call generated macros. CI now runs
tools/docsrs-typecheck.shto type-check those docs builds for every version, and an oracle test pins the documented cases where a shell differs from its macro. Release generation also writes the bindings and snapshot together or not at all and can require every version, and both are regenerated here on x86_64 Linux, the docs.rs target. Generated support items are named after the C declarations they adapt, such asInline_MemoryContextSwitchToandBitfield_ItemIdData__lp_off, instead of hashes, so their names no longer change when unrelated compiler facts change.