Repository navigation
Transpile PostgreSQL C macros into Rust macro_rules! - #2389
Draft
eeeebbbbrrrr wants to merge 161 commits into
Draft
eeeebbbbrrrr wants to merge 161 commits into
eeeebbbbrrrr wants to merge 161 commits into
Conversation
Discover macro definitions from configured PostgreSQL headers with pgrx-pg-config and Clang. Provide an owned library inventory and a CLI with a required positional PostgreSQL version.
Macro discovery now scans pgrx's complete version-specific header wrapper and records physical files and line spans. The CLI and primary library inventory contain only function macros from the selected PostgreSQL server header tree, with other definitions retained as conversion context.
Inspect the selected PostgreSQL headers with matching Clang and emit a bounded integer macro_rules family during pgrx-pg-sys binding generation. Generated macros preserve C type identities, promotions, conversions, evaluation counts and overflow policy, while unsupported constructs retain source spans and skip reasons. Track compiler, flags, headers and include-search changes for regeneration. Differential fixtures compare generated Rust with original C across PostgreSQL 15–19.
Preserve verified PostgreSQL binding names, literal spellings, and direct macro calls in generated Rust macros. Track dependencies with petgraph and skip macros and their callers when bindgen and Clang disagree on a constant, reporting both values without changing bindgen output. Include original C definitions in documentation and ship PostgreSQL 15–19 macro snapshots for docs.rs.
Generate pointer, field, enum, callback, mutation and function-call adapters from Clang declarations and fresh bindgen output without handwritten macro signatures. Preserve C evaluation contexts and partially initialized storage, compile native access primitives under the inspected profile, and retain PostgreSQL's FFI guard. Expose generated adapters through the docs.rs include path and reject unsupported constructs or inconsistent bindings with dependency-aware diagnostics. Add differential tests against original C, including the macros pgrx previously ported by hand.
Avoid recording preprocessing inventories in declaration-only Clang probes, and share snapshot checks between adjacent compiler phases. Index C and Rust storage identities once per immutable emission pass instead of once per macro. Optimize SHA-2 in debug builds so CLI input verification does not dominate runtime. Preserve compiler input checks, dependency skips, and generated output.
Generated macros now execute ordered C statements, preserve nested scopes and local initialization, and apply C assignment conversion to caller returns. Caller-scope captures are explicit operands, while guards reject local capture that Rust hygiene cannot reproduce. Compiler assertions remain skips instead of becoming runtime calls, and C-oracle tests cover sequencing, conversions, mutations, and unused arguments.
Generated macros now preserve if/else control flow, caller returns, and local initialization across every surviving branch. Unbraced conditionals require an explicit statement boundary. Original argument names and verified Rust type aliases remain visible while C rank, qualifiers, and null semantics stay intact. Refresh the documentation snapshots from the current generated bindings and isolate their native adapters from docs.rs builds.
Verify builtin prototypes and dynamic byte-swap behavior under the selected compiler profile before lowering to Rust. Preserve C argument conversions, result ranks, evaluation counts, and closed null constants, while isolating rejected intrinsics. Track canonical identities for availability-only header dependencies so unchanged symlinks pass validation and changed inputs are rejected. Refresh the PG15–19 macro snapshots and compare generated expressions with original C in standalone and binding integration tests.
Transpile named, generic, and dotted member offsets without reading objects, including packed, volatile, flexible, and unsupported value fields. Verify the C size_t rank and share layout checks across field adapters to keep generation within its source budget.
Prove the exact long(long, long) prototype and first-operand identity under the selected compiler profile. Preserve evaluation and C conversion of both operands, including side effects, and retain the existing byte-swap failure isolation. Add independent C oracles and PostgreSQL integration tests, and refresh the PG15–19 macro snapshots from host-generated bindings.
Use bounded Clang probes to resolve fixed token pastes while rejecting pastes that construct identifiers from caller operands. Track synthesized helpers and constants in the dependency graph so unsupported constructs and binding disagreements still skip every affected macro. Add independent C oracles and PG15-19 integration tests, and refresh macro snapshots from the host-generated bindings.
Move generated PostgreSQL macros into formatted modules by source header and expose them through pgrx. Share expression bodies and restrict native adapters to required capabilities while preserving C parameter names, aliases, evaluation and branch prediction hints. Reject ambiguous type applications and callback identities, propagate adapter failures, and validate fresh compiler artifacts instead of accepting stale output.
Credit pgrx-c-macros in the generated headers and describe regeneration of macros and their support code.
Publish pgrx-c-macros after pgrx-pg-config and before pgrx-bindgen, and include it in the preflight packaging check.
Remove the develop-only pull request filters from the test, packaging, and runas workflows.
Explain handwritten declarations, private helpers, modules, and tests across the C macro pipeline. Document the runtime access and conversion contracts while leaving generated snapshots unchanged.
eeeebbbbrrrr
force-pushed
the
macro_rules
branch
from
October 4, 2026 02:46
c3d3433 to
e0ebb0a
Compare
Keep the module, item, and test documentation while removing doc comments from use declarations.
Keep formatting optional when rustup lacks rustfmt, and handle Clang dependency paths and diagnostic excerpts across platforms. Accept reviewed LTO flags while producing ordinary native objects for the Rust linker. Compile native macro helpers and the C shim together for the active PostgreSQL version to avoid duplicate header definitions, and retain incompatible-pointer rejection coverage with the current compiler diagnostic.
eeeebbbbrrrr
marked this pull request as draft
October 4, 2026 03:33
Preprocess native helpers with the inspected flags before compiling PIC machine code, so backend flags cannot change header branches. Move PageSetPrunable coverage into the PostgreSQL harness and normalize Windows C stdio line endings in numeric oracle comparisons.
Describe the transpiler end to end, add extension examples, and introduce the feature in the top-level README. Enable macro warnings only with PGRX_MACRO_DEBUG=1 while preserving audit reports and tracking setting changes for rebuilds. Correct singular argument descriptions in generated documentation.
Generated pgstat_count_* expansions reference guarded PostgreSQL functions even when their test inputs short-circuit before calling them, so standalone Linux test binaries cannot link. Move the existing assertions and interrupt compile witnesses into pgrx-unit-tests while keeping pure prediction-hint checks in pgrx-pg-sys.
Select pgrx-c-macros alongside pgrx-bindgen so Cargo can resolve the local crate when testing unpublished release versions.
eeeebbbbrrrr
commented
Oct 4, 2026
eeeebbbbrrrr
left a comment
eeeebbbbrrrr
left a comment
Contributor
Author
There was a problem hiding this comment.
Static review of 5d0fc41. I didn't build or run anything. I read the hand-written code in pgrx-c-macros, pgrx-bindgen and the pgrx-pg-sys runtime, and sampled the pg15–19 snapshots.
What looks right
- Runtime: I found no path from safe code to UB. Every memory-touching entry point is an
unsafe fn, and the traits it trusts areunsafeor sealed. The generatedunsafe {}blocks that actually run never contain caller tokens. Native calls go throughpg_guard_ffi_boundary. - C typing: integer promotions, the usual arithmetic conversions, literal typing, shifts and
?:match the C rules for LP64.
What blocks this
- Builds that now fail: macro inspection errors fail the
pgrx-pg-sysbuild, with no opt-out. Three configurations that build ondevelopfail here:- RPM/Arch CFLAGS containing
-Wp,; - the cross-compile recipe used in CI and the book;
- hosts without a Clang driver that matches libclang.
- RPM/Arch CFLAGS containing
- Builds that silently lose every macro: Linux aarch64, and any profile with an unmodeled
-fflag. They compile, but export zero macros and print no warning.
Inline comments are tagged P0 (breaks a currently supported build), P1 (risk under specific conditions) or P2 (hardening or quality). "Latent" means the current PG15–19 output doesn't hit it. Three top-level comments cover the larger changes: failing soft, public API stability, and size and build cost.
The PR description should mention the new Clang-driver requirement and the change that passes CFLAGS to bindgen, since both affect existing users.
Contributor
Author
Larger change 1: make macro generation unable to break or silently empty a
|
Contributor
Author
Larger change 2: the public macro API varies by host; give users a way to deal with thatWhich
An extension that compiles against a pgrx-managed install can therefore fail against a distro or cassert build with only "cannot find macro Suggestions:
|
The varlena oracle passes -DUSE_ASSERT_CHECKING so that VARTAG_SIZE(0) reaches ExceptionalCondition. That only works when configure leaves the setting commented out in pg_config.h. Meson writes an active "#undef USE_ASSERT_CHECKING", so on PostgreSQL 17 for Windows the C oracle returned instead of failing and the test died. The test now reads whether USE_ASSERT_CHECKING is active after inspection, requires the invalid tag to fail exactly then, and otherwise compares the returned size and operand count with the generated macro.
The availability guard reported only the skip code, so the Windows failure showed budget_exceeded without saying which budget. It now prints the generator's skip message beside the code.
The closed token-paste proof poisons every active macro name plus one constant marker per macro, so its source grows with the inspected environment rather than the batch. PostgreSQL 17's headers on macOS already used 965 KB of the 1 MiB source budget, and adding 3,000 macros reproduces the Windows CI result: every closed paste, such as CALLED_AS_TRIGGER and IS_VALID_TIMESTAMP, skipped with budget_exceeded. The Windows SDK headers that PostgreSQL includes define more macros than that. The poison table is now rendered once and excluded from the budget, which still bounds the rest of the instrumentation; inspection already bounds the environment itself.
The braced token-paste pass refused every closed paste with budget_exceeded when the translation unit's whole -E output exceeded the 8 MiB expansion budget. That output starts with every declaration in the inspected headers, so it grows with the environment, while the authoritative pass budgets only probe bodies. Headers that print more than 8 MiB therefore emitted every ordinary macro and skipped every paste macro, the same symptom as the Windows CI paste skips. The pass now counts output from the first probe boundary on, and the frontend's 16 MiB limit still bounds the whole output.
With Microsoft extensions, which Clang enables by default for MSVC targets, a paste that forms // comments out the rest of the invoking line without a diagnostic. The paste proof admitted such a macro, so its generated Rust dropped nothing while C silently discarded the code after the call. The probe pragmas now make -Wmicrosoft-comment-paste an error, a group every Clang from 6 on has. Because -w silences even that error, the braced pass also ends each probe line with a tail marker and refuses a probe whose tail is missing.
The docs said the uncounted poison table names only the active macros and their constant markers, so it grows with the environment rather than the batch. It also names each prepared probe's markers, boundaries, wrapper and tail. That part is still bounded, by the macro count limit and by the invocation bytes that preparation already charged against the source budget.
The varlena oracle took its expected assertion branch from the frontend under test, so a profile that lost -DUSE_ASSERT_CHECKING would compare the unasserted branch everywhere and still pass. The test now reads the installation's pg_config.h and expects the unasserted branch only when it ends with an active #undef USE_ASSERT_CHECKING, as meson writes. It also requires the inspected environment to agree before comparing C and Rust.
pgrx and pgrx-macros' output templates call generated macros as pg_sys::NAME!. When the generator skips one, as Windows did for closed pastes, the build fails only with "cannot find macro" and never shows the generator's reason. The availability test now parses those crates from their roots, drops code that the build's PostgreSQL feature or target OS compiles out, and requires every remaining call to be emitted, printing the skip reason otherwise. Calls to pgrx-pg-sys's own exported macros, such as error!, are left out.
From Clang 9 on, -w drops the error the probe pragmas give a paste that forms a // comment under Microsoft extensions. Such a paste inside a macro argument drops the rest of the argument without cutting the line, so the paste proof admitted a macro whose C expansion silently lost tokens. The plain pass now pastes a canary // after the probe pragmas, and Clang must report it there. Without that report the proof refuses every closed paste, because nothing else shows such a comment. Profiles without Microsoft extensions are unaffected, since there // is an invalid paste that -w cannot silence.
The braced pass ended each probe line with a tail marker so that, under -w, a comment paste that cut the line short was still refused. The canary now refuses every closed paste whenever Clang drops the comment-paste error, and otherwise that error refuses the probe itself, so a missing tail no longer changes any result. This restores the braced pass and its boundary guard to their form before the marker, keeping the probe-only output budget.
Under Microsoft extensions, the braced pass refused a closed paste that forms a // comment with the generic message about pastes that depend on invocation operands. That message is wrong for these pastes, and it is what Windows users and the pgrx availability test print. A probe that owns Clang's comment-paste error is now refused with a reason that names the comment, still as TokenPaste.
The paste proof's docs and its Microsoft comment-paste test said Clang 11 is the first release whose -w drops the error that the probe pragmas map from an extension. The tagged sources show the change in LLVM 9: llvmorg-8.0.0 ignores only warnings under -w, while llvmorg-9.0.0 also ignores errors that are not errors by default. The test now expects the canary refusal under -w from upstream Clang 9 and Apple clang 12.
Two pgrx-pg-config tests write a shell script and run it as pg_config. When one test forked to run its script while the other still had its own script open for writing, the child inherited that descriptor, and running the second script failed with ETXTBSY ("Text file busy"). That failed Linux CI jobs on this branch and the one stacked on it. Both tests now go through one helper that holds a lock from writing its script until the script's process exits.
The expansion probes begin with a pragma that ignores -Wgnu-line-marker, a group that Clang 14 does not have. Naming an unknown group is itself a warning, so under -Werror or -pedantic-errors every probe failed and Clang 14 skipped every function-like macro. The pragma is now guarded by __has_warning, as is the same pragma in a test header. A new test expands macros under both options from a header that is not a system header, where the warning is not silenced.
The Microsoft comment-paste test expected Clang with only -std=c17 -w to leave Microsoft extensions off, but a Windows host's default target is MSVC, which turns them on, so the test failed on Windows CI. Its profiles that choose Microsoft extensions explicitly now name a Linux target. Two other tests passed options that Clang 15 added, -fdriver-only and -ffp-eval-method=double, and failed on Clang 14, which rejects them. The native support test now also accepts Clang 14's rejection of -fdriver-only, still requiring the existing outputs to stay unchanged, and the floating-point test skips the evaluation-method case that Clang 14 cannot express.
The Linux job installs clang-14, but PGDG's postgresql-server-dev package installs clang-15 too, and the tests load the newest libclang, so CI never exercised Clang 14. Clang 14 is what Ubuntu 22.04 installs for `apt install clang`, and three tests and the probes' -Werror handling had already broken there unnoticed. One matrix entry now runs the pgrx-c-macros tests with LIBCLANG_PATH and CLANG_PATH pointing at Clang 14, in its own target directory.
emit_with_bindings never generates function adapters, so a direct call to a function whose prototype passes or returns plain char, an enum or a record by value used the caller's binding. That binding's Rust storage cannot prove the call ABI, which is why batch generation routes such calls through a generated C adapter. Generated adapters now live in BindingCatalog::function_adapters, and lowering calls a native-transport function only through one of them. Any entry point without that adapter now skips the macro with a reason naming the C type. Adapter generation and lowering share one predicate for which prototypes need native transport, and batch output is unchanged.
BindingCatalog::function_adapters was public, so a caller could fill it and make emit_with_bindings call a native-transport function directly through a binding whose ABI is unproven. Generated adapters now reach rendering as a separate map that only batch generation fills, and the field is gone. The single-emission refusal no longer says that generate_with_bindings generates adapters, because batch generation also refuses declarations such as a static function that is not inline. The native-transport tests move to their own file and now cover an enum parameter, an enum result through a typedef and a by-value record, so dropping any of these categories from the shared predicate fails a test.
Rendering took generated function adapters as a plain map of function bindings, so code in the crate could pass caller bindings in their place without a type error. It now takes the adapter generator's own output, and single-macro emission passes none. Output is unchanged.
BindingCatalog had nine public fields for state that only batch generation should produce: callback signatures, function address getters, field and offset adapters, and the refusals for these, for enums and for function adapters. A caller could fill them, and rendering then treated a hand-written callback signature or address getter as verified, even in single-macro emission, which generates none. Rendering now reads this state from the generated adapters passed beside the catalog, as it already did for function adapters, so the fields and register_adapters are gone. Adapter passes after the second callback pass still see the signatures of both passes, and generated output is unchanged. CallbackBinding and FunctionAddressBinding are no longer public, and the address getter's unread marker field is removed. ffi_boundary stays public because it is caller configuration.
ARCHITECTURE.md still showed batch generation registering its adapters in a derived binding catalog, a step the previous commit removed. It also said single-macro emission skips every macro that needs a generated item, but such a macro can still name a C enum's generated identity, which only batch support defines. The build pseudocode now passes the caller's bindings and the generated adapters separately. ARCHITECTURE.md, README.md and the emit_with_bindings docs now say that an invocation naming such an identity compiles only when the defining crate includes batch support for that enum.
The installed-header oracles link programs that include the pgrx wrapper header, which defines pg_checksum_page. Against a PostgreSQL built with assertions, as cargo pgrx init builds it, that function calls ExceptionalCondition, which nothing defines. macOS strips the unused function per symbol, but GNU ld's --gc-sections drops only whole sections, so four oracle tests failed to link on Linux. Each function and object now gets its own section, which --gc-sections can remove.
Several pgrx-c-macros tests write a shell script and then run it as a compiler, archiver or pg_config. On Linux, a child that another test thread forks while such a script is still open for writing keeps that descriptor until it execs, and running the script meanwhile fails with ETXTBSY ("Text file busy"). The compiler-selection library tests failed this way in three of twenty runs on a 64-core host. A lock around the scripts cannot help there, because the production code under test forks too. The scripts are now written by a short-lived /bin/sh, so this process never holds a descriptor that a concurrent child could inherit.
The pgrx-bindgen tests wrote their fake pg_config, rustfmt proxy and failing formatter with fs::write and then ran them. On Linux, a child that another test thread spawns while such a file is still open for writing keeps that descriptor until it execs, and running the file meanwhile fails with ETXTBSY ("Text file busy"). On a 64-core Linux host, server_include_overrides_own_generated_macros failed this way in 10 of 300 library test runs with eight test threads, and in none of 300 runs after this change. These tests, and the scripts the macro_build and shipped_macros harnesses write, now use the pgrx-c-macros helper that writes the file from a short-lived /bin/sh, so the test process never holds a descriptor a concurrent child could inherit.
The pg_config script tests avoided ETXTBSY with a lock held from writing a script until its process exited. That works only while no other test in the binary spawns a process, because a child forked by any test inherits a descriptor that is still open for writing. The script is now written by a short-lived /bin/sh, so this process never holds such a descriptor, and the lock is gone. A counter replaces the clock in the script name, since the tests can now name scripts in the same microsecond.
The C and Rust oracles and emit.rs killed a compiler or test program after 30 seconds, and the shipped_macros harness killed Cargo after 300 seconds, so a slow or loaded runner could fail a correct test. Killing only the direct child could also leave its linker or compilers writing into a fixture directory that the test then removed. The runners now read stdout and stderr through pipes with the same byte limits and no time limit. A stream past its limit is closed instead of its process killed, so the next write fails and the tool exits through its own error handling, which waits for the processes it started. A hung tool is left to libtest's slow-test report and the CI job timeout.
Every compiler and archiver call failed with FrontendError::Timeout after 60 seconds. Symbolic preprocessing, closed token-paste validation and constant probes treated that timeout like an output budget and skipped or rejected every pending macro, so a stalled compiler on a loaded machine silently generated fewer macros and pgrx later failed with "cannot find macro". The time limit and the Timeout variant are gone, and only the deterministic output limits remain, so the generated macros depend only on the inputs. A compiler that hangs is left to the surrounding build or CI job timeout, as for the bindgen and cc steps of the same build script.
universal_macos_installations_build_both_slices compiles only on macOS and is ignored, but the only CI step that runs ignored macro_build tests is on Linux. So no job ever ran it, and the end-to-end build of both slices of a Postgres.app-style installation went untested. The PG16 macOS job now installs the other slice's standard library and runs this test alone.
eeeebbbbrrrr
force-pushed
the
macro_rules
branch
from
October 29, 2026 17:05
5d5cf53 to
95afee4
Compare
A compiler killed by a signal, such as the Linux out-of-memory killer on a busy CI runner, was reported as an ordinary nonzero exit. The symbolic expansion, constant and builtin probes treated that as a rejection of their candidates, so a loaded machine could finish the build with far fewer macros. Signal termination is now its own FrontendError::CompilerTerminated, which no probe counts as a rejection, so the build fails instead. Compiler lookup also stops at a killed version query rather than falling back to a lower-preference compiler. The per-site signal checks in the closed-paste and offsetof probes are no longer needed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This adds
pgrx-c-macros, whichpgrx-pg-sysruns on every build to transpile PostgreSQL server macros into documented Rustmacro_rules!that keep C's arithmetic, promotions, qualifiers and evaluation order (see the architecture guide and extension chapter). Macros that cannot be translated faithfully are skipped with a reason inOUT_DIR/pgNN_macro_report.json, and tests compare the generated Rust with the original C. Generation is mandatory, so a build needs a Clang driver that matcheslibclang, fails with context instead of skipping macros, and no longer readsPGRX_C_MACROS. A driver that loads Clang default configuration files is refused unlessCLANG_NO_DEFAULT_CONFIG=1is set, because their options would not appear in the recorded profile. Cross builds must supply server headers configured for the target throughPGRX_INCLUDEDIR_SERVER[_PGNN][_<target>],PGRX_PG_CONFIG_AS_ENVorPGRX_TARGET_INFO_PATH_PGNN, and headers built for another machine are refused. The committed bindings and docs.rs snapshot come from one run on x86_64 Linux against PGDG non-assert PostgreSQL 15–19. CI's aarch64 cross build therefore fetches arm64 server headers from PGDG, and CI now also runs the CLI tests, the binding build harnesses and pgrx-unit-tests on arm64.