Skip to content

feat(rules): describe runfiles groups for rules_runfiles_group - #4254

Draft
FrankPortman wants to merge 1 commit into
bazel-contrib:mainfrom
FrankPortman:runfiles_group_describers
Draft

FrankPortman wants to merge 1 commit into
bazel-contrib:mainfrom
FrankPortman:runfiles_group_describers

Conversation

@FrankPortman

Copy link
Copy Markdown
Contributor

Supersedes #3962, which targeted the provider-based API that rules_runfiles_group has since replaced (see bazelbuild/rules_java#368).

Groups are now collected on demand: a packager's aspect asks each target's describer what it adds and which attributes it merges from. The Python rules get two private attributes, and the describers live in python/private/runfiles_groups.bzl. Nothing runs unless a packager walks the graph, so the flags from #3962 are gone.

A py_binary / py_test gets these groups:

  • rules_python#runtime: interpreter and stdlib. Comes from a singleton target that resolves the same toolchain, so a packager builds it once per configuration instead of once per binary (same trick as the JDK group in rules_java).
  • rules_python#pypi/<name>: one per wheel, named by package since whl repo names aren't stable. weight is the file count.
  • One per py_library, and one per rule target in data. Plain files in data go in their owner's group.
  • The binary itself (executable, sources, bootstrap, build data, venv), as the executable_group.

A binary whose pyc_collection differs from the configuration's, or that generates legacy __init__.py files, keeps its deps in its own group, since the deps' groups wouldn't match its runfiles.

Describers only see providers, so this adds two fields: PyInfo.direct_implicit_pyc_files (the direct twin of transitive_implicit_pyc_files) and PyExecutableInfo.venv_runfiles.

Bazel 8 works too: only python/private/BUILD.bazel loads rules_runfiles_group, and the rule definitions refer to the describers by label, so there's no autoload cycle.

Testing: runfiles_group_analysis_test with overlapping_group_behavior = "error" checks that the groups add up to exactly default_runfiles for binaries with data that has default outputs, data whose outputs differ from its runfiles, a rule target in srcs, a binary in data, precompiling enabled, both fallbacks, and a py_test. Existing suites pass, and whl_library_test passes on 8.5.1 and 9.1.0. End to end with rules_img's rgi_aspect_wip: a py_binary with a pip dep gives 4 layers (runtime, six, its library, the app) on both 9.1.1 and 8.5.1.

Blocked on:

  • A rules_runfiles_group release. Until then .bazelrc points at the dev registry (marked to delete before merge), and the nested workspaces (examples, integration tests) fail to resolve it.
  • WORKSPACE: lib.bzl also loads the packager code, which needs bazel_features >= 1.51, so loading the describers would add a WORKSPACE setup step that rules_python doesn't need today. @malt3, could the describer side be loadable without the packager?

Known limitations: a plain file listed in several targets' data lands in each of their groups, and a py_library in data with precompiling enabled describes pyc files the binary doesn't get. Both overlap, neither drops files.

Written with heavy LLM assistance (Claude), steered by me; I understand it well enough to own it in review.

Fixes #3757

Give py_library, py_binary and py_test runfiles group describers, so
packaging rules that use rules_runfiles_group (e.g. rules_img's
image_from_binary) can split a binary's runfiles into layers: the
runtime, one group per PyPI package, one per library and per rule
target in data, and the binary's own files.

Groups are collected on demand by a packager's aspect, so nothing runs
otherwise. The runtime group comes from a singleton target resolving
the same toolchain as the binary, so a packager builds it once per
configuration. A binary whose pyc_collection differs from the
configuration's, or that generates legacy __init__.py files, keeps its
dependencies in its own group.

Describers only see providers, so this adds PyInfo.direct_implicit_pyc_files
and PyExecutableInfo.venv_runfiles.

Fixes bazel-contrib#3757

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Emit RunfilesGroupInfo in py_binary targets

1 participant