Use the symlink-resolved location of Python in getpath on 3.14+ by geofft · Pull Request #896 · astral-sh/python-build-standalone

geofft · 2025-12-01T19:32:19Z

Mostly fixes #380, and also fixes #713 because we prefer the actual
binary path to argv[0].

jjhelmus · 2025-12-02T22:23:24Z

Passing the path the the executable via the library variable in getpath.py has a smell as the variable is described as being the path to the dylib/DLL/so. When static linking there is no shared library so I see the logic in passing the path to the binary (as it is a DSO) itself.

jjhelmus · 2025-12-02T22:55:31Z

With this patch the directory pointed to by home in the pyvenv.cfg is still searched if the library search fails (but it should not fail).

The change from python/cpython#106045 (comment) (below) would update this path to the directory of the base_executable when it exists. I'm somewhat partial to that solution in addition to this change.

@@ -411,6 +412,9 @@ def search_up(prefix, *landmarks, test=isfile):
                             if isfile(candidate):
                                 base_executable = candidate
                                 break
+                if base_executable and isfile(base_executable):
+                    # Update the executable directory to be based on the resolved base executable
+                    executable_dir = real_executable_dir = dirname(base_executable)
             # home key found; stop iterating over lines
             break

jjhelmus · 2025-12-02T23:01:52Z

I've found following helpful check/reproduce/debug this issue:

./configure --prefix=/example
make -j
make DESTDIR=<some_real_path> install
cd <some_real_path>
./example/bin/python3.14 # can be used to create symlinks/venv/etc

zanieb · 2025-12-02T23:14:13Z

Did you mean to close this?

jjhelmus · 2025-12-02T23:16:57Z

Did you mean to close this?

Nope, oops

geofft · 2025-12-05T14:37:29Z

I think adding both this patch and the one in #903 is a good idea, though even with both patches I would consider it not quite a complete fix.

I've added some test cases based on #380 and #713. The patch in #903 by itself passes the tests for #380 but not the one in #713. (On the other hand the patch in #903 works on 3.11+.)

Getting this patch to work for 3.11-3.13 isn't hard, but I'm not sure if I will get to that today.

Mostly fixes #380, and also fixes #713 because we prefer the actual binary path to argv[0].

Patch CPython to allow venvs to be created from symlinks. When inside a venv and base_executable is passed available. Use the resolved path as the executable_dir rather than the one specified by home in pyvenv.cfg

zanieb · 2025-12-05T18:33:31Z

+        # TODO: does not yet work on ARM64
+        # with self.subTest(msg="weird argv[0]"):
+        #     assertPythonWorks(sys.executable, argv0="/dev/null")


Can this be a part of the skipif?

I don't think you can skipIf a single subTest. You can call self.skipTest("doesn't yet work on arm64"). I probably could have structured this one as a separate test case, though....

zanieb · 2025-12-05T19:32:38Z

Includes #903?

jjhelmus · 2025-12-05T19:34:16Z

Includes #903?

Yes, this supersedes #903

geofft · 2025-12-05T19:34:23Z

Yes, to fix the aarch64 failures.

…#293) ## Problem On macOS, `uv run pytest selftest/test_pinned_gate_tests_action.py` fails 11 of 25 tests on every local run. That includes a clean `origin/main` (2815beb). `Tests (Python)` passes on CI. Every failure is the same error: ``` Error: Command '[.../runner_temp/pinned-gate-tests.XXXX/venv/bin/python', '-m', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit status 1. ``` `_run_action()` made `shim/python` a **symlink** to `sys.executable` and put `shim/` first on PATH. That makes the action's bare `python -I -m venv "$venv"` run the test's interpreter. When CPython starts through a symlink (outside a macOS framework build), it keeps the symlink's own path as `sys._base_executable`. So `venv` writes `home = <tmp>/shim` into the new venv's `pyvenv.cfg`. That directory has no stdlib. The new venv's python looks for its stdlib starting from `home`, finds none there, and falls back: | Interpreter | Fallback | Result | |---|---|---| | CI's `setup-python`; Docker `python:3.13` (`/usr/local`) | compile-time prefix, which is where it is really installed | works by accident | | uv python-build-standalone with [astral-sh/python-build-standalone#896](astral-sh/python-build-standalone#896) (merged 2025-12-05; seen in 3.13.12 and 3.13.16) | patched getpath re-derives the directory from the symlink-resolved interpreter | works | | uv python-build-standalone without it (the 3.13.9 build uv cached here, built 2025-10-28) | compile-time prefix `/install` | `Could not find platform independent libraries <prefix>`, then `No module named 'encodings'`, so ensurepip exits 1 | So the bug is not specific to macOS. uv's cpython 3.13.9 fails the same 11 tests on Linux too. ## Fix - **The shim is now an exec wrapper.** `_write_python_shim()` writes `shim/python` as `#!/bin/sh` + `exec <realpath of sys.executable> "$@"`, with the path `shlex.quote`d. The interpreter starts from its real path, so `venv` records the real bin directory as `home`. No fallback is involved. - **The action still gets the same interpreter.** The old symlinked python also ran as the base interpreter, not inside the uv `.venv`: its `pyvenv.cfg` lookup happened next to the shim. Only the path venv sees has changed. - **`_run_action()` now checks every venv the action built.** - It checks that `home` is not the shim's directory, and that the python in `home` is this interpreter. - It compares interpreters, not directories. A macOS framework build (Homebrew) records `/opt/homebrew/opt/python@3.13/bin` as `home` whatever path it starts through, and that is correct there. - CI's interpreter hides the bug, so through a symlink shim the cases' own assertions stay green on CI. This check is what fails there, whether the symlink comes back in the helper or at `_run_action`'s own call. - It reads each workdir's `venv/pyvenv.cfg` with no existence check, so if the action's layout changes the check fails loudly instead of being skipped. The action itself is unchanged, and so are all existing assertions. The suite still has 25 tests. ## Verification All runs below are of the whole file. A mutant is the fixed file with the shim reverted to a symlink, in the helper or at `_run_action`'s call. | Interpreter | `origin/main` | Fixed | Helper back to symlink | Call site back to symlink | |---|---|---|---|---| | macOS (Darwin 25.5, arm64), uv cpython 3.13.9 | 11 failed, 14 passed | **25 passed** | 15 failed, 10 passed | 15 failed, 10 passed | | macOS, Homebrew 3.13.12 (framework build) | — | **25 passed** | 25 passed (bug can't occur) | — | | Linux (Docker `python:3.13`, arm64), `/usr/local` 3.13.16, standing in for `setup-python` | 25 passed (bug hidden) | **25 passed** | 15 failed, 10 passed | 15 failed, 10 passed | | Linux (Docker, arm64), uv cpython 3.13.9 | 11 failed, 14 passed (same ensurepip error) | **25 passed** | — | — | - **What the mutants fail on:** every mutant failure is the new check: `venv home is the python shim's dir`. The 10 cases that still pass are input validation and structural checks; none of them builds a venv. - **Exec-bit mutant (macOS):** a shim left non-executable fails loudly (`FileNotFoundError` on the missing `pyvenv.cfg`). - **ruff 0.15.10 (CI's pin):** `ruff format --check` and `ruff check` are clean. - **Docker runs:** each copies the worktree from a read-only mount, then runs `uv sync --frozen --all-extras` and `uv run --no-sync pytest`, like `Tests (Python)` does. - **Independent review:** a code-reviewer lane returned COMMENT with three findings, all addressed in this version. - The first guard compared directories and failed falsely on Homebrew. It now compares interpreters. - The guard didn't cover `_run_action`'s own call. The check now runs in `_run_action`. - Two docstring claims weren't true for framework builds. They are now qualified. - **Codex pre-review:** skipped; this is a selftest-only diff, below the threshold. **Not changed here:** on `origin/main`, four must-fail cases passed on macOS only because the venv build failed: `test_ungated_recursive_delete_fails` and three rows of `test_count_cases_fail`. They now fail at the new check instead. They still assert only `rc != 0`, so a failure after a good venv build would still satisfy them (pip unable to install from the wheelhouse, for example). Tightening them is a separate change. Auto-merge rationale: none. This is a manual merge, as for every topcoder1/ci-workflows PR (no automerge lane is wired on this repo). The diff is selftest-only (`safe_test`). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

geofft added platform:darwin Specific to the macOS platform platform:linux Specific to the Linux platform python:3.14 python:3.15 arch:x86_64 and removed python:3.15 labels Dec 1, 2025

geofft force-pushed the geofft/getpath branch from f6b631d to 43739ac Compare December 1, 2025 19:44

jjhelmus closed this Dec 2, 2025

jjhelmus reopened this Dec 2, 2025

geofft removed arch:x86_64 python:3.14 labels Dec 5, 2025

geofft force-pushed the geofft/getpath branch from 43739ac to 7e51c1e Compare December 5, 2025 14:34

geofft changed the title ~~Short-term getpath fix~~ Use the symlink-resolved location of Python in getpath on 3.14+ Dec 5, 2025

geofft marked this pull request as ready for review December 5, 2025 14:35

zanieb reviewed Dec 5, 2025

View reviewed changes

Comment thread cpython-unix/build-cpython.sh Resolved

Use the symlink-resolved location of Python in getpath on 3.14+

52917c5

Mostly fixes #380, and also fixes #713 because we prefer the actual binary path to argv[0].

geofft force-pushed the geofft/getpath branch from e0b5803 to 7033e22 Compare December 5, 2025 14:44

Add test cases for invoking Python via getpath

e1daa30

geofft force-pushed the geofft/getpath branch from 7033e22 to e1daa30 Compare December 5, 2025 14:45

geofft and others added 4 commits December 5, 2025 13:14

Disable tests on arm64 pending more adventures with pointers

10e1330

patch to allow venv creation from symlinks

3e4d886

Patch CPython to allow venvs to be created from symlinks. When inside a venv and base_executable is passed available. Use the resolved path as the executable_dir rather than the one specified by home in pyvenv.cfg

split patches for python version

b114efc

Jonathan's patch works as far back as 3.11

d0e2e16

zanieb reviewed Dec 5, 2025

View reviewed changes

jjhelmus approved these changes Dec 5, 2025

View reviewed changes

ruff ruff!

735385c

geofft added the ci:skip label Dec 5, 2025

zanieb approved these changes Dec 5, 2025

View reviewed changes

geofft merged commit 91c11ac into main Dec 5, 2025
7 of 19 checks passed

geofft deleted the geofft/getpath branch December 5, 2025 19:37

palfrey mentioned this pull request Jan 29, 2026

python_path symlink issue mitogen-hq/mitogen#1435

Closed

jjhelmus mentioned this pull request Feb 5, 2026

[Linux] Packages cannot be installed into / #972

Open

geofft mentioned this pull request Apr 30, 2026

Do not canonicalize the executable in getpath quite so much #1112

Open

topcoder1 mentioned this pull request Oct 8, 2026

fix(selftest): make the pinned-gate-tests python shim an exec wrapper topcoder1/ci-workflows#293

Merged

Navigation Menu

Repository navigation

Use the symlink-resolved location of Python in getpath on 3.14+ - #896

Use the symlink-resolved location of Python in getpath on 3.14+#896
geofft merged 7 commits into
mainfrom
geofft/getpath

geofft commented Dec 1, 2025 •

edited

Loading

Uh oh!

jjhelmus commented Dec 2, 2025

Uh oh!

jjhelmus commented Dec 2, 2025

Uh oh!

jjhelmus commented Dec 2, 2025

Uh oh!

zanieb commented Dec 2, 2025

Uh oh!

jjhelmus commented Dec 2, 2025

Uh oh!

Uh oh!

geofft commented Dec 5, 2025

Uh oh!

zanieb Dec 5, 2025

Uh oh!

geofft Dec 5, 2025

Uh oh!

zanieb commented Dec 5, 2025

Uh oh!

jjhelmus commented Dec 5, 2025 •

edited

Loading

Uh oh!

geofft commented Dec 5, 2025

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

geofft commented Dec 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jjhelmus commented Dec 2, 2025

Uh oh!

jjhelmus commented Dec 2, 2025

Uh oh!

jjhelmus commented Dec 2, 2025

Uh oh!

zanieb commented Dec 2, 2025

Uh oh!

jjhelmus commented Dec 2, 2025

Uh oh!

Uh oh!

geofft commented Dec 5, 2025

Uh oh!

zanieb Dec 5, 2025

Choose a reason for hiding this comment

Uh oh!

geofft Dec 5, 2025

Choose a reason for hiding this comment

Uh oh!

zanieb commented Dec 5, 2025

Uh oh!

jjhelmus commented Dec 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

geofft commented Dec 5, 2025

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

geofft commented Dec 1, 2025 •

edited

Loading

jjhelmus commented Dec 5, 2025 •

edited

Loading