Skip to content

🐛 fix(discovery): let -p replace the python set in config or env - #3494

Merged
gaborbernat merged 2 commits into
pypa:mainfrom
darrenhuai:fix/cli-python-overrides-config
Oct 30, 2026
Merged

gaborbernat merged 2 commits into
pypa:mainfrom
darrenhuai:fix/cli-python-overrides-config

Conversation

@darrenhuai

@darrenhuai darrenhuai commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

With python = 3.13 in virtualenv.ini, a -p on the command line loses to the config file. This machine has 3.13 and 3.14 installed and no 3.12. Run against main:

$ cat virtualenv.ini
[virtualenv]
python = 3.13
$ VIRTUALENV_CONFIG_FILE=virtualenv.ini python -m virtualenv -p 3.14 --no-seed env1
created virtual environment CPython3.13.7.final.0-64-x86_64 in 1415ms
$ VIRTUALENV_CONFIG_FILE=virtualenv.ini python -m virtualenv -p 3.12 --no-seed env2
created virtual environment CPython3.13.7.final.0-64-x86_64 in 614ms

Without the config file, -p 3.14 builds 3.14.3 and -p 3.12 stops with failed to find interpreter for Builtin discover of python_spec='3.12'. The env var goes wrong in a different way: VIRTUALENV_PYTHON=3.14,3.13 with no -p at all builds 3.13, so the second value wins.

-p/--python and --try-first-with are append actions. _fix_default seeds the namespace with the env var or ini value, argparse appends the command-line values after it, and Builtin.run returns the first spec that resolves, so the configured interpreter wins whenever it exists. #2821 patched this for VIRTUALENV_PYTHON only, by rotating python_spec by one whenever the variable is set. That puts a single env value behind -p, but with two or more env values it moves the second one to the front, and it never covered the config file. tox passes its base_python list as a newline-joined VIRTUALENV_PYTHON with no -p, so a tox env with more than one base_python gets the rotated order too. In #2285, the issue #2821 closed, the answer was that a CLI value should overwrite the env var.

Both options now use the ExtendAction from #3440, which already does this for --extra-search-dir: the first command-line value replaces the configured list and later ones are appended in order. nargs=1 is there because ExtendAction extends the list with whatever it receives, and a bare 3.14 would otherwise be split into characters. Since the parser now gets the order right, the rotation in Builtin is gone. A configured list is still tried in order when there is no -p, several -p are still all tried in order, and the help output doesn't change.

One behavior does change: VIRTUALENV_PYTHON=3.13 plus a -p that matches nothing used to fall back to 3.13 and now fails, the same as without the env var. That follows the documented priority (command line, then env vars, then the config file), and the changelog fragment says so. With this branch the two runs above build 3.14.3 and fail with failed to find interpreter, VIRTUALENV_PYTHON=3.14,3.13 builds 3.14.3, and the ini on its own still builds 3.13.7.

test_cli_replaces_configured_interpreters records what discovery tries for env and ini values with zero, one or two values on the command line, for both options. On main it fails whenever the command line has a value, and for two env values without one. The env-var-and-cli case in test_returns_python_specified encoded the old merged list plus the rotation, so I replaced it with an env-var-multiple case that expects the first env value.

Thanks for contributing, make sure you address all the checklists (for details on how see development documentation)

  • ran the linter to address style issues (tox -e fix). I ran the same hooks through prek on the changed files; the semgrep hook only looks at activate.sh.

  • wrote descriptive pull request text

  • ensured there are test(s) validating the fix

  • added a docs/changelog/+short-description.<type>.rst fragment for each user-facing change, or applied
    skip news

  • updated/extended the documentation: the multi-value options section of docs/how-to/usage.rst now says a command-line value replaces the configured list

  • for changes to activation scripts, pyvenv.cfg, wheel downloads, app-data or release workflows: explained in the
    PR text which outside input the change handles, such as a path, a prompt or a downloaded file, and how it stops that
    input from running as code. See the
    security scope. (none of these are touched; the change is in how the discovery options are parsed)

  • ran tests on Python 3.9 and 3.15 (tox run -e 3.9,3.15). I ran the discovery, config and seed-embed test files on 3.9 and 3.15 and the full suite on 3.13, and left the full 3.9 and 3.15 runs to CI.

  • linked upstream issues beside any temporary interpreter overrides (none)

  • kept the PR in draft until CI passes

--python and --try-first-with were plain append actions, so a value from virtualenv.ini or VIRTUALENV_PYTHON became
the starting list and the command-line values were added after it. Discovery takes the first spec that resolves, so
with python = 3.13 in virtualenv.ini, virtualenv -p 3.14 env built a 3.13 environment, and -p 3.12 on a machine
without 3.12 succeeded with 3.13 instead of reporting that no interpreter was found. Builtin worked around this for the
env var alone by rotating the list by one whenever VIRTUALENV_PYTHON was set. That covered a single env value, but with
several values it moved the second one to the front, with or without -p, and it never helped the config file.

Both options now use the ExtendAction from pypa#3440, which already does this for --extra-search-dir: the first
command-line value replaces the configured list and later ones are added in order. nargs=1 hands the action a one-item
list per flag, since it extends with whatever it receives. A configured list is still tried in order when the command
line gives none, and several -p are still all tried in order. The rotation in Builtin is gone because the parser now
gets the order right. One visible change follows from the precedence: a -p that matches nothing now fails the same way
it does without any configuration, instead of falling back to VIRTUALENV_PYTHON.
@darrenhuai
darrenhuai marked this pull request as ready for review October 30, 2026 01:47
Document that -p and the other multi-value options replace the configured
list, and why --python needs nargs=1 with ExtendAction. The precedence test
becomes a table of expected discovery calls with an autospec mock, so it
has no branches.
@gaborbernat
gaborbernat merged commit bf99d28 into pypa:main Oct 30, 2026
77 checks passed
EliahKagan added a commit to EliahKagan/GitPython that referenced this pull request Oct 31, 2026
The `ruff`, `format`, `mypy`, `html`, and `misc` environments are each
meant to run once, on the first available interpreter in a preference
order: Python 3.9 first, then 3.10, 3.11, 3.12, and 3.8. They wrote
that as `base_python = py{39,310,311,312,38}`, a form introduced in
62c024e (gitpython-developers#1693).

tox expands braces in generative environment lists and section names,
such as `env_list = py{38,...}` and `[testenv:py{38,39,310}]` in this
file, and in factor conditions, but not in the values of settings
like `base_python`. There, it just splits on commas, giving the five
specs `py{39`, `310`, `311`, `312`, and `38}`. The first and last are
not valid interpreter specs, so 3.9 and 3.8 were never candidates.
The bare `310`, `311`, and `312` are accepted as version specs, so the
environments still ran if 3.10, 3.11, or 3.12 was found. If none was,
the environment did not run: with tox 4.11.3 running on 3.13 and only
3.9 on `PATH`, it reported "skipped because could not find python
interpreter with spec(s): py{39, 310, 311, 312, 38}". Since tox 4.56.0
changed the default of `skip_missing_interpreters` to false, current
tox fails it instead.

Write each list as `py39, py310, py311, py312, py313, py314, py315`,
the documented form for several `base_python` values, of which the
first one found is used. This writes the intended order, except that
3.8 is left out, and adds the newer versions at the end. virtualenv
20.29 to 21.14 do not try them in that order, as described below.

62c024e listed as fallbacks all the Python versions the project
supported, but the lists were not extended when da58f39 (gitpython-developers#2196) added
3.13, 3.14, and 3.15 to `env_list`. So on a system with only 3.13 or
later, these environments could not run at all. With only 3.13, 3.14,
and 3.15 available, a probe with `py39, py310, py311, py312` was skipped
with tox 4.23.2 and virtualenv 20.28.1 and failed with tox 4.67.1 and
virtualenv 21.14.13, while the full list ran on 3.13 with both.

The brace form never selected 3.8, so leaving it out takes away
nothing that worked, and listing it could cause failures. Since
virtualenv 21.5.0, virtualenv cannot create Python 3.8 environments.
tox 4.56.0 and later work around that by bootstrapping an older
virtualenv, but only when the installed virtualenv can create none of
the `base_python` candidates, which is not the case for a list that
also names newer versions. virtualenv 20.29 to 21.14 also try the
first listed interpreter last, as described below, so 3.8 would be
tried before 3.9. With tox 4.67.1 and virtualenv 21.14.13 and only
3.8 and 3.9 available, a probe with `py39, py310, py311, py312, py38`
accepted 3.8 and then failed with "could not find python interpreter
matching any of the specs". With `py39, py310, py311, py312`, it ran
on 3.9.

Even where 3.8 environments can be created, `mypy` and `html` would
not work as intended on 3.8. The `test` extra installs mypy only on
3.9 and later, so the `mypy` environment would not have the `mypy`
command it runs, and `ignore_outcome` would turn that failure into a
warning. The `doc` group's Sphinx requirement also applies only on
3.9 and later, so `html` would build with whatever Sphinx version the
other documentation dependencies happened to pull in.

Checked with a minimal tox project whose only environment prints its
interpreter's version, with Python 3.9 and 3.12 on `PATH` and tox
itself running on 3.13.

| Tooling                          | Brace form | Comma form |
|----------------------------------|------------|------------|
| tox 4.11.3, virtualenv 20.24.5   | 3.12       | 3.9        |
| tox 4.23.2, virtualenv 20.28.1   | 3.12       | 3.9        |

`tox config` shows the intended entries for each environment.
With tox 4.23.2 and virtualenv 20.28.1, `tox -e ruff` and
`tox -e format` with this `tox.ini` ran on Python 3.9.6 and passed.

With tox 4.67.1, each of the five environments also ran and passed on
each of Python 3.13.17, 3.14.9, and 3.15.1, selected with
`-x testenv:<name>.base_python=py313` and so on. `html` reports no
Sphinx warnings on 3.13, but 13 on each of 3.14 and 3.15, all about
`__annotate_func__`, which Python 3.14 adds to the `__dict__` of
classes with annotations. The build used Sphinx 7.4.7, as the `doc`
group requires Sphinx below 8. CI's documentation build on 3.14
reports the same 13 warnings.

virtualenv 20.29 to 21.14 do not keep this order. tox passes the
`base_python` list to virtualenv in `VIRTUALENV_PYTHON`, and when that
variable is set, those versions move the first spec to the end of the
list. That change, pypa/virtualenv@bc7a91a5 (pypa/virtualenv#2821), was
meant to let a `--python` option take precedence over a single
interpreter set in the variable. So with tox 4.67.1 and virtualenv
21.14.13, the first listed interpreter was tried last: a probe with
`py39, py312, py311` ran on 3.12, and one with `py312, py39` ran on
3.9. virtualenv 21.15.2 tries the values in the order given again
(pypa/virtualenv#3494). With it and tox 4.67.1, a probe with
`py39, py313` ran on 3.9, where virtualenv 21.14.13 ran it on 3.13.
This was a problem in how tox and virtualenv interacted, not in this
configuration. With the affected virtualenv versions, these
environments are tried on 3.9 only if none of 3.10 through 3.15 is
found.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants