Python images (At least bookwork-slim) relased today break installation of sdist packages

Following the discussion her pypa/wheel#660 (comment) - images relesed today started to break CI - Airflow, but also other people have similar issue.

I explained it in the issue but let me copy it here as well:

Example failure: https://github.com/apache/airflow/actions/runs/14347343802/job/40220981486#step:6:809

6 13.37 Building wheels for collected packages: pyspark
docker-library/official-images#6 13.37   Building wheel for pyspark (setup.py): started
docker-library/official-images#6 14.11   Building wheel for pyspark (setup.py): finished with status 'error'
docker-library/official-images#6 14.11   error: subprocess-exited-with-error
docker-library/official-images#6 14.11   
docker-library/official-images#6 14.11   × python setup.py bdist_wheel did not run successfully.
docker-library/official-images#6 14.11   │ exit code: 1
docker-library/official-images#6 14.11   ╰─> [8 lines of output]
docker-library/official-images#6 14.11       /home/airflow/.local/lib/python3.9/site-packages/setuptools/dist.py:717: UserWarning: Usage of dash-separated 'description-file' will not be supported in future versions. Please use the underscore name 'description_file' instead
docker-library/official-images#6 14.11         warnings.warn(
docker-library/official-images#6 14.11       usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
docker-library/official-images#6 14.11          or: setup.py --help [cmd1 cmd2 ...]
docker-library/official-images#6 14.11          or: setup.py --help-commands
docker-library/official-images#6 14.11          or: setup.py cmd --help
docker-library/official-images#6 14.11       
docker-library/official-images#6 14.11       error: invalid command 'bdist_wheel'
docker-library/official-images#6 14.11       [end of output]
docker-library/official-images#6 14.11   
docker-library/official-images#6 14.11   note: This error originates from a subprocess, and is likely not a problem with pip.
docker-library/official-images#6 14.11   ERROR: Failed building wheel for pyspark
docker-library/official-images#6 14.12   Running setup.py clean for pyspark

And after some (non-obvious investigation I found out what the root cause is and that's somethign you might want to talk to the maintainers of official Python images in dockerhub, because probably a lot of people using them will have similar issue after 0.46+ wheel that does not contain bdist_wheel any more and rely on setuptools > 70.1.

The problem is that some - latest - official python images available in dockerhub are causing this problem to happen.

If you look - for example - at latest Python debian bookworm slim images you will see this:

[jarek:~/code/airflow] main ± docker run -it python:3.9-slim-bookworm bash
root@fb321ef4723e:/# pip list
Package    Version
---------- -------
pip        23.0.1
setuptools 58.1.0
wheel      0.45.1

[notice] A new release of pip is available: 23.0.1 -> 25.0.1
[notice] To update, run: pip install --upgrade pip
root@fb321ef4723e:/# exit
exit
[jarek:~/code/airflow] main 14s ± docker run -it python:3.10-slim-bookworm bash 
root@2234dee7aba0:/# pip list
Package    Version
---------- -------
packaging  24.2
pip        23.0.1
setuptools 65.5.1
wheel      0.46.1

[notice] A new release of pip is available: 23.0.1 -> 25.0.1
[notice] To update, run: pip install --upgrade pip
root@2234dee7aba0:/# exit
exit
[jarek:~/code/airflow] main 25s ± docker run -it python:3.11-slim-bookworm bash
root@b62343d129a6:/# pip list
Package    Version
---------- -------
packaging  24.2
pip        24.0
setuptools 65.5.1
wheel      0.46.1

[jarek:~/code/airflow] main 1m5s ± docker run -it python:3.12-slim-bookworm bash
root@411574e67a7e:/# pip list
Package Version
------- -------
pip     25.0.1
root@411574e67a7e:/#

As you can see - default version of setuptools in all those images causes problem:

Python 3.9 -> both wheel and setuptools are old, but if you just upgrade wheel but do not upgrade setuptools, sdist installations will fail
Python 3.10, 3.11 -> wheel is already updated, but setuptools is < 70.1 - which mean that by default installing somethign from sdist wil fail there - even if you do not touch setuptools and wheel
Python 3.12 -> all good, no setuptools, no wheel, newest versions of those will be installed

I am going to upgrade setuptools and wheels to latest versions in our images, so we will solve the problem in Airflow, but I think currently released "official images" need to be rebuild and latest setuptools should be installed in all of them.

@tianon

Hi Jarek!

The repo for the Docker Python images is:
https://github.com/docker-library/python/issues

(Maybe someone with permissions can use the issue transfer feature to move this issue there? I'm not a maintainer)

Those images intentionally don't use latest setuptools (since there are breaking changes in newer setuptools), but instead use the same version provided by the ensurepip command (with a slight fixup for 65.5.0 -> 65.5.1).

The issue is that ensurepip doesn't include wheel (only pip and setuptools; at least until Python 3.12+ where it also drops setuptools), so the Python docker image scripts don't have a "reference" version of wheel they can use, and instead pull in the latest wheel version here:

python/Dockerfile-linux.template

Lines 299 to 304 in f3c69c5

    
           pip3 install \ 
        
           	--disable-pip-version-check \ 
        
           	--no-cache-dir \ 
        
           	--no-compile \ 
        
           	{{ "setuptools==\( .setuptools.version )" | @sh }} \ 
        
           	wheel \

This hasn't historically caused problems, since wheel hasn't made many breaking changes. However, with wheel 0.46 dropping support for older setuptools, this is no longer the case.

The options presumably are:

Add a cap to the wheel requirement (eg wheel<0.46) so the installed wheel version remains compatible with the older setuptools. This could even just be hardcoded, since for Python 3.12+ the images don't install either setuptools or wheel, so there won't be any image combination where wheel 0.46+ needs to be installed.
Upgrade setuptools to a newer version too, so it's compatible with the newer wheel version. (But this would be breaking both now and any time setuptools makes changes, and diverge from the current strategy of matching ensurepip behaviour.)
Stop installing wheel in the Python 3.9-3.12 images entirely, to truly match the ensurepip behaviour. This will cause pip to use isolated build environments by default when building sdists, and inside those isolated environments newer setuptools will be used that doesn't depend on wheel. This presumably would be almost as breaking as (2) from a setuptools perspective, but not quite (since it doesn't affect the version of setuptools used at run-time, outside of those PEP 517 isolated build environments). However, it would be more breaking for people who use wheel directly.

Out of those, (1) seems the safest to me - and I suppose is in keeping with the ~"we don't upgrade tooling to latest, but generally prefer stability" approach?

cc @tianon @yosifkit (who are the Python docker image maintainers)

Just a comment on that one -> while possibly somewhat breaking, I think possibly fixing (i.e. hard-coding) setuptools version is likely the solution I would recommend (so that no future releases of setuptools will accidentally break things - like it happened few weeks ago). This is variant of 2. where you also fix / hardcode the version of setuptools.

One of the problems with the ensurepip strategy is that the old setuptools that are in the images have unpatched security vulnerabilities

https://www.cvedetails.com/vulnerability-list/vendor_id-10210/product_id-25692/Python-Setuptools.html

This is something we saw in security scans of our images, and this problem today finally made me to the approach that should make our images "more secure".

My 2c , of course not me to decide, because it should likely be automated somehow to bump the hard-coded version and might cause extra burden on mainrainers.

I think possibly fixing (i.e. hard-coding) setuptools version is likely the solution I would recommend

The versions of setuptools in the images is already fixed, eg:

python/versions.sh

Lines 158 to 188 in f3c69c5

    
           # Note: We don't extract the pip version here, since our policy is now to use the pip version 
        
           # that is installed during the Python build (which is the version bundled in ensurepip), and 
        
           # to not support overriding it. 
        
           # TODO remove setuptools version handling entirely once Python 3.11 is EOL 
        
           setuptoolsVersion="$(sed -nre 's/^_SETUPTOOLS_VERSION[[:space:]]*=[[:space:]]*"(.*?)".*/\1/p' <<<"$ensurepipVersions")" 
        
           case "$rcVersion" in 
        
           	3.9 | 3.10 | 3.11) 
        
           		if [ -z "$setuptoolsVersion" ]; then 
        
           			echo >&2 "error: $version: missing setuptools version" 
        
           			exit 1 
        
           		fi 
        
           		if ! wget -q -O /dev/null -o /dev/null --spider "https://pypi.org/pypi/setuptools/$setuptoolsVersion/json"; then 
        
           			echo >&2 "error: $version: setuptools version ($setuptoolsVersion) seems to be invalid?" 
        
           			exit 1 
        
           		fi 
        
           		# https://github.com/docker-library/python/issues/781 (TODO remove this if 3.10 and 3.11 embed a newer setuptools and this section no longer applies) 
        
           		if [ "$setuptoolsVersion" = '65.5.0' ]; then 
        
           			setuptoolsVersion='65.5.1' 
        
           		fi 
        
           		;; 
        
           	*) 
        
           		# https://github.com/python/cpython/issues/95299 -> https://github.com/python/cpython/commit/ece20dba120a1a4745721c49f8d7389d4b1ee2a7 
        
           		if [ -n "$setuptoolsVersion" ]; then 
        
           			echo >&2 "error: $version: unexpected setuptools: $setuptoolsVersion" 
        
           			exit 1 
        
           		fi 
        
           		;; 
        
           esac

python/versions.json

Lines 8 to 9 in f3c69c5

    
           "setuptools": { 
        
             "version": "65.5.1"

If you meant fixing the wheel version too, to match, then yeah I agree - and that's "option 1" above :-)

If you meant fixing the wheel version too, to match, then yeah I agree - and that's "option 1" above :-)

Not really - wheel version does not matter or it even can be not installed at all if setuptools > 70.1 . Setuptools 70.1 vendored-in the bdist_wheel command so essentially they don't care what version of wheel is installed. So my proposal is quite different from 1 :) .. Just bump to a fixed setuptools version that is > 70.1

Ah you're meaning option (2) described above then?

Yep. I updated the description to clarify it.

And this is what I essentially did in Airlfow: apache/airflow#48989

Python 3.9 -> both wheel and setuptools are old, but if you just upgrade wheel but do not upgrade setuptools, sdist installations will fail

Btw the different behaviour for Python 3.9 is temporary - and due to the 3.9.22 release being delayed by python/cpython#132287 - ie: the Python 3.9 images haven't been rebuilt yet since newer wheel came out. Once docker-library/official-images#18807 is merged the Python 3.9.22 images will have newer wheel too (and so also be broken like the 3.10-3.11 images).

I am using 3.11-slim, and 78.1.0 is not fixing it for me. I don't really touch the wheel version. Should I ?

I am using 3.11-slim, and 78.1.0 is not fixing it for me. I don't really touch the wheel version. Should I ?

It fixed it for us, so maybe you have some typo. Maybe posting the error and logging versions of setuptools and wheel you had just before the error migth help to explain things ?

Btw the different behaviour for Python 3.9 is temporary

Yeah. That's what I suspected :)

Once this is fixed, it might be worth adding an sdist pip install test - either in this existing test, or in a new one:
https://github.com/docker-library/official-images/blob/master/test/tests/python-pip-requests-ssl/container.py

Once this is fixed, it might be worth adding an sdist pip install test - either in this existing test, or in a new one:

Good idea. We are a bit more "lucky" with Airflow - because we build and use the image in our CI tests, so we don't have to do explicitly "unit" testing as we have a complete set of end-to-end tests with our image, but we implemented a set of unit tests anyway - testing if things are importable, that entrypoint works etc. etc. It saved us a few times.

We even test that the images can be extended following our examples (we use the runnable examples to build the documentation) - this is actually how we discovered that issue so quickly - because one of those tests failed.

It's also worth noting, that even if the wheel maintainer were to yank the 0.46.1 wheel release it wouldn't automatically fix the Python docker images, since they would still need a rebuild (and a manifest bump like docker-library/official-images#18803) to pick up the yank. So the fix needs to occur on the docker images side regardless.

For 3.11-slim

RUN pip install --no-cache-dir setuptools==78.1.0 wheel==0.46.1
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

Also, not really sure how to log the version that was working previously :(

252.1 Downloading jsonpointer-3.0.0-py2.py3-none-any.whl (7.6 kB)
252.2 Downloading mypy_extensions-1.0.0-py3-none-any.whl (4.7 kB)
252.3 Downloading pycparser-2.22-py3-none-any.whl (117 kB)
252.4    ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 117.6/117.6 kB 881.3 kB/s eta 0:00:00
252.5 Downloading webencodings-0.5.1-py2.py3-none-any.whl (11 kB)
252.6 Building wheels for collected packages: starkbank-ecdsa, jstyleson
252.6   Building wheel for starkbank-ecdsa (setup.py): started
252.7   Building wheel for starkbank-ecdsa (setup.py): finished with status 'error'
252.7   error: subprocess-exited-with-error
252.7   
252.7   × python setup.py bdist_wheel did not run successfully.
252.7   │ exit code: 1
252.7   ╰─> [6 lines of output]
252.7       usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
252.7          or: setup.py --help [cmd1 cmd2 ...]
252.7          or: setup.py --help-commands
252.7          or: setup.py cmd --help
252.7       
252.7       error: invalid command 'bdist_wheel'
252.7       [end of output]
252.7   
252.7   note: This error originates from a subprocess, and is likely not a problem with pip.
252.7   ERROR: Failed building wheel for starkbank-ecdsa
252.7   Running setup.py clean for starkbank-ecdsa
252.7   Building wheel for jstyleson (setup.py): started
252.8   Building wheel for jstyleson (setup.py): finished with status 'error'
252.8   error: subprocess-exited-with-error
252.8   
252.8   × python setup.py bdist_wheel did not run successfully.
252.8   │ exit code: 1
252.8   ╰─> [8 lines of output]
252.8       /usr/local/lib/python3.11/site-packages/setuptools/dist.py:771: UserWarning: Usage of dash-separated 'description-file' will not be supported in future versions. Please use the underscore name 'description_file' instead
252.8         warnings.warn(
252.8       usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
252.8          or: setup.py --help [cmd1 cmd2 ...]
252.8          or: setup.py --help-commands
252.8          or: setup.py cmd --help
252.8       
252.8       error: invalid command 'bdist_wheel'
252.8       [end of output]
252.8   
252.8   note: This error originates from a subprocess, and is likely not a problem with pip.
252.8   ERROR: Failed building wheel for jstyleson
252.8   Running setup.py clean for jstyleson
252.9 Failed to build starkbank-ecdsa jstyleson
252.9 ERROR: Could not build wheels for starkbank-ecdsa, jstyleson, which is required to install pyproject.toml-based projects
253.2 
253.2 [notice] A new release of pip is available: 24.0 -> 25.0.1
253.2 [notice] To update, run: pip install --upgrade pip
------
failed to solve: process "/bin/sh -c pip install --no-cache-dir -r requirements.txt" did not complete successfully: exit code: 1

Correcting myself. It actually worked. I just needed to run a no cache build.

Not really - wheel version does not matter or it even can be not installed at all if setuptools > 70.1 . Setuptools 70.1 vendored-in the bdist_wheel command so essentially they don't care what version of wheel is installed.

I wish this was the case, but since June 2024, setuptools does import and use wheel if it's installed! A bad decision IMHO.

@yosifkit

Wheel have yanked 0.46.1 due to issues with setuptools - can the python images be rebuilt urgently please? https://pypi.org/project/wheel/#history @yosifkit

Please amend

python/3.11/bookworm/Dockerfile

Line 99 in 70fc099

wheel \

to point version 0.45..1

We'll be making a PR to pin wheel shortly as that seems to be the least breaking change.

fix: pin docker-python base image to a working version (#803)

Navigation Menu

Repository navigation

Python images (At least bookwork-slim) relased today break installation of sdist packages #1023

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	pip3 install \
	--disable-pip-version-check \
	--no-cache-dir \
	--no-compile \
	{{ "setuptools==\( .setuptools.version )" \| @sh }} \
	wheel \

	# Note: We don't extract the pip version here, since our policy is now to use the pip version
	# that is installed during the Python build (which is the version bundled in ensurepip), and
	# to not support overriding it.

	# TODO remove setuptools version handling entirely once Python 3.11 is EOL
	setuptoolsVersion="$(sed -nre 's/^_SETUPTOOLS_VERSION[[:space:]]=[[:space:]]"(.?)"./\1/p' <<<"$ensurepipVersions")"
	case "$rcVersion" in
	3.9 \| 3.10 \| 3.11)
	if [ -z "$setuptoolsVersion" ]; then
	echo >&2 "error: $version: missing setuptools version"
	exit 1
	fi
	if ! wget -q -O /dev/null -o /dev/null --spider "https://pypi.org/pypi/setuptools/$setuptoolsVersion/json"; then
	echo >&2 "error: $version: setuptools version ($setuptoolsVersion) seems to be invalid?"
	exit 1
	fi

	# https://github.com/docker-library/python/issues/781 (TODO remove this if 3.10 and 3.11 embed a newer setuptools and this section no longer applies)
	if [ "$setuptoolsVersion" = '65.5.0' ]; then
	setuptoolsVersion='65.5.1'
	fi
	;;

	*)
	# https://github.com/python/cpython/issues/95299 -> https://github.com/python/cpython/commit/ece20dba120a1a4745721c49f8d7389d4b1ee2a7
	if [ -n "$setuptoolsVersion" ]; then
	echo >&2 "error: $version: unexpected setuptools: $setuptoolsVersion"
	exit 1
	fi
	;;
	esac

Python images (At least bookwork-slim) relased today break installation of sdist packages #1023

Description

Activity

edmorley commented on Apr 9, 2025

potiuk commented on Apr 9, 2025

edmorley commented on Apr 9, 2025

edmorley commented on Apr 9, 2025

potiuk commented on Apr 9, 2025

edmorley commented on Apr 9, 2025

potiuk commented on Apr 9, 2025

potiuk commented on Apr 9, 2025

edmorley commented on Apr 9, 2025

aysan0 commented on Apr 9, 2025

potiuk commented on Apr 9, 2025

potiuk commented on Apr 9, 2025

edmorley commented on Apr 9, 2025

potiuk commented on Apr 9, 2025

potiuk commented on Apr 9, 2025

edmorley commented on Apr 9, 2025

aysan0 commented on Apr 9, 2025

aysan0 commented on Apr 9, 2025

agronholm commented on Apr 9, 2025

niblORRRR commented on Apr 9, 2025

yosifkit commented on Apr 9, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions