setuptools 50 breaks pip installation on Debian/Ubuntu

On newly created Ubuntu 16.04 AWS instances, we started seeing that python3.6 -m pip no longer works after get-pip.py (version 19.3.1). get-pip puts pip in /usr/lib/python3.6/site-packages/pip-20.2.2.dist-info, but sys.path for our 3.6 executable is ['', '/usr/lib/python36.zip', '/usr/lib/python3.6', '/usr/lib/python3.6/lib-dynload', '/usr/local/lib/python3.6/dist-packages', '/usr/lib/python3/dist-packages'].

On a newly created box I was able to confirm that using SETUPTOOLS_USE_DISTUTILS=stdlib fixes the problem:

    1  python3.6 /get_pip.py 
    2  python3.6 -m pip  # works
    3  python3.6 -m pip install -U setuptools  # updates from 41.0.1 to 50.0.0
    4  python3.6 -m pip  # still works
    5  python3.6 /get_pip.py 
    6  python3.6 -m pip  # can't find pip any more
    7  SETUPTOOLS_USE_DISTUTILS=stdlib python3.6 /get_pip.py 
    8  python3.6 -m pip  # works again

Sorry this isn't a completely self-contained repro case for the bug; hopefully this is enough information to figure out what's going on.

What you're seeing is pypa/setuptools#2232 which has been re-introduced, mentioned pypa/setuptools@04e3df2

In short, packaged Debian/ubuntu Python installs things into dist-packages. site-packages is left alone in-case you like to build your own upstream python and install it. Debuntu overrides the distutils it ships to redirect installs to dist-packages.
It gets blurry when you install pip from upstream. But it would call setuptools, which would use the patched distutils so things worked.

Now setuptools vendors distutils this and always installs things into site-packages, which isn't on the packaged python interpreter path, so it can't find modules.

Another consequence is the binary location, which moved from /usr/local/bin/ to /usr/bin. That's kind of a moot point though, because your binaries won't be able to find their libraries anyway.

Another consequence of this is broken development installations on Ubuntu (pip install -e). pth files are written in site-packages instead of dist-packages and thus do not work.

[ci] Force setuptools not to use version 50.0.0 for now

[ci] Force setuptools version < 50.0.0

Another consequence of this is broken development installations on Ubuntu (pip install -e). pth files are written in site-packages instead of dist-packages and thus do not work.

I think it's all related; if you hit the inbuilt distutils you're not getting the overrides that Debian has set. So pip install -e is the most straight forward way to do this.

It is confusing however; if you just do pip install . then you'll build and install a wheel, which won't exhibit this problem. But if you try pip install --no-binary :all: . in your source directory it will be using distutils and and install it to site-packages, not dist-packages. You could also run setup.py directly which will exhibit the problem too.

Fix deployed lambda functions not working due to setuptools monkeypat…

Freezing setuptools due to its incorrect operation

Quoting the changelog entry, for the relevant change:

pypa/setuptools#2232: Once again, Setuptools overrides the stdlib distutils on import. For environments or invocations where this behavior is undesirable, users are provided with a temporary escape hatch. If the environment variable SETUPTOOLS_USE_DISTUTILS is set to stdlib, Setuptools will fall back to the legacy behavior. Use of this escape hatch is discouraged, but it is provided to ease the transition while proper fixes for edge cases can be addressed.

Please see pypa/setuptools#2232 for the relevant discussion on this.

Note that this affects other distros than just Ubuntu/Debian. It seems that distutils is the only place where you can define install schemes (and even if that's not the case, people assume that). So, it's quite widely patched.
See also: https://discuss.python.org/t/how-can-we-keep-pip-from-updating-uninstalling-system-libraries-in-usr-lib-python/5059

@doko42

Looks like @doko42 is a maintainer for distutils (and setuptools) on Debian: can you help with the above?

I am not a Debian maintainer, but looking at their patch, their needs are the same as in Fedora (and PyPy, too): that is, defining and selecting install schemes.
On Discourse, there is an idea of moving this data/logic to sysconfig. Would this work as a better approach (for option 4)?

Unfortunately, on today's systems that customize distutils in this way, the stdlib's distutils is the only source of this information. I don't see a way to change this until Python 3.10.

@kitterma

I believe @kitterma is also involved in Debian's Python. :)

FYI, this doesn't only affect Debian. The AWS lambdas running Python were also broken.

https://stackoverflow.com/questions/63688774/some-aws-lambda-functions-stopped-working-with-no-module-named-setuptools-dist

* Back out the overriding of distutils for all Setuptools users until Debian can devise a better approach.

If this means "undo the changes in 50.0 and release 51.0 which behaves like 49.latest worked" then I think that might be the preference of the vast majority of the community that uses setuptools.

In this thread you can see just a snapshot of the work being generated right now, just in the open-source community on GitHub, and it's jarring. People are pinning setuptools to versions before 50, despite the fact it's not recommended to pin setuptools in requirements.txt-style installs, and PyPA's own documentation recommends the recipe python -m pip install --upgrade pip setuptools wheel when installing packages via pip

Add this to all of the community who have adopted Docker containers and there will be weeks of people discovering the various open issues on setuptools right now and hoping to find a fix but ultimately not being super knowledgeable about distutils or setuptools or how they interact with distro X because, honestly, nobody outside of PyPA and a small circle in the Python community Venn diagram have any idea what all this is about.

I would add an argument about how now would be the perfect time for CVEs in pre-setuptools 50.0 to drop but I'm only security conscious enough to know it would be bad so I'll leave the speculation for someone more intelligent than I am there.

It may be your least preferred option but from the point of view of myself, and I reckon much of the Python-using community, it's the only option you should be considering given the widespread effects of this issue.

/2c, and I meant every word from a place of good faith. "don't break userland" should apply to more things than the Linux kernel

I'm convinced the behavior needs to be suppressed for now. I was hoping we could work through the impacts and push through this time (based on the fallout from the Setuptools 48 release being much less severe).

Make stdlib distutils the default again. Stop the burning. Ref #2350 …

Setuptools 50.1.0 changes contains pypa/setuptools#2376, which changes the default to be more conservative.

Updated bazel/test_utils.sh to specify a compatible `setuptools`.

Closing as a duplicate of #2.

Fix an issue with ubuntu, distutils, and pip

Navigation Menu

Repository navigation

setuptools 50 breaks pip installation on Debian/Ubuntu #17

3 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

setuptools 50 breaks pip installation on Debian/Ubuntu #17

Description

Activity

ianw commented on Aug 31, 2020

vmarkovtsev commented on Aug 31, 2020

ianw commented on Aug 31, 2020

pradyunsg commented on Sep 1, 2020

encukou commented on Sep 2, 2020

3 remaining items

hugovk commented on Sep 2, 2020

encukou commented on Sep 2, 2020

pradyunsg commented on Sep 2, 2020

canassa commented on Sep 2, 2020

unchris commented on Sep 2, 2020

jaraco commented on Sep 2, 2020

pradyunsg commented on Sep 3, 2020

jaraco commented on Sep 13, 2020

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions