You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Namespace packags and setup.py develop do not place nicely. #12
setup.py develop uses an egg-link (cross platform symlink basically) in order to make it work. However this fails miserably when the packages you're trying to develop have a setuptools style namespace package.
fwiw, I've only ever cared about setup.py develop for writing Pylons and Pyramid applications, which rely on entry points because they're run with Paste. If I could just launch them from the project directory like I do everything else then this would be less of a problem.
Entry points themselves are a glorious mess, too, but we don't have anything better.
We do actually have a bettter option: *.pth files in virtual environments are a much better solution to this than dependency links, since they can handling multi-portion namespace packages properly and transparently.
*.pth files are only evil in global installations (since they implicitly alter sys.path for everything run in that environment with site module processing enabled)
After reading through various stuff on this, I'm still unclear as to what needs to happen for this to be fixed? Does setuptools need to be modified to write a .nspth instead of an .egg-link when running setup.py develop for namespace packages? What are the ramifications of not using .egg-links?
I'm asking these questions because I'm willing to do the work, but I want to make sure I'm carving the right pumpkin.
@jonparrott I think you're unclear because we're all unclear ;)
Something I think may fix this is if setup.py develop changed to work as follows:
if it sees a venv is active, drop a "-develop.pth" file into the site-packages of the active virtualenv that references the project directory, rather than doing what it does now
if it doesn't see an active venv, it keeps doing the same thing it does today
We may be trading one problem for another (e.g. anyone using a "shared virtualenv" model for multiple projects that they want to keep separate isn't going to like that change), but we would gain a fair bit in simplicity and predictability (since we'd be using a core import system feature, rather than something specific to setuptools)
Related to the "shared virtualenv" problem, I also just realised that .pth files could be used to implement virtualenv inheritance: putting a .pth in venv B that references the site-packages of venv A will give you a situation where venv B sees all changes made in venv A, but anything new you install (including via setup.py develop changed as suggested above) will go solely into venv B.
Starting off with the new behaviour behind a flag sounds like an excellent idea, as that enables two things:
if the default doesn't work, we can suggest "--use-pth" as a possible fix
folks for whom the default currently works can readily test the new behaviour to see if it breaks anything
That would suggest it also makes sense to add a "--use-egg-link" option to start setting up for a possible future change to the default behaviour (assuming "--use-pth" actually does turn out to fix the current problems).
We now have standards-based namespace packages and develop install. I hope they work well together*, but if not this would be a normal bug to report to the build backends.
A setuptools ticket exists and was closed. I think this ticket can be closed @woodruffw
*(not a fan myself of PEP 420 that makes every data directory an implicit namespace package 😦)
Activity
jaraco commentedon Nov 3, 2013
See pip #3 for a related discussion.
eevee commentedon Nov 3, 2013
fwiw, I've only ever cared about
setup.py developfor writing Pylons and Pyramid applications, which rely on entry points because they're run with Paste. If I could just launch them from the project directory like I do everything else then this would be less of a problem.Entry points themselves are a glorious mess, too, but we don't have anything better.
ncoghlan commentedon Nov 3, 2013
We do actually have a bettter option: *.pth files in virtual environments are a much better solution to this than dependency links, since they can handling multi-portion namespace packages properly and transparently.
*.pth files are only evil in global installations (since they implicitly alter sys.path for everything run in that environment with site module processing enabled)
theacodes commentedon Sep 15, 2016
After reading through various stuff on this, I'm still unclear as to what needs to happen for this to be fixed? Does
setuptoolsneed to be modified to write a.nspthinstead of an.egg-linkwhen runningsetup.py developfor namespace packages? What are the ramifications of not using.egg-links?I'm asking these questions because I'm willing to do the work, but I want to make sure I'm carving the right pumpkin.
ncoghlan commentedon Sep 15, 2016
@jonparrott I think you're unclear because we're all unclear ;)
Something I think may fix this is if
setup.py developchanged to work as follows:We may be trading one problem for another (e.g. anyone using a "shared virtualenv" model for multiple projects that they want to keep separate isn't going to like that change), but we would gain a fair bit in simplicity and predictability (since we'd be using a core import system feature, rather than something specific to setuptools)
ncoghlan commentedon Sep 15, 2016
Related to the "shared virtualenv" problem, I also just realised that .pth files could be used to implement virtualenv inheritance: putting a
.pthin venv B that references the site-packages of venv A will give you a situation where venv B sees all changes made in venv A, but anything new you install (including viasetup.py developchanged as suggested above) will go solely into venv B.theacodes commentedon Sep 15, 2016
Oh, that makes me feel better.
That sounds reasonable. Could we be even more reasonable and put the new behavior behind a flag (e.g.
setup.py develop --use-pth)?That sounds interesting, but not a use case I wanna focus on for the moment.
ncoghlan commentedon Sep 15, 2016
Starting off with the new behaviour behind a flag sounds like an excellent idea, as that enables two things:
That would suggest it also makes sense to add a "--use-egg-link" option to start setting up for a possible future change to the default behaviour (assuming "--use-pth" actually does turn out to fix the current problems).
theacodes commentedon Sep 15, 2016
SGTM, I'll start trying to make this materialize.
FYI, I'm familiar with pip's codebase but not setuptools, so the first draft might be slightly more hacky than expected. :)
theacodes commentedon Sep 15, 2016
(rough) PR sent: pypa/setuptools#789
merwok commentedon Feb 11, 2025
We now have standards-based namespace packages and develop install. I hope they work well together*, but if not this would be a normal bug to report to the build backends.
A setuptools ticket exists and was closed. I think this ticket can be closed @woodruffw
*(not a fan myself of PEP 420 that makes every data directory an implicit namespace package 😦)