Issues with the Stable ABI

@hauntsaninja

During PyCon, @hauntsaninja mentioned some surprising data: out of 400,000 PyPI projects, only about 300 are using the Stable ABI, and only two of those have a name anyone recognizes. (Shantanu could you link to those results and verify the numbers?) [UPDATE: There are better numbers further down in the thread.]

So I'd like to add the existence of the Stable ABI to the list of problems. It makes evolving certain APIs hard -- in particular, the ob_refcnt field is exposed through macros in the Stable ABI which means that for immortal objects in 3.12, we had to go through contortions to keep supporting wheels built with the 3.11 or older versions of Py_INCREF and friends.

Maybe the Stable ABI could at least be revised to not contain any macros that access object fields directly (replacing them with equivalent functions)?

Here's where I got that number: https://blog.trailofbits.com/2022/11/15/python-wheels-abi-abi3audit/

To get a list of auditable packages to feed into abi3audit, I used PyPI’s public BigQuery dataset to generate a list of every abi3-wheel-containing package downloaded from PyPI in the last 21 days:
[...]
From that query, I got 357 packages, which I’ve uploaded as a GitHub Gist.

Not sure how much the "downloaded in the last 21 days" affects things. From a single pass skim over the gist just now I recognised five packages.

Probably could twist abi3audit into telling you specifically how many packages would break from removing some macros.

@woodruffw

cc @woodruffw

Thanks for the ping! Yeah, I got those numbers from the BigQuery dataset for the last 21 days (which was a purely arbitrary limitation that should be lifted for a full analysis).

Because of how weak the controls around the abi3 tag on wheels and extensions are, it's possible that there are other Stable ABI-built wheels on PyPI that are just tagged incorrectly (i.e. they use the Stable ABI but their wheel filename doesn't indicate so). These arguably don't matter since pip won't treat them as Stable ABI compatible without the abi3 filename tag, but that's another limitation of the query that's worth considering.

Probably could twist abi3audit into telling you specifically how many packages would break from removing some macros.

Yeah, this should be doable -- you can use the big JSON blob in that blog post to see which symbols are actually in use, and I'm open to further improvements to abi3audit itself around that 🙂

Yeah, this should be doable -- you can use the big JSON blob in that blog post to see which symbols are actually in use, and I'm open to further improvements to abi3audit itself around that 🙂

How would that be able to tell you that a given binary is or isn't using Py_DECREF?

One of the big projects using the stable ABI is cryptography, here is their pypi download page. I think they would not be happy if they had to make wheels for all the versions of CPython they support on all the platforms they support. This is the flip-side of the faster (yearly) release cycle. If I recall correctly the stable ABI was one of the carrots offered to package maintainers worried about continually chasing new CPython versions.

Maybe the Stable ABI could at least be revised to not contain any macros that access object fields directly (replacing them with equivalent functions)?

It makes sense, I don't think the intent of the stable ABI was to freeze the ABI forever. Could there be an abi31 tag that would mark the next version of the stable C-API?

How would that be able to tell you that a given binary is or isn't using Py_DECREF?

It only works for symbols and not macros, unfortunately, so abi3audit can't reveal which extensions/wheels happen to use that macro (unless in doing so they leak an underlying symbol, as Py_DECREF does for _Py_DECREF). So in that particular case you can infer that Py_DECREF was used, but there's no guarantee in the general case.

It makes sense, I don't think the intent of the stable ABI was to freeze the ABI forever. Could there be an abi31 tag that would mark the next version of the stable C-API?

I'm biased as a package maintainer, but I'm personally a fan of something like this: the problem with the Stable ABI isn't the idea itself, but the restrictions that it's accumulated over the years.

A similar idea was floated on the forums a few months ago: https://discuss.python.org/t/lets-get-rid-of-the-stable-abi-but-keep-the-limited-api/18458

Stable ABI considered harmful
only two of those have a name anyone recognizes

Could we stick to facts, please?
As someone working on the stable ABI, I find the tone of this issue needlessly negative.

So I'd like to add the existence of the Stable ABI to the list of problems. It makes evolving certain APIs hard -- in particular, the ob_refcnt field is exposed through macros in the Stable ABI which means that for immortal objects in 3.12, we had to go through contortions to keep supporting wheels built with the 3.11 or older versions of Py_INCREF and friends.

AFAIK, ob_refcnt, ob_type and ob_size are the very last ones remaining. The rest of the structs are “blueprints” for initialization (not used at runtime), and the special case of Py_buffer.

This is the biggest issue in the stable ABI (which is 14 years old now). Your post suggests there are other big issues, are there? (I know about many of issues, of course, but nothing this big...)

Maybe the Stable ABI could at least be revised to not contain any macros that access object fields directly (replacing them with equivalent functions)?

Replacing them with functions would mean an abi4 -- essentially, telling everyone to recompile. It'd be a bunch of work, in CPython and packaging tools, but doable.

@encukou

In this repo we're trying to just do an inventory of problems, without immediately jumping to solutions (unless the problem is small and the solution is similarly contained).

With apologies for the provocative title, it does seem that the remaining struct fields exposed by the Stable ABI, even if they are just ob_refcnt, ob_type and ob_size are a big problem, that will require elbow grease, resolve, and probably a PEP for a solution.

Looking at the discourse thread Let's get rid of the stable ABI, but keep the limited API I actually see a spectrum of opinions on how to address the problem (and some confusion between Stable ABI and Limited API), but no additional problems, so let's just say that struct fields (including the all-important ob_refcnt) are the main problem that is exacerbated by the existence of the Stable ABI.

Another problem is that the Stable ABI doesn't provide the full functionality needed by some projects, e.g. PyObjC. (This may explain that relatively few projects use it.)

That thread also confirms the rough count of abi3-using projects (around 320).

Anyway, I suggest that we stick to this for the inventory for now, unless additional problems related to the Stable ABI are uncovered. (@encukou Feel free to add the other problems you know of, maybe we'll eventually discern a pattern leading to an innovative solution.)

Most other problems of the stable ABI (and limited API) are shared with the C API in general.

Let me give some thoughts, hoping that they're useful even if they are a bit off-topic.

The C API is vast, and my approach to improving it was to find a manageable subset, and then improve its quality and make it complete enough for general usage. Limited API (defined before my time, in PEP-384) was a reasonable starting point, and the stability guarantees make a good “carrot" for people to try it out (and find its shortcomings).

That thread also confirms the rough count of abi3-using projects (around 320).

Yup. It's not big. But it's now possible to use limited API in real-world project, although core dev involvement (cryptography) or lots of hacks (PySide) help. I haven't done too much marketing (e.g. adapt the tutorial), as I still see some incompleteness to fix. It's easier to iron out issues before the whole world uses it :)

As for “only two of those have a name anyone recognizes” -- that might not change much: for a popular leaf project, releasing yearly in the 2-month RC window is not that big of a problem. I expect Stable ABI to be much more useful for the long tail of smaller extensions.

Anyway, for the inventory, I see several issues identified here. I'm not sure how you want to organize them:

Exposed struct fields (ob_refcnt, ob_type, ob_size) force either contortions on us or recompilations on users
Existence of the Stable ABI (?)
Incompleteness of the Stable ABI
Low adoption of the Stable ABI

👋, one of the cryptography maintainers here. Happy to answer any questions about our setup, or what's important to us.

👋, one of the cryptography maintainers here. Happy to answer any questions about our setup, or what's important to us.

If we’re ever going to propose to deprecate the stable ABI I promise there will be a debate where you will be heard. But that’s not on the table now. We’re just trying to inventory problems with the C API here. So if anything about the C API bugs you, please create an issue for it! (Come to think of it, specific praise for the C API might also be useful.)

(Closed by mistake, clumsy fingers. :-)

I think that there would be far more stable ABI packages if it would be straightforward to compile C++ projects into stable ABI wheels. But it isn't. For one thing, the limited API has so far been too limited for it to be usable by various general C++ binding tools. That is changing now with 3.12, which has a number of relevant API calls exposed in the limited API.

Drawing a conclusion from the PyPI data point may be missing part of the big picture.

@vstinner

@vstinner Instead of killing _PyObject_New (called by PyObject_New), may I propose that PyObject_New is made part of the stable ABI, perhaps as an inline function? The alternative (PyType_GenericAlloc) is comparably complex and seems mainly useful for fancy types that are registered with the garbage collector. It also zero-initializes the object memory, which is wasteful when the object is big and the caller will directly initialize it as the next step.

What does it mean for an inline function to be in the Stable ABI?

Edit: I think I meant to say "Limited API". Which I interpret as: this will compile to something that is supported by the stable ABI. (As is happening right now, by calling _PyObject_New , which is part of the stable ABI).

Instead of killing _PyObject_New (called by PyObject_New)

What do you mean by "killing" it? _PyObject_New() is part of the stable ABI. There is no plan to remove it. It's just that it's declared as a "ABI-only" symbol, since it's not exposed at the API level. In the API, you use the #define PyObject_New(type, typeobj) ((type *)_PyObject_New(typeobj)) macro.

The PyObject_New() macro is part of the limited C API. It is declared in Include/objimpl.h, not in Include/cpython/objimpl.h which is excluded by the limited C API.

What does it mean for an inline function to be in the Stable ABI?

Static inline functions are part of the limited C API, but not part of the stable ABI: the code is copied into the built binary, it's not shipped by libpython. It's similar to macros.

Simplified implementation of Py_INCREF() of the limited C API (version 3.10 or newer):

static inline Py_ALWAYS_INLINE void Py_INCREF(PyObject *op)
{
    _Py_IncRef(op);
}

The compiler copies _Py_IncRef(op); code into your C extension, as the preprocessor would do with a macro, and then it's the _Py_IncRef() function which is called in practice in your binary. When Python loads your C extensions, the dynamic loader of your shared library (.so in Linux) checks if the current Python process has the _Py_IncRef symbol. If if does, you're good, you can call it :-)

In the past, C extensions were linked to libpython. That's no longer needed. The C extension just looks for symbols in the process which loads it. It just works :-) It solves issues depending on if Python is built with or without --enable-shared (if the python3 program itself is linked to libpython, or "contains" libpython: single binary without libpython).

The previous limited C API implementation of Py_INCREF() didn't call _Py_IncRef(), but was fully inlined: so a C extension would modify PyObject.ob_refcnt directly, and Py_DECREF() would call _Py_Dealloc(). Please don't look into the exact implementation of Py_INCREF() / Py_DECREF() in Python 3.13, it became very complicated ;-) It's good that it's now implemented as an opaque function in the limited C API version 3.12 and newer!

When Py_INCREF() and Py_DECREF() were fully initialized (were not implemented as opaque function calls), it was a headache to support debug build (Py_DEBUG) and the special --with-trace-refs (Py_TRACE_REFS). I just made the former ABI compatible in Python 3.13.

The PyObject_New() macro is part of the limited C API.

No, it is not. The limited API is defined by an explicit list, not just by what happens to be visible to the compiler. This is a very important feature: we avoid #34 for the limited subset of the API.

The limited API is defined in Misc/stable_abi.toml along with the stable ABI, and shows up in the list (and as notes) in the docs.

Static inline functions are part of the limited C API, but not part of the stable ABI: the code is copied into the built binary, it's not shipped by libpython. It's similar to macros.

Please avoid macros/inline functions in the limited API (except as optimizations). Such functions can only be used by C/C++, not even by ctypes. (FWIW, we even use ctypes to test if the symbols are available!)

@vstinner

@vstinner -- I seem to have misunderstood your message a few posts up. Weren't you proposing an abi4 that removes ABI-only entry points with no corresponding API? And then you specifically listed many symbols including _PyObject_New. My message is simply a request to not remove accessibility of PyObject_New via _PyObject_New from stable ABI modules. If that wasn't your plan, then please do ignore my message.

I don't need abi4. I'm not aware of anyone actively pushing for this. PyObject_New() is not going away.

No, it is not. The limited API is defined by an explicit list, not just by what happens to be visible to the compiler. This is a very important feature: we avoid #34 for the limited subset of the API.

The limited API is defined in Misc/stable_abi.toml along with the stable ABI, and shows up in the list (and as notes) in the docs.

That's very confusing for me. For me, the limited C API is what is accessible by #include <Python.h> when Py_LIMITED_API macro is defined.

If PyObject_New() is not part of the limited C API, it should be removed when the macro is defined, no? What is the point of having _PyObject_New() in the stable ABI if PyObject_New() is not part of the limited C API?

I don't need abi4. I'm not aware of anyone actively pushing for this. PyObject_New() is not going away.

Phew. Thank you for clarifying!

If PyObject_New() is not part of the limited C API, it should be removed when the macro is defined, no? What is the point of having _PyObject_New() in the stable ABI if PyObject_New() is not part of the limited C API?

Wait, what? That is exactly the thing I asked not to do. ;)

Please avoid macros/inline functions in the limited API (except as optimizations). Such functions can only be used by C/C++, not even by ctypes. (FWIW, we even use ctypes to test if the symbols are available!)

Well, Python has a long history, and C was the main target of the C API. Right, we can slowly convert macros and static inline functions to regular functions, but it's a slow incremental work since it can affect performance and may introduce surprising issues (like how macros can be badly abused in C, see PEP 674).

I got bitten by converting PyType_HasFeature() macro to an opaque function which made Python slower on macOS because Python wasn't built with LTO there (on Linux, there was no impact).

I wrote some articles on the topic:

Well, and https://peps.python.org/pep-0670/ obviously.

So far, what worked the best is to have an opaque function call in the limited C API, and override the name with a macro or static inline function in the non-limited C API.

That's very confusing for me. For me, the limited C API is what is accessible by #include <Python.h> when Py_LIMITED_API macro is defined.

That would be good to have.
But it can't always be that way -- for example, _Py_IncRef is not part of the limited API. So, the toml file is the source of truth.

If PyObject_New() is not part of the limited C API, it should be removed when the macro is defined, no? What is the point of having _PyObject_New() in the stable ABI if PyObject_New() is not part of the limited C API?

That, or it should be added to the limited API :)

Exposed struct fields (ob_refcnt, ob_type, ob_size) force either contortions on us or recompilations on users

Discussion for solving this: https://discuss.python.org/t/making-pyobject-opaque-in-the-limited-api/77206

let's just say that struct fields (including the all-important ob_refcnt) are the main problem that is exacerbated by the existence of the Stable ABI.

This is solved in PEP 803 (abi3t). The ABI limitations will be lifted when/if non-free-threaded builds are retired (after "phase III" of the free-threading rollout).
Or, of course, we can decide to retire abi3 without that. Or only provide features that need to change the PyObject structure on free-threaded builds, even if they're not related to threading.

I think this issue is done. There's no more planning to do; we now wait out the (long) deprecation period.

Another problem is that the Stable ABI doesn't provide the full functionality needed by some projects, e.g. PyObjC.

Such additions can be -- and are -- handled individually.
(PyObjC is on my radar, but since it's a single project with rather special needs, making it compatible with abi3 isn't really a priority.)

It would be nice to consider designing an abi4 to cleanup the dust

(“the dust” being the identified problem)

I've moved this to #82.

I think that's all the problems identified here, so I'll close. Feel free to reopen if you disagree.

Thanks for inventorizing & discussing!

Navigation Menu

Repository navigation

Issues with the Stable ABI #4

17 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issues with the Stable ABI #4

Description

Activity

hauntsaninja commented on May 9, 2023

hauntsaninja commented on May 9, 2023

woodruffw commented on May 9, 2023

gvanrossum commented on May 9, 2023

mattip commented on May 9, 2023

woodruffw commented on May 9, 2023

encukou commented on May 9, 2023

gvanrossum commented on May 10, 2023

encukou commented on May 10, 2023

alex commented on May 10, 2023

gvanrossum commented on May 10, 2023

gvanrossum commented on May 10, 2023

wjakob commented on May 10, 2023

17 remaining items

wjakob commented on Sep 4, 2023

gvanrossum commented on Sep 4, 2023

wjakob commented on Sep 4, 2023

vstinner commented on Sep 4, 2023

encukou commented on Sep 5, 2023

wjakob commented on Sep 5, 2023

vstinner commented on Sep 5, 2023

vstinner commented on Sep 5, 2023

wjakob commented on Sep 5, 2023

vstinner commented on Sep 5, 2023

encukou commented on Sep 7, 2023

encukou commented on Jan 15, 2025

encukou commented on Jun 4, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions