Repository navigation
[C API] Convert PyTuple_GET_ITEM() macro to a static inline function #85250
Description
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = <Date 2021-09-21.21:59:56.853>
created_at = <Date 2020-06-22.14:50:13.657>
labels = ['expert-C-API', '3.10']
title = '[C API] Convert PyTuple_GET_ITEM() macro to a static inline function'
updated_at = <Date 2021-09-21.21:59:56.853>
user = 'https://github.com/vstinner'bugs.python.org fields:
activity = <Date 2021-09-21.21:59:56.853>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2021-09-21.21:59:56.853>
closer = 'vstinner'
components = ['C API']
creation = <Date 2020-06-22.14:50:13.657>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 41078
keywords = ['patch']
message_count = 13.0
messages = ['372091', '372093', '372095', '372097', '372100', '372121', '372123', '372126', '372128', '372129', '372132', '372149', '402379']
nosy_count = 2.0
nosy_names = ['scoder', 'vstinner']
pr_nums = ['21056', '21057', '21058', '21059']
priority = 'normal'
resolution = 'rejected'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue41078'
versions = ['Python 3.10']
Activity
vstinner commentedon Jun 22, 2020
PyTuple_GET_ITEM() can be abused to access directly the PyTupleObject.ob_item member:
Giving a direct access to an array or PyObject* (PyObject**) is causing issues with other Python implementations, like PyPy, which don't use PyObject internally.
I propose to convert the PyTuple_GET_ITEM() and PyList_GET_ITEM() macros to static inline functions to disallow that.
vstinner commentedon Jun 22, 2020
New changeset 384621c by Victor Stinner in branch 'master':
bpo-41078: Rename pycore_tupleobject.h to pycore_tuple.h (GH-21056)
384621c
vstinner commentedon Jun 22, 2020
New changeset c45dbe9 by Victor Stinner in branch 'master':
bpo-41078: Add pycore_list.h internal header file (GH-21057)
c45dbe9
vstinner commentedon Jun 22, 2020
New changeset c96d00e by Victor Stinner in branch 'master':
bpo-41078: Fix bltinmodule.c with Py_TRACE_REFS (GH-21058)
c96d00e
vstinner commentedon Jun 22, 2020
The PR 21059 breaks Cython. I reported the issue to Cython upstream:
cython/cython#3701
numpy is also affected: code generated by Cython (numpy/random/_mt19937.c) contains the issue.
vstinner commentedon Jun 22, 2020
PySequence_Fast_ITEMS() must also be deprecated since it also gives a direct access to PyTupleObject.ob_item and PyListObject.ob_item members (PyObject**). PySequence_Fast_GET_ITEM() should be used instead.
vstinner commentedon Jun 22, 2020
PySequence_Fast_ITEMS() is used in numpy at 7 lines:
numpy/core/src/common/ufunc_override.c:118: *out_objs = PySequence_Fast_ITEMS(seq);
---
=> PyObject** is part of the numpy C API (it's even a pointer to a PyObject** array)!
numpy/core/src/multiarray/arrayfunction_override.c:80: PyObject **items = PySequence_Fast_ITEMS(relevant_args);
---
=> here PySequence_Fast_GET_ITEM() can be used instead.
numpy/core/src/multiarray/arrayfunction_override.c:167: PyObject **items = PySequence_Fast_ITEMS(types);
numpy/core/src/multiarray/common.c:434: objects = PySequence_Fast_ITEMS(seq);
numpy/core/src/multiarray/iterators.c:1368: ret = multiiter_new_impl(n, PySequence_Fast_ITEMS(fast_seq));
numpy/core/src/multiarray/methods.c:1015: in_objs = PySequence_Fast_ITEMS(fast);
numpy/core/src/umath/override.c:38: arg_objs = PySequence_Fast_ITEMS(args);
scoder commentedon Jun 22, 2020
I'm wondering – if the intention is to help other implementations, then why do we need to break extensions in *CPython* for that? In CPython, this usage seems perfectly valid with the current data structure. If that ever changes, we can still make this change as well, but do you really see the internal layout of tuples change at some point? (Lists are a different case, I'd say, but tuples?)
vstinner commentedon Jun 22, 2020
The long rationale is explained in PEP-620:
https://www.python.org/dev/peps/pep-0620/
PyTupleObject and PyListObject structures must become opaque. Also, later, these structures may change to be more efficient.
scoder commentedon Jun 22, 2020
Tuples? Really?
Ok, quoting PEP-620:
I honestly think the reason for that might simply be that there's not so much to improve for tuples.
I certainly understand that that is a problem, especially if "PyObject" may change in the future. And this is essentially what the current "PyTuple_GET_ITEM()" macro does in a binary module.
Should we also turn "_PyTuple_ITEMS()" into a public inline function then to make up for the loss of the "&PyTuple_GET_ITEM(t, 0)" pattern? It would make it explicit what is intended. I think that should be our main goal in the CPython side.
vstinner commentedon Jun 22, 2020
I suggest you to discuss the PEP-620 on python-dev, rather than on the bug tracker. I just posted it there today.
This issue tracks the implemetation of one part of the PEP-620.
vstinner commentedon Jun 23, 2020
See "(PEP-620) C API for efficient loop iterating on a sequence of PyObject** or other C types" thread on python-dev:
https://mail.python.org/archives/list/python-dev@python.org/thread/632CV42376SWVYAZTHG4ROOV2HRHOVZ7/
vstinner commentedon Sep 21, 2021
The PEP-620 is still a draft. I prefer to work on other C API changes before coming back to this change which may break many C extensions. I failed finding time to design an API to expose a PyObject** array "view" with a function to release/destroy the view.