Repository navigation
ctypes is too slow to convert a Python list to a C array #72113
Description
| BPO | 27926 |
|---|---|
| Nosy | @eryksun |
| Files |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = None
created_at = <Date 2016-09-01.09:31:15.237>
labels = ['ctypes', '3.8', '3.9', '3.10', 'performance']
title = 'ctypes is too slow to convert a Python list to a C array'
updated_at = <Date 2021-03-05.19:07:48.471>
user = 'https://bugs.python.org/TomCornebize'bugs.python.org fields:
activity = <Date 2021-03-05.19:07:48.471>
actor = 'eryksun'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['ctypes']
creation = <Date 2016-09-01.09:31:15.237>
creator = 'Tom Cornebize'
dependencies = []
files = ['44320']
hgrepos = []
issue_num = 27926
keywords = []
message_count = 3.0
messages = ['274111', '274138', '274148']
nosy_count = 2.0
nosy_names = ['eryksun', 'Tom Cornebize']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'performance'
url = 'https://bugs.python.org/issue27926'
versions = ['Python 3.8', 'Python 3.9', 'Python 3.10']
Activity
TomCornebize commentedon Sep 1, 2016
It is much faster to construct a Python array from the list and then cast this array, rather than using the "standard" constructor. See attached file to compare the performances.
This issue was previously asked on Stackoverflow: http://stackoverflow.com/questions/39225263/why-is-ctypes-so-slow-to-convert-a-python-list-to-a-c-array/
eryksun commentedon Sep 1, 2016
This is a consequence of several factors. It starts with the __init__ method of ctypes.Array, Array_init. This function doesn't hard-code calling the base sq_ass_item slot function, Array_ass_item. If it did, it wouldn't be nearly as slow. Instead it calls the abstract function PySequence_SetItem. Doing it this way accommodates an array subclass that overrides __setitem__.
What I'd like to do here is check whether the sq_ass_item slot is defined as Array_ass_item, and if so call it directly instead of PySequence_SetItem. But it turns out that it's not set as Array_ass_item even if the subclass doesn't override __setitem__, and more than anything this is the real culprit for the relative slowness of Array_init.
If a built-in type such as ctypes.Array defines both mp_ass_subscript and sq_ass_item, then the __setitem__ wrapper_descriptor wraps the more generic mp_ass_subscript slot function. Then for a subclass, update_one_slot in Objects/typeobject.c plays it safe when updating the sq_ass_item slot. It sees that the inherited __setitem__ descriptor doesn't call wrap_sq_setitem, so it defines the slot in the subclass to use the generic function slot_sq_ass_item.
This generic slot function goes the long way around to look up and bind the __setitem__ method and convert the Py_ssize_t index to a Python integer, to call the wrapper that calls the mp_ass_subscript slot. To add insult to injury, the implementation of this slot for a ctypes Array, Array_ass_subscript, has to convert back to a Py_ssize_t integer via PyNumber_AsSsize_t.
I don't know if this can be resolved while preserving the generic design of the initializer. As is, calling PySequence_SetItem in a tight loop is ridiculously slow. I experimented with calling Array_ass_item directly. With this change it's as fast as assigning to a slice of the whole array. Actually with a list it's a bit slower because *t has to be copied to a tuple. But it takes about the same amount of time as assigning to a slice when t is already a tuple, such as tuple(range(1000000)).
I doubt any amount of tweaking will make ctypes as fast as an array.array. ctypes has a generic design to accommodate simple C data, pointers, and aggregate arrays, structs, and unions. This comes with some cost to performance. However, you can and should make use of the buffer protocol to use arrays from the array module or numpy where performance is critical. It's trivial to create a ctypes array from an object that supports the buffer protocol. For example:
There's no need to use the array.array's buffer_info() or ctypes.cast(). The from_buffer() method creates an array that shares the buffer of the source object, so it's relatively fast. It's also returning a sized array instead of a lengthless pointer (though it is possible to cast to an array pointer and immediately dereference the array).
TomCornebize commentedon Sep 1, 2016
Thank you for these explanations.
I understand that we get a generic function to the cost of performances.
However, I think we should at least tell in the documentation that the constructor (ctypes.c_uint32 * len(t))(*t) is slow and that we can do much faster in some specific cases (e.g. an array of integers).
It would be even better to have some specific method(s) to do this in ctypes, instead of having to rely on an array.array just to build a ctypes array from a list. I am not familiar with CPython code, so I do not know if it would be easily feasible.
[-]ctypes is too slow to convert a Python list to a C array[/-][+][doc] ctypes is too slow to convert a Python list to a C array[/+][-][doc] ctypes is too slow to convert a Python list to a C array[/-][+]ctypes is too slow to convert a Python list to a C array[/+]