gh-157242: Fix PyBytesWriter_Resize() on MemoryError

[vstinner]

If PyBytesWriter_Resize() fails, leave the writer unchanged.

Add a new internal _PyBytes_ResizeKeepOnError() function similar to _PyBytes_Resize() but leaves the bytes object unchanged on error.

• Issue: bytearray.resize() must not clear the object on error (memory allocation failure) #157242

[cmaloney]

It would be nice if the "guaranteed no global" case was also used for the PyBytes_FromStringAndSize(NULL, size); call. That the first parameter there must be NULL to avoid a global is unintuitive.

[serhiy-storchaka]

No need to add a new parameter, it saves nothing. _PyBytes_Resize can be implemented via _PyBytes_ResizeKeepOnError.

I wonder if we can simply change the behavior of _PyBytes_Resize.

[vstinner]

Please review the updated PR. I addressed reviews.

@maurycy found a fix for the memmove() code path which worried me. I applied his suggestion and added a test.

@serhiy-storchaka:

No need to add a new parameter, it saves nothing. _PyBytes_Resize can be implemented via _PyBytes_ResizeKeepOnError. I wonder if we can simply change the behavior of _PyBytes_Resize.

Thanks for the advice. I reworked _PyBytes_Resize(): for the in-place resize code path, no longer call _Py_ForgetReference() and _PyReftracerTrack() before PyObject_Realloc(). Only call them on success. With this change, I was able to easy implement _PyBytes_Resize() with _PyBytes_ResizeKeepOnError().

[vstinner]

It would be nice if the "guaranteed no global" case was also used for the PyBytes_FromStringAndSize(NULL, size); call. That the first parameter there must be NULL to avoid a global is unintuitive.

Maybe PyBytes_FromStringAndSize() documentation should be elaborated to explain that PyBytes_FromStringAndSize(NULL, size) can be mutated, whereas PyBytes_FromStringAndSize(str, size) must not be mutated?

Note: PR gh-156996 does fix PyBytes_FromStringAndSize() usage in bytearray. I don't try to replace this fix.

[vstinner]

bytearray.resize() and bytearray.take_bytes() have been fixed to no longer use a singleton: I merged main in my PR to get the PR gh-156996 fix.

[vstinner]

@maurycy: I modified resize() and take_bytes() to avoid memmove() usage if we would be unable to revert the bytesarray to its previous state on MemoryError. Does it look correct to you?

I also added more tests injecting MemoryError.

[maurycy]

@vstinner: Thank you. I will take a look more carefully later today.

[maurycy]

@vstinner Two more findings. That's such a nice PR!

Good news: the PR fixes this regression:

2026-09-10T17:44:57.890346000+0200 maurycy@gimel /Users/maurycy/work/cpython (main 9bd670c?) % ./python.exe 
Python 3.16.0a0 (heads/main:9bd670cba21, Sep 10 2026, 17:37:43) [Clang 21.0.0 (clang-2100.1.1.101)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import _testcapi
... import sys
... 
... ba = bytearray(b"abc123")
... try:
...     _testcapi.set_nomemory(0)
...     del ba[1:-1]
... except MemoryError: pass
... finally: _testcapi.remove_mem_hooks()
... 
>>> len(ba)
Traceback (most recent call last):
  File "<python-input-1>", line 1, in <module>
    len(ba)
    ~~~^^^^
SystemError: <built-in function len> returned NULL without setting an exception
>>> bytes(ba)
Traceback (most recent call last):
  File "<python-input-2>", line 1, in <module>
    bytes(ba)
    ~~~~~^^^^
ValueError: size must be >= 0
>>> 
2026-09-10T17:45:16.126365000+0200 maurycy@gimel /Users/maurycy/work/cpython-pr157243 (pr157243 f51cba5?) % ./python.exe 
Python 3.16.0a0 (heads/pr157243:f51cba57d3e, Sep 10 2026, 14:31:04) [Clang 21.0.0 (clang-2100.1.1.101)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import _testcapi
... import sys
... 
... ba = bytearray(b"abc123")
... try:
...     _testcapi.set_nomemory(0)
...     del ba[1:-1]
... except MemoryError: pass
... finally: _testcapi.remove_mem_hooks()
... 
>>> len(ba)
2
>>> bytes(ba)
b'a3'
>>> 

Perhaps we cover this with test, too?

Bad news:

2026-09-10T17:47:19.394053000+0200 maurycy@gimel /Users/maurycy/work/cpython-pr157243 (pr157243 f51cba5?) % ./python.exe
Python 3.16.0a0 (heads/pr157243:f51cba57d3e, Sep 10 2026, 14:31:04) [Clang 21.0.0 (clang-2100.1.1.101)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import _testcapi
... import sys
... 
... ba = bytearray(b"abc123456")
... ba += b"7"
... 
... try:
...     _testcapi.set_nomemory(0)
...     del ba[::2]
... except MemoryError: pass
... finally: _testcapi.remove_mem_hooks()
... 
... ba
... 
bytearray(b'b135734567')
>>> 
2026-09-10T17:48:34.225959000+0200 maurycy@gimel /Users/maurycy/work/cpython (main 9bd670c?) % ./python.exe 
Python 3.16.0a0 (heads/main:9bd670cba21, Sep 10 2026, 17:37:43) [Clang 21.0.0 (clang-2100.1.1.101)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import _testcapi
... import sys
... 
... ba = bytearray(b"abc123456")
... ba += b"7"
... 
... try:
...     _testcapi.set_nomemory(0)
...     del ba[::2]
... except MemoryError: pass
... finally: _testcapi.remove_mem_hooks()
... 
... ba
... 
bytearray(b'')

Unfortunately, I don't have bandwitch to investigate what's the exact fix here

[vstinner]

The PR became quite big, so I modified it to focus strictly on _PyBytes_ResizeKeepOnError(). Once this PR will be merged, I will open a second issue to fix bytearray methods.

[vstinner]

Ok, I merged the first part. I created the follow-up PR for bytearray: PR gh-157340.

[bedevere-bot]

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 Arch Linux TraceRefs 3.x (tierless) has failed when building commit f5dd52d.

What do you need to do:

1. Don't panic.
2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
3. Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/484/builds/7361) and take a look at the build logs.
4. Check if the failure is related to this commit (f5dd52d) or if it is a false positive.
5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#/builders/484/builds/7361

Failed tests:

• test_capi

Summary of the results of the build (if available):

==

Click to see traceback logs

Note: switching to 'f5dd52df16e1b3f3f8cc2762f1de783a41d69f4f'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD is now at f5dd52df16e gh-157242: Fix PyBytesWriter_Resize() on MemoryError (#157243)
Switched to and reset branch 'main'

make: *** [Makefile:2481: buildbottest] Error 2

[vstinner]

The buildbot AMD64 Arch Linux TraceRefs 3.x (tierless) has failed when building commit f5dd52d.

Ah, Py_TRACE_REFS builds are special, they cannot test set_nomemory(). I wrote PR gh-157350 to fix test_capi on Py_TRACE_REFS build.

[miss-islington-app]

Thanks @vstinner for the PR 🌮🎉.. I'm working now to backport this PR to: 3.14.
🐍🍒⛏🤖

[miss-islington-app]

Thanks @vstinner for the PR 🌮🎉.. I'm working now to backport this PR to: 3.15.
🐍🍒⛏🤖 I'm not a witch! I'm not a witch!

[miss-islington-app]

Sorry, @vstinner, I could not cleanly backport this to 3.14 due to a conflict.

Please backport manually with cherry_picker, see the devguide for more information.

cherry_picker f5dd52df16e1b3f3f8cc2762f1de783a41d69f4f 3.14

[miss-islington-app]

Sorry, @vstinner, I could not cleanly backport this to 3.15 due to a conflict.

Please backport manually with cherry_picker, see the devguide for more information.

cherry_picker f5dd52df16e1b3f3f8cc2762f1de783a41d69f4f 3.15

[bedevere-app]

GH-159260 is a backport of this pull request to the 3.15 branch.