Repository navigation
_remote_debugging: RemoteUnwinder caches the interpreters list head in __init__ as the main interpreter #159262
Description
Bug report
An existing RemoteUnwinder misses interpreters created after its constructor runs. If the saved interpreter closes, later samples can read freed memory. The default mode also selects the main thread of the newest interpreter and can omit the main interpreter.
Found while reviewing #158900, #158971 and working on #159259.
Reproduction
A new interpreter is missing from an existing unwinder
The existing unwinder returns only interpreter 0. A new unwinder sees both interpreters:
import os
from concurrent import interpreters
from _remote_debugging import RemoteUnwinder
unwinder = RemoteUnwinder(os.getpid(), all_threads=True)
interp = interpreters.create()
try:
print([i.interpreter_id for i in unwinder.get_stack_trace()])
fresh = RemoteUnwinder(os.getpid(), all_threads=True)
print([i.interpreter_id for i in fresh.get_stack_trace()])
finally:
interp.close()2026-10-12T22:43:31.492120000+0200 maurycy@gimel /Users/maurycy/work/cpython (main 5e049c3) % ./python.exe /tmp/aaa.py
[0]
[1, 0]The saved interpreter closes
maurycy@gimel cpython (main 9028df3) % cat /tmp/repro_dangling.py
import os
from concurrent import interpreters
from _remote_debugging import RemoteUnwinder
interp = interpreters.create()
unwinder = RemoteUnwinder(os.getpid(), all_threads=True)
interp.close()
print([i.interpreter_id for i in unwinder.get_stack_trace()])
maurycy@gimel cpython (main 9028df3) % ./python.exe /tmp/repro_dangling.py
Traceback (most recent call last):
File "/tmp/repro_dangling.py", line 8, in <module>
print([i.interpreter_id for i in unwinder.get_stack_trace()])
~~~~~~~~~~~~~~~~~~~~~~~~^^
RuntimeError: mach_vm_read_overwrite failed for PID 32948 at address 0xdddddddddddddddd (size 888): kern_return_t 10xdd is the debug allocator's dead byte (with PYTHONMALLOC=debug)
The default mode omits the main interpreter
from concurrent import interpreters
interp = interpreters.create()
try:
interp.exec("""
import os
from _remote_debugging import RemoteUnwinder
for all_threads in (False, True):
unwinder = RemoteUnwinder(os.getpid(), all_threads=all_threads)
unwinder.get_stack_trace()
print(all_threads, [i.interpreter_id for i in unwinder.get_stack_trace()])
""")
finally:
interp.close()2026-10-12T22:54:47.076383000+0200 maurycy@gimel /Users/maurycy/work/cpython (main 5e049c3) % ./python.exe /tmp/xoxo.py
False [1]
True [1, 0]Additional cases: an idle interpreter and a freed thread state
Idle subinterpreter (main interpreter has two threads):
2026-10-12T22:54:49.923205000+0200 maurycy@gimel /Users/maurycy/work/cpython (main 5e049c3) % cat /tmp/repro_idle_default.py
import os, threading, time
from concurrent import interpreters
from _remote_debugging import RemoteUnwinder
ev = threading.Event()
t = threading.Thread(target=ev.wait); t.start()
interp = interpreters.create()
for kwargs in ({}, {"all_threads": True}):
u = RemoteUnwinder(os.getpid(), **kwargs); u.get_stack_trace()
print(kwargs, [(i.interpreter_id, len(i.threads)) for i in u.get_stack_trace()])
ev.set(); t.join(); interp.close()
2026-10-12T22:55:14.833746000+0200 maurycy@gimel /Users/maurycy/work/cpython (main 5e049c3) % ./python.exe /tmp/repro_idle_default.py
{} [(1, 0), (0, 2)]
{'all_threads': True} [(1, 0), (0, 2)]The subinterpreter goes idle after __init__, releasing the saved thread state. The sample still omits the main interpreter:
maurycy@gimel cpython (main 9028df3) % cat /tmp/repro_stale_tstate.py
import os, threading, time
from concurrent import interpreters
from _remote_debugging import RemoteUnwinder
interp = interpreters.create()
t = threading.Thread(target=interp.exec, args=("import time; time.sleep(0.5)",)); t.start(); time.sleep(0.2)
unwinder = RemoteUnwinder(os.getpid())
t.join()
print([(i.interpreter_id, [f.funcname for t in i.threads for f in t.frame_info]) for i in unwinder.get_stack_trace()])
interp.close()
maurycy@gimel cpython (main 9028df3) % ./python.exe /tmp/repro_stale_tstate.py
[(1, [])]Cause
The constructor saves runtime->interpreters.head as self->interpreter_addr.
With all_threads=False, it also saves head->threads_main as self->tstate_addr:
cpython/Modules/_remote_debugging/module.c
Lines 422 to 423 in 9028df3
| if (populate_initial_state_data(all_threads, self, self->runtime_start_address, | |
| &self->interpreter_addr ,&self->tstate_addr) < 0) |
cpython/Modules/_remote_debugging/threads.c
Lines 92 to 128 in 9028df3
| uintptr_t interpreter_state_list_head = | |
| (uintptr_t)unwinder->debug_offsets.runtime_state.interpreters_head; | |
| uintptr_t address_of_interpreter_state; | |
| int bytes_read = _Py_RemoteDebug_PagedReadRemoteMemory( | |
| &unwinder->handle, | |
| runtime_start_address + interpreter_state_list_head, | |
| sizeof(void*), | |
| &address_of_interpreter_state); | |
| if (bytes_read < 0) { | |
| set_exception_cause(unwinder, PyExc_RuntimeError, "Failed to read interpreter state address"); | |
| return -1; | |
| } | |
| if (address_of_interpreter_state == 0) { | |
| PyErr_SetString(PyExc_RuntimeError, "No interpreter state found"); | |
| set_exception_cause(unwinder, PyExc_RuntimeError, "Interpreter state is NULL"); | |
| return -1; | |
| } | |
| *interpreter_state = address_of_interpreter_state; | |
| if (all_threads) { | |
| *tstate = 0; | |
| return 0; | |
| } | |
| uintptr_t address_of_thread = address_of_interpreter_state + | |
| (uintptr_t)unwinder->debug_offsets.interpreter_state.threads_main; | |
| if (_Py_RemoteDebug_PagedReadRemoteMemory( | |
| &unwinder->handle, | |
| address_of_thread, | |
| sizeof(void*), | |
| tstate) < 0) { | |
| set_exception_cause(unwinder, PyExc_RuntimeError, "Failed to read main thread state address"); | |
| return -1; |
Where:
- New interpreters are prepended. A saved head can omit new entries or point to freed memory.
- The head provides the thread to sample. It is the newest interpreter, and
get_stack_trace()stops after it when the saved thread address is non-zero.
Line 740 in 9028df3
| interpreters->head = interp; |
Lines 1050 to 1066 in 9028df3
| *p = interp->next; | |
| if (interpreters->main == interp) { | |
| interpreters->main = NULL; | |
| if (interpreters->head != NULL) { | |
| Py_FatalError("remaining subinterpreters"); | |
| } | |
| } | |
| HEAD_UNLOCK(runtime); | |
| _Py_qsbr_fini(interp); | |
| _PyObject_FiniState(interp); | |
| PyConfig_Clear(&interp->config); | |
| free_interpreter(interp); |
cpython/Modules/_remote_debugging/module.c
Lines 887 to 890 in 9028df3
| // If we're targeting a specific thread, stop after first interpreter | |
| if (self->tstate_addr != 0) { | |
| break; | |
| } |
An idle subinterpreter has threads_main == NULL. That sets tstate_addr to zero, which seems to select all threads. A running subinterpreter can also release its saved main thread state after exec() returns:
Line 2621 in 9028df3
| _PyInterpreterState_SetNotRunningMain(tstate->interp); |
Two questions:
- The CLI docs say "only the main thread". The method docstring says "the main thread of each interpreter". Sampling each interpreter's main thread would follow the method docstring.
- Some embedders may not set
threads.main. The proposed selection returns no default-mode threads for those interpreters, instead of accidentally returning all threads.
cpython/Modules/_remote_debugging/module.c
Lines 670 to 674 in 9028df3
| - If only_active_thread was True: returns only the thread holding | |
| the GIL across all interpreters | |
| - If all_threads was True: returns all threads across all | |
| interpreters | |
| - Otherwise: returns only the main thread of each interpreter |
cpython/Doc/library/profiling.sampling.rst
Line 477 in 9028df3
| By default, the profiler samples only the main thread. The :option:`--all-threads` |
Lines 1164 to 1170 in 9028df3
| // Embedders might not know to call _PyInterpreterState_SetRunningMain(), | |
| // so their main thread wouldn't show it is running the main interpreter's | |
| // program. (Py_Main() doesn't have this problem.) For now this isn't | |
| // critical. If it were, we would need to infer "running main" from other | |
| // information, like if it's the main interpreter. We used to do that | |
| // but the naive approach led to some inconsistencies that caused problems. | |
| return 0; |
CPython versions tested on:
CPython main branch
Operating systems tested on:
macOS
Activity
maurycy commentedon Oct 12, 2026
@pablogsal I will be working on this one, too
The obvious thing here is avoiding looping over interpreters over and over (so we don't read remote memory). To put aside optimizations like leveraging batched reads or adding a yet another
interpreters_generation, the best would be having the main in_Py_DebugOffsets. Can we afford this?Or do we want to introduce and backport
--interpreters?