sys.remote_exec() can fail when subinterpreters exist

Bug report

When there's a subinterpreters, sys.remote_exec() can fail to find the main thread or run the script in the subinterpreter.

Reproduction

An idle subinterpreter makes a previously successful call fail:

import os, sys, tempfile
from pathlib import Path
from concurrent import interpreters

with tempfile.TemporaryDirectory() as directory:
    script = Path(directory) / "remote.py"
    script.write_text("print('script executed')")
    sys.remote_exec(os.getpid(), str(script))
    interp = interpreters.create()
    try:
        sys.remote_exec(os.getpid(), str(script))
    except RuntimeError as error:
        print(type(error).__name__ + ": " + str(error))
    finally:
        interp.close()

Output on 9028df36bb9:

maurycy@gimel cpython (main 9028df3) % ./python.exe /tmp/re1_repro.py
script executed
RuntimeError: Can't find the main thread in the remote process
maurycy@gimel cpython (main 9028df3) %

Both sys.remote_exec calls should execute the script.

Also, a running subinterpreter receives the script instead of the main interpreter.

This example calls remote_exec() from the subinterpreter itself:

import os, sys, tempfile
from pathlib import Path
from concurrent import interpreters

with tempfile.TemporaryDirectory() as directory:
    script = Path(directory) / "remote.py"
    script.write_text("from concurrent import interpreters; "
                      "print('executed in interpreter', interpreters.get_current().id)")
    interp = interpreters.create()
    try:
        interp.exec(f"import os, sys; sys.remote_exec(os.getpid(), {str(script)!r})")
    finally:
        interp.close()

Output on 9028df36bb9:

maurycy@gimel cpython (main 9028df3) % ./python.exe /tmp/re2_repro.py
executed in interpreter 1
maurycy@gimel cpython (main 9028df3) %

I believe the expected is 0:

Cause

New interpreters enter the start of _PyRuntime.interpreters. The list head is the newest interpreter, not the main interpreter:

cpython/Python/pystate.c

Line 740 in 9028df3

interpreters->head = interp;

Ans send_exec_to_proc_handle() reads only the head:

cpython/Python/remote_debugging.c

Lines 125 to 133 in 9028df3

    
           uintptr_t interpreter_state_list_head = (uintptr_t)debug_offsets.runtime_state.interpreters_head; 
        
           uintptr_t interpreter_state_addr; 
        
           if (0 != read_memory( 
        
                   handle, 
        
                   runtime_start_address + interpreter_state_list_head, 
        
                   sizeof(void*), 
        
                   &interpreter_state_addr)) 
        
           {

For tid == 0, which sys.remote_exec() passes, it uses that interpreter's threads_main:

cpython/Python/remote_debugging.c

Lines 202 to 216 in 9028df3

    
           if (0 != read_memory( 
        
                   handle, 
        
                   interpreter_state_addr + (uintptr_t)debug_offsets.interpreter_state.threads_main, 
        
                   sizeof(void*), 
        
                   &thread_state_addr)) 
        
           { 
        
               return -1; 
        
           } 
        
           if (thread_state_addr == 0) { 
        
               PyErr_SetString( 
        
                   PyExc_RuntimeError, 
        
                   "Can't find the main thread in the remote process"); 
        
               return -1; 
        
           }

An idle subinterpreter has no main thread state. A running subinterpreter has one, but I think (TBD) it'd belong to the wrong interpreter.

The HOWTO repeats the same lookup for the main thread:

cpython/Doc/howto/remote_debugging.rst

Lines 498 to 532 in 9028df3

    
           1. Use the offset ``runtime_state.interpreters_head`` to obtain the address of 
        
              the first interpreter in the ``PyRuntime`` structure. This is the entry point 
        
              to the linked list of active interpreters. 
        
           2. Use the offset ``interpreter_state.threads_main`` to access the main thread 
        
              state associated with the selected interpreter. This is typically the most 
        
              reliable thread to target. 
        
           3. Optionally, use the offset ``interpreter_state.threads_head`` to iterate 
        
              through the linked list of all thread states. Each ``PyThreadState`` 
        
              structure contains a ``native_thread_id`` field, which may be compared to 
        
              a target thread ID to find a specific thread. 
        
           4. Once a valid ``PyThreadState`` has been found, its address can be used in 
        
              later steps of the protocol, such as writing debugger control fields and 
        
              scheduling execution. 
        
           The following is an example implementation that locates the main thread state:: 
        
               def find_main_thread_state( 
        
                   pid: int, py_runtime_addr: int, debug_offsets: DebugOffsets, 
        
               ) -> int: 
        
                   # Step 1: Read interpreters_head from PyRuntime 
        
                   interp_head_ptr = ( 
        
                       py_runtime_addr + debug_offsets.runtime_state.interpreters_head 
        
                   ) 
        
                   interp_addr = read_pointer(pid, interp_head_ptr) 
        
                   if interp_addr == 0: 
        
                       raise RuntimeError("No interpreter found in the target process") 
        
                   # Step 2: Read the threads_main pointer from the interpreter 
        
                   threads_main_ptr = ( 
        
                       interp_addr + debug_offsets.interpreter_state.threads_main 
        
                   ) 
        
                   thread_state_addr = read_pointer(pid, threads_main_ptr)

I'm working on the fix.

CPython versions tested on:

CPython main branch (likely backport to 3.14 and 3.15)

Operating systems tested on:

macOS

Linked PRs

gh-159194: Fix sys.remote_exec() with a subinterpreter #159259

@pablogsal

cc @pablogsal

Confirmed send_exec_to_proc_handle() relies on runtime_state.interpreters_head to select a thread state, but this can point to a subinterpreter rather than the main interpreter, causing tid=0 to fail when the subinterpreter has no usable threads_main and selecting the wrong interpreter for execution. Same goes for 3.14, 3.15, and main

The fix should select the intended interpreter and thread state explicitly, with regression tests covering idle and running subinterpreters, nonzero thread IDs, and execution from a subinterpreter

@maurycy

@maurycy are you planning a fix?

@pablogsal

@pablogsal Yes

Navigation Menu

`sys.remote_exec()` can fail when subinterpreters exist #159194

Bug report

Reproduction

Cause

CPython versions tested on:

Operating systems tested on:

Linked PRs

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

	uintptr_t interpreter_state_list_head = (uintptr_t)debug_offsets.runtime_state.interpreters_head;

	uintptr_t interpreter_state_addr;
	if (0 != read_memory(
	handle,
	runtime_start_address + interpreter_state_list_head,
	sizeof(void*),
	&interpreter_state_addr))
	{

	if (0 != read_memory(
	handle,
	interpreter_state_addr + (uintptr_t)debug_offsets.interpreter_state.threads_main,
	sizeof(void*),
	&thread_state_addr))
	{
	return -1;
	}

	if (thread_state_addr == 0) {
	PyErr_SetString(
	PyExc_RuntimeError,
	"Can't find the main thread in the remote process");
	return -1;
	}

	1. Use the offset ``runtime_state.interpreters_head`` to obtain the address of
	the first interpreter in the ``PyRuntime`` structure. This is the entry point
	to the linked list of active interpreters.

	2. Use the offset ``interpreter_state.threads_main`` to access the main thread
	state associated with the selected interpreter. This is typically the most
	reliable thread to target.

	3. Optionally, use the offset ``interpreter_state.threads_head`` to iterate
	through the linked list of all thread states. Each ``PyThreadState``
	structure contains a ``native_thread_id`` field, which may be compared to
	a target thread ID to find a specific thread.

	4. Once a valid ``PyThreadState`` has been found, its address can be used in
	later steps of the protocol, such as writing debugger control fields and
	scheduling execution.

	The following is an example implementation that locates the main thread state::

	def find_main_thread_state(
	pid: int, py_runtime_addr: int, debug_offsets: DebugOffsets,
	) -> int:
	# Step 1: Read interpreters_head from PyRuntime
	interp_head_ptr = (
	py_runtime_addr + debug_offsets.runtime_state.interpreters_head
	)
	interp_addr = read_pointer(pid, interp_head_ptr)
	if interp_addr == 0:
	raise RuntimeError("No interpreter found in the target process")

	# Step 2: Read the threads_main pointer from the interpreter
	threads_main_ptr = (
	interp_addr + debug_offsets.interpreter_state.threads_main
	)
	thread_state_addr = read_pointer(pid, threads_main_ptr)

sys.remote_exec() can fail when subinterpreters exist #159194

Description

Bug report

Reproduction

Cause

CPython versions tested on:

Operating systems tested on:

Linked PRs

Activity

maurycy commented on Oct 11, 2026

sharktide commented on Oct 12, 2026

sharktide commented on Oct 12, 2026

pablogsal commented on Oct 12, 2026

maurycy commented on Oct 12, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

`sys.remote_exec()` can fail when subinterpreters exist #159194