Bug: thread_safe_remove_keys in one thread or task removes the key everywhere that copied the context #8771

Expected Behaviour

The logger docs say "Remove keys persist across all future logs in a specific thread via thread_safe_remove_keys". So a remove in a worker thread should only drop the key in that thread, and the caller should keep logging it. Same for asyncio tasks, which run in a copy of the caller's context. That's how thread_safe_append_keys and thread_safe_clear_keys already behave.

Current Behaviour

remove_context_keys in aws_lambda_powertools/logging/formatter.py pops the keys out of the dict stored in the THREAD_LOCAL_KEYS ContextVar in place, then sets the same dict back. A copied context (asyncio.to_thread, asyncio.create_task, contextvars.copy_context().run) holds a reference to that same dict, so the removal shows up everywhere that copied it.

Append and clear don't have this problem because set_context_keys and clear_context_keys both set a new dict.

Code snippet

A remove inside a to_thread worker wipes the key for the caller:

import asyncio
from aws_lambda_powertools import Logger

logger = Logger(service="payment")

def sync_work():
    logger.thread_safe_remove_keys(["request_id"])  # only meant for this worker thread
    logger.info("worker thread")

async def main():
    logger.thread_safe_append_keys(request_id="abc")
    await asyncio.to_thread(sync_work)
    logger.info("caller, request_id should still be here")

asyncio.run(main())

Output on 3.39.0 (same on develop), the caller's line has no request_id:

{"level":"INFO","location":"sync_work:8","message":"worker thread","timestamp":"2026-11-04 10:48:31,324-0800","service":"payment"}
{"level":"INFO","location":"main:13","message":"caller, request_id should still be here","timestamp":"2026-11-04 10:48:31,324-0800","service":"payment","taskName":"Task-1"}

And the other direction with tasks, the caller removing a key drops it from tasks it already started:

import asyncio

from aws_lambda_powertools import Logger

logger = Logger(service="payment")


async def process(order_id):
    await asyncio.sleep(0)  # let the caller move on, like any real await would
    logger.info("processing order")


async def main():
    tasks = []
    for order_id in ("order-1", "order-2"):
        logger.thread_safe_append_keys(order_id=order_id)
        tasks.append(asyncio.create_task(process(order_id)))  # task gets a copy of the current context
        logger.thread_safe_remove_keys(["order_id"])  # caller is done with the key
    await asyncio.gather(*tasks)


asyncio.run(main())

{"level":"INFO","location":"process:10","message":"processing order","timestamp":"2026-11-04 10:41:24,436-0800","service":"payment","taskName":"Task-2"}
{"level":"INFO","location":"process:10","message":"processing order","timestamp":"2026-11-04 10:41:24,437-0800","service":"payment","taskName":"Task-3"}

Swap the thread_safe_remove_keys line for logger.thread_safe_clear_keys() and both lines carry "order_id":"order-1" / "order_id":"order-2" (checked on develop), so it's only the remove path.

Possible Solution

Copy the dict before popping, context_values = dict(context.get()), so remove sets a new dict like append and clear do. Fix ready with a regression test, PR to follow.

used claude code to help with this.

Steps to Reproduce

1. pip install aws-lambda-powertools==3.39.0
2. Run either snippet (I used Python 3.14.5 on macOS 26.6.2, no AWS calls involved).
3. The to_thread one: the caller's log line has no request_id. The task one: neither line has order_id.

Did you run the reproduction yourself?

Yes, I ran it and saw the failure

Powertools for AWS Lambda (Python) version

3.39.0, also develop at a1806e1

AWS Lambda function runtime

3.14

Packaging format used

PyPi

Debugging logs

No response