Skip to content

fix(logger): copy thread-local keys in thread_safe_remove_keys - #8772

Open
haeganm wants to merge 1 commit into
aws-powertools:developfrom
haeganm:fix/logger-thread-safe-remove-keys-copy
Open

haeganm wants to merge 1 commit into
aws-powertools:developfrom
haeganm:fix/logger-thread-safe-remove-keys-copy

Conversation

@haeganm

@haeganm haeganm commented Nov 4, 2026

Copy link
Copy Markdown

Issue number: closes #8771

Summary

Changes

remove_context_keys popped keys out of the dict stored in the THREAD_LOCAL_KEYS ContextVar in place. asyncio tasks and asyncio.to_thread run in a copy of the caller's context, and that copy points at the same dict, so a thread_safe_remove_keys in one place removed the key for the caller and for every task that copied the context. set_context_keys and clear_context_keys already set a new dict, so append and clear were fine. Remove now copies the dict before popping, one line in aws_lambda_powertools/logging/formatter.py.

Added test_thread_safe_remove_keys_does_not_affect_tasks_sharing_the_context next to the existing thread-safe tests. It appends a key, starts a task, removes the key, and checks the task still logs it while the caller doesn't. Only the public Logger methods are used. No existing tests changed.

User experience

Before: appending order_id, spawning a task and then removing order_id drops it from the task's logs too. A remove inside a task or a to_thread worker also wipes the key for the caller.

After: remove only affects the current context, same as append and clear. Code running in a single context sees no difference.

Verification

Fail before, on develop:

>       assert logs[0].get("order_id") == "order-1"
E       AssertionError: assert None == 'order-1'
============ 1 failed, 4 passed, 34 deselected, 1 warning in 0.06s =============

Pass after:

================= 5 passed, 34 deselected, 1 warning in 0.02s ==================

Also ran, all on this branch:

  • uv run --locked pytest -m "not perf" --ignore tests/e2e: 4054 passed, 73 skipped (Python 3.14.5, env from make dev-quality-code)
  • uv run --locked pytest tests/performance: 10 passed
  • tests/functional/logger on Python 3.10.21: 177 passed, 1 skipped, and the new test fails there without the fix
  • ruff format --check and ruff check: clean. make mypy: no issues in 949 files. bandit baseline and xenon: clean
  • the repro from the issue, on PyPI 3.39.0 and on this branch: order_id missing before, present after

Didn't run the 3.11 to 3.13 matrix locally, or the docker based pre-commit hooks (markdownlint, actionlint, cloudformation). No AWS service is involved here, it's in-process contextvars behaviour, so there's no real-service check to do.

used claude code to help with this.


By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Disclaimer: We value your time and bandwidth. As such, any pull requests created on non-triaged issues might not be successful.

remove_context_keys popped keys out of the dict held by the
THREAD_LOCAL_KEYS ContextVar in place. asyncio tasks and
asyncio.to_thread run in a copy of the caller's context, and that copy
points at the same dict, so removing a key in one of them removed it
for the caller and every other task that copied it.

thread_safe_append_keys and thread_safe_clear_keys already set a new
dict. thread_safe_remove_keys now does the same.
@haeganm
haeganm requested a review from a team as a code owner November 4, 2026 19:10
@boring-cyborg

boring-cyborg Bot commented Nov 4, 2026

Copy link
Copy Markdown

Thanks a lot for your first contribution! Please check out our contributing guidelines and don't hesitate to ask whatever you need.
In the meantime, check out the #python channel on our Powertools for AWS Lambda Discord: Invite link

@powertools-for-aws-oss-automation powertools-for-aws-oss-automation Bot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Nov 4, 2026
@sonarqubecloud

sonarqubecloud Bot commented Nov 4, 2026

Copy link
Copy Markdown

@Manishram-ai Manishram-ai left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for tracking this down. The new test fails on develop (assert None == 'order-1') and passes here, and tests/functional/logger passes (178). I also ran the two snippets from #8771, plus a remove inside contextvars.copy_context().run and a plain thread, against both versions. On develop the to_thread remove drops request_id from the caller's next log line, both tasks lose order_id, and the remove inside ctx.run leaks back to the caller. With this change the caller keeps request_id, the tasks log order-1 and order-2, and the copied context no longer leaks back. Remove now works the same way append and clear already did, by setting a new dict.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

logger size/M Denotes a PR that changes 30-99 lines, ignoring generated files. tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bug: thread_safe_remove_keys in one thread or task removes the key everywhere that copied the context

2 participants