Bug: a result that is not JSON serializable leaves the idempotency record stuck as INPROGRESS #8721

Expected Behaviour

If the decorated function returns something the persistence layer cannot save, the invocation fails with IdempotencyPersistenceLayerError and the INPROGRESS record is removed, so the next call with the same payload runs the function again. That is what already happens when the function itself raises.

Current Behaviour

save_success fails (for example on a datetime, set or bytes in the result), the error is wrapped as IdempotencyPersistenceLayerError, and the INPROGRESS record stays. Every following call with the same payload gets IdempotencyAlreadyInProgressError until the in-progress expiry, and without register_lambda_context there is no in-progress expiry, so the key is locked for expires_after_seconds (one hour by default).

Code snippet

from datetime import datetime
from aws_lambda_powertools.utilities.idempotency import IdempotencyConfig, idempotent_function
from aws_lambda_powertools.utilities.idempotency.persistence.redis import CachePersistenceLayer

layer = CachePersistenceLayer(client=redis_client)

@idempotent_function(data_keyword_argument="order", persistence_store=layer, config=IdempotencyConfig(event_key_jmespath="id"))
def handler(order):
    return {"created": datetime(2026, 1, 1)}

handler(order={"id": 1})  # IdempotencyPersistenceLayerError: Object of type datetime is not JSON serializable
handler(order={"id": 1})  # IdempotencyAlreadyInProgressError, and it stays like this for an hour

Same with DynamoDBPersistenceLayer; the in-memory layer used in the tests reproduces it too.

Possible Solution

In aws_lambda_powertools/utilities/idempotency/base.py, IdempotencyHandler._process: when save_success raises, call self.persistence_store.delete_record(...) before re-raising, mirroring what the exception path already does when the function raises. Add a test in tests/functional/idempotency/_boto3/test_idempotency.py with a non-serializable return: first call raises IdempotencyPersistenceLayerError, second call runs the function again instead of raising IdempotencyAlreadyInProgressError.

Item 4 of the Idempotency audit in #8603. Everything is in base.py and the existing test helpers, no AWS resources needed.

Powertools for AWS Lambda (Python) version

latest

AWS Lambda function runtime

3.13

Packaging format used

PyPi

Debugging logs

No response