Repository navigation
Bug: etag is a required field in S3EventNotificationObjectModel however is not present in DeleteObject events #4156
Description
Expected Behaviour
etag should be an Optional field in this pydantic schema
Current Behaviour
Good afternoon chaps.
I'm noticing what I think is an issue with one of the pydantic models in powertools
class S3EventNotificationObjectModel(BaseModel):
key: str
size: Optional[NonNegativeFloat] = None
etag: str
version_id: str = Field(None, alias="version-id")
sequencer: Optional[str] = Noneetag here is required but it seems that for S3 DeleteObject events, etag is sometimes not set:
For example, here's an excerpt of an event I received from S3 via eventbridge:
{
"version": "0",
"id": "5afe384b-7341-a76f-25ab-29bac47a969a",
"detail-type": "Object Deleted",
"source": "aws.s3",
"account": "REDACTED",
"time": "REDACTED",
"region": "eu-west-1",
"resources": [
"arn:aws:s3:::my_bucket"
],
"detail": {
"version": "0",
"bucket": {
"name": "my_bucket"
},
"object": {
"key": "REDACTED",
"sequencer": "REDACTED"
},
"request-id": "REDACTED",
"requester": "REDACTED",
"source-ip-address": "REDACTED",
"reason": "DeleteObject",
"deletion-type": "Permanently Deleted"
}
}The pydantic schema is expecting detail.object.etag to be present here, however since this is a deletion event it doesn't exist and the event therefore seems invalid to pydantic
I think the resolution needed here is just to make etag an Optional field in the pydantic schema.
In my code I've tested this by making a custom version of S3EventNotificationObjectModel where etag is Optional and this does resolve my issue.
Code snippet
class MySqsRecordModel(SqsRecordModel):
"""Model for an individual record from the SQS event we received.
We expect this to be an EventBridge event, which contains an S3 event.
"""
body: Json[S3EventNotificationEventBridgeModel]
class MySqsEventModel(BaseModel):
"""Model for the SQS event we expect to receive."""
Records: Sequence[MySqsRecordModel]
def lambda_handler(event: dict, context: LambdaContext) -> dict:
"""Handles the main entry point for the Lambda function."""
logger.info("Received event", extra={"event": event, "context": context})
# Attempt to parse the event according to the model we expect
try:
parsed_event: MySqsEventModel = parse(
event=event, model=MySqsEventModel
)
except ValidationError as exc:
logger.error("Failed to parse event", extra={"error": exc})
raisePossible Solution
Make etag an optional field in the pydantic schema so validation succeeds
Steps to Reproduce
In my own code I have an S3 bucket with events going to EventBridge. EventBridge forwards to SQS queue and lambda pulls from SQS queue.
All that's needed to observe the issue is to delete an object from the bucket, which creates a DeleteObject event. This event goes from S3 bucket -> EventBridge -> SQS Queue -> Lambda and the issue is visible in the lambda when it tries to parse the event according to schema in the code snippet.
Powertools for AWS Lambda (Python) version
2.37
AWS Lambda function runtime
3.10
Packaging format used
PyPi
Debugging logs
No response
Activity
boring-cyborg commentedon Apr 18, 2024
Thanks for opening your first issue here! We'll come back to you as soon as we can.
In the meantime, check out the #python channel on our Powertools for AWS Lambda Discord: Invite link
leandrodamascena commentedon Apr 19, 2024
Hey @benjamingorman! Thanks for opening this issue and reporting this bug.
Looks like we need to fix the problem in our Event Source Data Class we well: https://github.com/aws-powertools/powertools-lambda-python/blob/develop/aws_lambda_powertools/utilities/data_classes/s3_event.py#L28
Do you want to submit a PR to fix this? We'd love to have your first contribution here.
heitorlessa commentedon May 7, 2024
Update: we're merging this shortly. We've got a few other fixes to get done and will look to make a release this or next week.
Thank you @benjamingorman for all the effort in reporting and creating a PR for this, we can't thank you enough!
For anyone reading later, what we've confirmed is:
etag(EventBridge->S3) andeTag(S3->Lambda/SNS/SQS) are omitted for Object Deletion eventssizeis omitted for Object Deletion eventsWhat we've done to complete the PR:
sizereturn type to beintinstead of `strObject deletion omitting both etag and size
{ "version": "0", "id": "46bf2718-5e27-a74d-ec6b-2d95d4e21844", "detail-type": "Object Deleted", "source": "aws.s3", "account": "536254204126", "time": "2024-05-07T09:31:52Z", "region": "eu-west-1", "resources": [ "arn:aws:s3:::s3teststack-mybucketf68f3ff0-gscrrokfnvnw" ], "detail": { "version": "0", "bucket": { "name": "s3teststack-mybucketf68f3ff0-gscrrokfnvnw" }, "object": { "key": "surprised-pikachu.gif", "sequencer": "006639F508B96A7690" }, "request-id": "11K82SGF9X2HBEKD", "requester": "536254204126", "source-ip-address": "15.248.3.15", "reason": "DeleteObject", "deletion-type": "Permanently Deleted" } }Object creation containing both etag and size
{ "version": "0", "id": "52cc447b-01d6-df6f-8ae6-aea40b330218", "detail-type": "Object Created", "source": "aws.s3", "account": "536254204126", "time": "2024-05-07T09:25:51Z", "region": "eu-west-1", "resources": [ "arn:aws:s3:::s3teststack-mybucketf68f3ff0-gscrrokfnvnw" ], "detail": { "version": "0", "bucket": { "name": "s3teststack-mybucketf68f3ff0-gscrrokfnvnw" }, "object": { "key": "surprised-pikachu.gif", "size": 2638774, "etag": "fb21c5a0ff18e29aab890d1d1f6d6e64", "sequencer": "006639F39F56511781" }, "request-id": "B5Z9SBP1A8R7DERF", "requester": "536254204126", "source-ip-address": "15.248.3.15", "reason": "PutObject" } }github-actions commentedon May 7, 2024
This issue is now closed. Please be mindful that future comments are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.
github-actions commentedon May 17, 2024
This is now released under 2.38.0 version!