Skip to content

Tolerate missing storage objects in the file-size backfill - #2094

Merged
netomi merged 1 commit into
mainfrom
fix/1777-filesize-migration-missing-object
Aug 25, 2026
Merged

netomi merged 1 commit into
mainfrom
fix/1777-filesize-migration-missing-object

Conversation

@netomi

@netomi netomi commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

What

Fixes the crash reported after merging #2089:

WARN --- [backgroundjob-worker] o.jobrunr.server.BackgroundJobPerformer  : Job(..., jobName='Determine file size for published file resource') processing failed
software.amazon.awssdk.services.s3.model.NoSuchKeyException: (Service: S3, Status Code: 404, ...)

FileResourceSizeJobRequestHandler (the size backfill job from #1777/#2089) crashes whenever a FileResource row points at an object that's no longer actually in storage. This isn't transient — retrying will always fail the same way — but the job let it propagate as a generic failure, so it burned through @Retryable's 3 attempts + JobRunr's own 3 retries, then permanently failed and left the migration_item row behind for investigation (per #2092's MigrationItemCleanupFilter), even though there's nothing to investigate: the file is just gone.

This is a known, already-tolerated category of problem in this codebase — see FixMissingFilesJobRequestHandler, which exists specifically to detect and repair extensions with missing files. This PR makes the size backfill tolerate it the same way instead of crash-looping on it.

How

  • New FileNotFoundInStorageException (an IOException) signals "the object is confirmed absent", as opposed to some other, possibly-transient, storage error.
  • Each storage backend translates its own "definitely not found" signal into it:
    • AWS: a HEAD response has no body, so S3 can't return a specific error code the way a GET can — some SDKs/implementations throw NoSuchKeyException regardless, but a plain S3Exception with a 404 status is what's actually guaranteed. I verified against a real LocalStack S3 instance that LocalStack's HeadObject on a missing key throws the plain S3Exception, not NoSuchKeyException — so the fix checks the status code rather than the exception subtype (matching the pattern this codebase already uses in AzureBlobStorageService.removeFile).
    • Azure: BlobErrorCode.BLOB_NOT_FOUND.
    • GCS: already represented as a null Blob — now translated explicitly instead of a generic IOException.
    • Local: NoSuchFileException from Files.size(...).
    • Any other storage error (auth failure, network blip, throttling, etc.) still propagates unchanged and is still retried as before.
  • MigrationService.getFileSize excludes FileNotFoundInStorageException from its @Retryable — retrying a confirmed absence just spends 3 more round trips (and delay) confirming the same thing again.
  • FileResourceSizeJobRequestHandler catches it, logs a warning, and leaves the resource's size unset instead of letting the job fail. There's nothing more this job can do for a resource whose file is gone; FixMissingFilesMigration is the actual repair mechanism.

Testing

  • New FileResourceSizeJobRequestHandlerTest: the job completes normally (no size set, no exception) when storage reports the object missing; still records the size on the normal path.
  • New AwsStorageServiceIntegrationTest.testGetFileSizeThrowsFileNotFoundInStorageExceptionForAMissingObject, run against real LocalStack — this is what caught that LocalStack throws a plain S3Exception, not NoSuchKeyException, for a missing HEAD target.
  • Full unitTests suite passes (948 tests) and s3IntegrationTests passes (24 tests).

🤖 Generated with Claude Code

The FileResourceSizeMigration job (#1777) crashes with a raw
NoSuchKeyException/BlobStorageException/NoSuchFileException whenever a
FileResource points at an object that's no longer in storage, exhausting
JobRunr's retries and leaving the migration_item behind for investigation
(per MigrationItemCleanupFilter) even though the outcome will never change
on retry. Resources with genuinely missing backing files are a known,
already-tolerated condition elsewhere in this codebase (see
FixMissingFilesJobRequestHandler), so treat it the same way here.

- New FileNotFoundInStorageException (IOException) signals "the object is
  confirmed absent", distinct from other storage errors that might be
  transient and worth retrying.
- Each backend translates its own "definitely not found" signal into it:
  AWS (a HEAD response has no body, so S3 can't return a specific error
  code -- some implementations throw NoSuchKeyException regardless, but a
  plain S3Exception with a 404 status is what's actually guaranteed, and
  what LocalStack sends, so the status code is checked rather than the
  exception type), Azure (BlobErrorCode.BLOB_NOT_FOUND), GCS (already
  represented as a null Blob), Local (NoSuchFileException). Other storage
  errors continue to propagate unchanged.
- MigrationService.getFileSize excludes FileNotFoundInStorageException
  from its @retryable, since retrying a confirmed absence just spends 3
  more round trips (and delay) confirming the same thing again.
- FileResourceSizeJobRequestHandler catches it, logs a warning, and
  leaves the resource's size unset instead of letting the job fail --
  there's nothing more this job can do for a resource whose file is
  gone; FixMissingFilesMigration is what actually repairs that.

Verified against real LocalStack S3 that a HEAD on a missing key surfaces
as a plain S3Exception rather than NoSuchKeyException specifically, which
is why the AWS side checks status code rather than catching the specific
subtype.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@netomi
netomi merged commit f4a2fd4 into main Aug 25, 2026
5 checks passed
@netomi
netomi deleted the fix/1777-filesize-migration-missing-object branch August 25, 2026 07:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant