Skip to content

fix(no-ticket): do not retry a POST after a 500, 502 or 504 response - #563

Draft
cloudsmith-iduffy wants to merge 1 commit into
iduffy/fix-post-read-timeoutfrom
iduffy/fix-post-5xx-retry
Draft

cloudsmith-iduffy wants to merge 1 commit into
iduffy/fix-post-read-timeoutfrom
iduffy/fix-post-5xx-retry

Conversation

@cloudsmith-iduffy

Copy link
Copy Markdown
Collaborator

Description

The CLI retried a 500, 502 or 504 response for every HTTP method, POST and PATCH included. These codes do not prove that the API ignored the request. The API CDN sends a 504 after 60 seconds while the API keeps working, and a 502 or 500 can arrive after the work is done. The retry then sent the POST again and could create a second entitlement, token, package or copy.

To reproduce, I used a local server that reads each POST and replies 504, and ran cloudsmith entitlements create acme/repo --name ci:

Before (base branch):

Creating ci entitlement for the repo repository ... ERROR
Failed to create the entitlement! (status: 0 - Unknown Status)
Detail: RetryError
... Max retries exceeded ... (Caused by ResponseError('too many 504 error responses'))

The server got 7 POSTs, and the command took 62 seconds.

After:

Creating ci entitlement for the repo repository ... ERROR
Failed to create the entitlement! (status: 504 - Gateway Timeout)
Detail: Gateway Timeout

The server got 1 POST, and the command took 0.2 seconds.

What changes:

  • RetryWithCallback.is_retry no longer retries a POST or PATCH after a 500, 502 or 504 response. It uses the same NON_IDEMPOTENT_METHODS set as the read-error rule from fix(no-ticket): stop a stalled POST from waiting forever #556.
  • A 503 or a 429 is still retried for every method, and the CLI still honours Retry-After. These codes mean that the server did not handle the request.
  • GET, PUT, DELETE and other methods keep the old behaviour.
  • --error-retry-codes: the CLI ignores 500, 502 and 504 for POST and PATCH, even when you list them. A user who sets this option wants retries for transient errors, not a second entry. The help text now says this.
  • File upload: upload_file sends a POST to the upload URL. A second upload of the same file replaces the first, so the upload opts back in with the new create_requests_session(repeat_non_idempotent=True). The upload also keeps its retries after a read error or a dropped connection, which fix(no-ticket): stop a stalled POST from waiting forever #556 had turned off for it.

Type of Change

  • Bug fix
  • New feature
  • Breaking change
  • Documentation update
  • Refactoring
  • Other (please describe)

Additional Notes

This PR is stacked on #556.

Why it has the breaking change label: the CLI now retries less for POST and PATCH. A script that ran a create command against a flaky gateway and depended on the automatic retry now gets the 500/502/504 error at once. The error also changes: before, the CLI reported status: 0 with a RetryError after it used up its retries. Now it reports the real status (for example 504 - Gateway Timeout), so the exit code changes too. In the repro above, it went from 1 to 248.

Upload retries need #544. On this base, the upload body cannot be rewound, so a retry sends an empty body under the original Content-Length. Upload retries already do this on master, so this PR does not make it worse, and #544 fixes it.

Other POSTs that are safe to repeat, such as the OIDC token exchange and the Azure DevOps OIDC token request, also lose their 500/502/504 retries. A transient 502 during the OIDC exchange in CI now fails at once. They can opt in with repeat_non_idempotent=True in a follow-up if we want that.

Tests: POST + 504 is sent once, POST + 503 is retried, GET + 504 is retried, and an upload POST + 504 is retried.

🤖 Generated with Claude Code

A gateway can send these codes after the API processed the request.
A retry then sends the POST again and can create a second entry.
The CLI still retries a 429 or 503 response. A file upload keeps its
retries, because a second upload of the same file is safe.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Copilot AI balanced review requested due to automatic review settings October 24, 2026 18:14
@cloudsmith-iduffy cloudsmith-iduffy added the breaking change Changes behaviour that scripts or users can depend on label Oct 24, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Upload read-error retries are re-enabled before the body is rewindable, allowing exhausted-body retries to hang.

1 open issue
What changed in this PR

This PR extends the shared retry policy to prevent duplicate API operations after ambiguous server errors, building on #556’s read-error protection.

Changes:

  • Stops POST and PATCH retries after 500, 502, or 504 responses while preserving other retry behavior.
  • Adds an explicit retry opt-in for repeatable requests and enables it for uploads.
  • Documents the behavior and adds request-count regression tests.
File Description
cloudsmith_cli/​core/​tests/​test_rest.py Tests method-specific status retries.
cloudsmith_cli/​core/​tests/​test_files.py Tests upload gateway-error retries.
cloudsmith_cli/​core/​session.py Adds status-retry protection and the repeatable-request opt-in.
cloudsmith_cli/​core/​api/​files.py Enables upload retries through the opt-in.
cloudsmith_cli/​cli/​decorators.py Clarifies retry-code option restrictions.
CHANGELOG.md Documents API and upload retry behavior.

Review effort: Balanced


session = create_requests_session()
# A second upload of the same file replaces the first one.
session = create_requests_session(repeat_non_idempotent=True)

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

breaking change Changes behaviour that scripts or users can depend on

Development

Successfully merging this pull request may close these issues.

2 participants