Skip to content

[AISAGE-503] feat(usage-ingest): bound queue retries and concurrency - #7722

Merged
johnnyeric merged 1 commit into
mainfrom
johnnyeric/usage-ingest-queue-policy
Oct 16, 2026
Merged

johnnyeric merged 1 commit into
mainfrom
johnnyeric/usage-ingest-queue-policy

Conversation

@johnnyeric

@johnnyeric johnnyeric commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Bound receipt-consumer delivery to one message per batch and one active invocation in each environment, with no batch-fill wait, three retries and a 60-second retry delay. Route exhausted failures to separate production/staging DLQs; unused producer bindings let pinned Wrangler provision those queues declaratively.

Document the queue policy, declarative provisioning bindings, and retention configuration. Receipt-only ACK behavior is unchanged, including malformed-message discard; direct usage writes remain authoritative.

Verification

  • Manually inspected production and staging dry-run binding output and pinned Wrangler schema/provisioning source for environment isolation and supported fields.
  • Hosted provisioning, consumer settings, concurrency and retention have not been verified. This change has only been built and checked locally; the README documents retention configuration.

Automated checks

  • Service suite: 26 tests passed across 3 files. Wrangler 4.135.0's local createTestHarness runs the configured Worker in workerd/Miniflare for both environments, covering HTTP publish → queue → receipt → ACK and malformed-message discard.
  • Isolated synthetic failure fixture using the configured source queue policy in both environments: transient failure recovered on attempt 2; persistent failure reached the correct DLQ after 4 attempts, with approximately 60 seconds between retries and 1 message per batch.
  • The failure fixture is outside the committed suite and uses a diagnostic DLQ consumer to observe forwarding: pinned Miniflare drops messages on queues without local consumers. It does not prove unattended DLQ retention.
  • Service typecheck and lint passed.
  • Production and staging wrangler deploy --dry-run builds passed with the correct environment-specific source and DLQ bindings.
Recorded local retry/DLQ log

Output from the isolated synthetic fixture described above; the diagnostic DLQ consumer acknowledges forwarded messages.

[wrangler:info] QUEUE usage-ingest-processing 0/1 (8ms)
[wrangler:info] QUEUE usage-ingest-processing-staging 0/1 (7ms)
[wrangler:info] QUEUE usage-ingest-processing-staging 0/1 (0ms)
[wrangler:info] QUEUE usage-ingest-processing 0/1 (0ms)
[wrangler:info] QUEUE usage-ingest-processing 1/1 (0ms)
[wrangler:info] QUEUE usage-ingest-processing-staging 1/1 (1ms)
[wrangler:info] QUEUE usage-ingest-processing 0/1 (0ms)
[wrangler:info] QUEUE usage-ingest-processing-staging 0/1 (1ms)
[wrangler:info] QUEUE usage-ingest-processing 0/1 (2ms)
[wrangler:info] QUEUE usage-ingest-processing-staging 0/1 (2ms)
[wrangler:warn] Moving message "c05327dd0968bed0707cba08951845a5" on queue "usage-ingest-processing" to dead letter queue "usage-ingest-processing-dlq" after 4 failed attempts...
[wrangler:warn] Moving message "f058da473935ed6da4ec9fd8e4e04b74" on queue "usage-ingest-processing-staging" to dead letter queue "usage-ingest-processing-dlq-staging" after 4 failed attempts...
[wrangler:info] QUEUE usage-ingest-processing 0/1 (2ms)
[wrangler:info] QUEUE usage-ingest-processing-staging 0/1 (2ms)
[wrangler:info] QUEUE usage-ingest-processing-dlq 1/1 (0ms)
[wrangler:info] QUEUE usage-ingest-processing-dlq-staging 1/1 (0ms)
{"env":"staging","attempts":[{"event":"fixture_attempt","case":"transient","attempt":1,"size":1,"time":1791906192416},{"event":"fixture_attempt","case":"terminal","attempt":1,"size":1,"time":1791906192434},{"event":"fixture_attempt","case":"transient","attempt":2,"size":1,"time":1791906252433},{"event":"fixture_attempt","case":"terminal","attempt":2,"size":1,"time":1791906252444},{"event":"fixture_attempt","case":"terminal","attempt":3,"size":1,"time":1791906312452},{"event":"fixture_attempt","case":"terminal","attempt":4,"size":1,"time":1791906372468}],"received":[{"event":"fixture_dlq","queue":"usage-ingest-processing-dlq-staging","case":"terminal"}],"sourceBacklog":0,"dlqBacklog":0,"result":"PASS"}
{"env":"production","attempts":[{"event":"fixture_attempt","case":"transient","attempt":1,"size":1,"time":1791906192416},{"event":"fixture_attempt","case":"terminal","attempt":1,"size":1,"time":1791906192434},{"event":"fixture_attempt","case":"transient","attempt":2,"size":1,"time":1791906252434},{"event":"fixture_attempt","case":"terminal","attempt":2,"size":1,"time":1791906252444},{"event":"fixture_attempt","case":"terminal","attempt":3,"size":1,"time":1791906312452},{"event":"fixture_attempt","case":"terminal","attempt":4,"size":1,"time":1791906372468}],"received":[{"event":"fixture_dlq","queue":"usage-ingest-processing-dlq","case":"terminal"}],"sourceBacklog":0,"dlqBacklog":0,"result":"PASS"}
PASS: transient failure recovered; terminal failure had initial + 3 retries then observed DLQ delivery; one message per batch in both environments. Hosted concurrency/provisioning/retention unverified.

Visual Changes

N/A

Reviewer Notes

The production concurrency ceiling of one is an initial conservative bound, not measured capacity. Retry delay adds about three minutes before a promptly failing message reaches the DLQ, plus execution/scheduling time. Local emulation does not enforce hosted consumer concurrency.

Wrangler 4.135.0 cannot encode retention in this config; four-day retention must be set and verified separately at deployment. No DLQ consumer is attached. A DLQ retains exhausted failures within retention; it does not retain acknowledged malformed messages or turn receipt consumption into billing recovery.

// Unused by the Worker: ensures Wrangler provisions the DLQ on deployment.
{ "binding": "USAGE_INGEST_DLQ", "queue": "usage-ingest-processing-dlq" },
],
"consumers": [

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Conservative values - will adjust after testing.

@johnnyeric
johnnyeric marked this pull request as ready for review October 16, 2026 10:28
@kilo-code-bot

kilo-code-bot Bot commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

Code Review Summary

Status: No Issues Found | Recommendation: Merge

Queue-consumer bounding (max_batch_size/max_batch_timeout/max_retries/retry_delay/max_concurrency), per-environment DLQ routing, and the accompanying README/type/test updates are internally consistent and match the config; no high-confidence defects found in the changed lines.

Files Reviewed (4 files)
  • services/usage-ingest/wrangler.jsonc - per-environment consumer policy (batch/concurrency/retry) and unused USAGE_INGEST_DLQ producer bindings that provision the DLQs
  • services/usage-ingest/README.md - queue policy table, deploy notes, and retention section; statements align with the config (max_retries: 3 = initial + 3 retries, 345600s = 4 days)
  • services/usage-ingest/src/index.test.ts - adds the now-required USAGE_INGEST_DLQ binding to the typed test Env
  • services/usage-ingest/worker-configuration.d.ts - generated Wrangler types reflecting the new binding (not hand-edited)

Notes (not defects, no action required):

  • The USAGE_INGEST_DLQ producer binding deviates from sibling services that reference a DLQ only via dead_letter_queue (e.g. services/session-ingest, services/notifications); the PR documents why it exists (declarative provisioning), and it does grant the Worker send access it never uses.
  • Local emulation does not enforce hosted consumer concurrency and Wrangler 4.135.0 cannot encode retention, so hosted concurrency/provisioning/retention remain unverified, as the PR description already states.

Reviewed by deepseek-v4.1-flash · Input: 60.3K · Output: 16.8K · Cached: 604.9K

Review guidance: REVIEW.md from base branch main

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants