Skip to content

Fix dead letters pane issues found in review - #712

Merged
bgentry merged 19 commits into
brandur-dead-letters-in-uifrom
bg-dead-letter-ui-fixes
Oct 24, 2026
Merged

bgentry merged 19 commits into
brandur-dead-letters-in-uifrom
bg-dead-letter-ui-fixes

Conversation

@bgentry

@bgentry bgentry commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

This stacks on #711 and fixes the problems I found while reviewing it. Each fix is its own commit, so any of them can be dropped or cherry-picked on its own.

The biggest change is to the dead letters pane itself. In the light theme most of its text was invisible, because the root style sets a near-white default text color and the new components set no color of their own. The pane also had its own table and detail layout, which hid every column but the first on a phone and left out the state, priority, tags, and timeline. The list and detail pages now reuse the Jobs layouts, the not-found view uses the same 404 layout as Jobs, and both components have stories.

The list now returns the newest dead letters first. Before, it sorted by ID ascending with a cap of 200 rows, so once the table held more than 200 rows the newest failures could only be reached with a filter. An ID filter now runs through its own query that searches the primary key; with the previous OR form, an exact ID lookup read the whole table on both Postgres and SQLite. Filters on kind, queue, priority, and tags still read the table, because it has no indexes for them.

The rest are smaller fixes:

  • A URL with a filter that can't be parsed, such as ?priority=5, opens the list without that filter instead of the error page, as the Jobs list does.
  • An unknown URL, or /dead-letter-jobs on a server without dead letter support, shows a "Page not found" page instead of a blank one.
  • A failed background refresh keeps the loaded rows and shows an inline alert, instead of replacing the pane with the error page.
  • Autocomplete matches and sorts the same way on Postgres and SQLite, and a NUL byte or invalid UTF-8 in a filter returns a 400 on both.
  • A job search suggestion request is aborted when a newer keystroke replaces it. This applies to Jobs too.
  • Tests cover the missing-table guard, the real size limits, and the autocomplete guards.

Two commits are matters of taste and are easy to drop: one moves Dead Letters below Workflows in the sidebar, and one truncates a long job kind in the detail heading with a copy button, as the workflow detail page already does for workflow names.

Not included: a pagination cursor, a delete action, and indexes for the kind, queue, and tag filters.

Screenshots

View Before After
List, light mode 1-list-light-before 1-list-light-after
Detail, light mode 2-detail-light-before 2-detail-light-after
No filter matches 3-no-matches-before 3-no-matches-after
Direct URL with the feature off 4-feature-off-before 4-feature-off-after
Mobile 5-mobile-before 5-mobile-after
List, dark mode 6-list-dark-before 6-list-dark-after

The Jobs list owns its empty state and its Fewer/More controls, the
job detail page owns its whole layout, and `JobNotFound` owns the 404
layout. Another pane that shows jobs has to copy that markup, and the
copies drift.

Move the empty state, the pagination controls, and the 404 layout into
their own components. Split the job detail page into `JobDetailView`,
which takes its actions as a prop, so a pane with different actions can
reuse the same layout. Jobs renders the same as before.
Each keystroke in the job search field starts a new suggestion request,
but the earlier requests keep running. The server finishes a query for
every one of them, which is costly when the query reads a large table.

Create an `AbortController` per suggestion request and abort it when a
newer request replaces it. Pass the signal to the suggestion fetchers
for both Jobs and dead letters so the browser cancels the HTTP request.
A test confirms that the first signal aborts when the second request
starts.
In the light theme, most text in the dead letters pane is invisible.
The root style sets a near-white default text color, and the list
headers, list cells, detail heading, and detail labels set no color of
their own. The pane also uses its own table and detail layout: on a
phone only the first column is visible, and the detail page omits the
state, priority, tags, and timeline that the Jobs pane shows.

Render the list with the same rows as the Jobs list (state dot, kind,
time, attempts, queue badge), the shared empty state, and the shared
pagination controls. Render the detail page with `JobDetailView` and a
single Retry action, and the not-found view with the shared 404 layout.
Every element now sets a color for both themes, and the layout follows
the responsive behavior of the Jobs pane.

The empty state now says how to enable the dead letter queue, because
the pane appears for every River Pro install whether or not the queue
is on. Add stories for both components.
No route defines a `notFoundComponent`, so the router falls back to its
bare "Not Found" text. That text has no color and no top navigation, so
the page looks blank in the light theme. This happens for any unknown
URL, and for `/dead-letter-jobs` on a server without dead letter
support, where the route throws `notFound()`.

Add a `notFoundComponent` to the root route that uses the shared 404
layout and links back to the job list.
A URL such as `/dead-letter-jobs?priority=5` or `?id=abc` fails search
validation, and the route shows the generic error page. The Jobs list
ignores the same values and still opens.

Validate the route's search params with a schema that drops any filter
it can't parse and keeps the rest. Edits in the search field still use
the strict schema, so a bad value typed there shows an inline error
instead of running an unfiltered query.
The list query sets `throwOnError`, so one failed background refresh
replaces the whole pane with the error page and stops polling. A short
backend restart leaves the tab broken until a reload. The Jobs list
keeps its rows and continues to poll.

Remove `throwOnError` from the list query and show the failure in an
inline alert above the rows. The loaded jobs, the search field, and
polling all stay active, so the list recovers by itself.

Add route tests for this case, for a URL with filters that can't be
parsed, and for the not-found page on a server without dead letter
support.
The test waits 300 ms of real time for a 200 ms debounce. Under load
the debounce can fire late, and then the test passes without checking
that the URL stays as restored. Advance fake timers past the debounce
instead.
The sidebar lists the Jobs search params to clear by hand, and the Jobs
route lists the same params to retain. A param added to one list and
not the other leaks filters between panes with no type error.

Export the list from the Jobs search schema and derive both uses from
it.
Jobs, Queues, and Workflows are the panes operators open every day.
Dead Letters is used less often, so list it after them.
A long job kind has no break points, so on the job detail page it runs
past the edge of the heading and pushes the action buttons out of
place. The workflow detail page already solves this for workflow names:
it cuts the name with an ellipsis, shows the full name in a tooltip,
and puts a copy button next to it.

Move that heading into a shared component and use it for the job kind
on the job detail page, which the dead letter detail page also uses.
The heading stays on one line at every width, and the copy button gives
the full kind.
The dead letters entry promises bounded database queries, but only the
number of returned rows is bounded: a filter can still read the whole
table. Remove that claim, say that the list is newest first, and add
entries for the user-visible fixes outside the new pane.
`JobDeadLetterList` orders by ID ascending and caps the result at 200
rows, so once the dead letter table holds more rows than that the most
recently dead-lettered jobs can't be reached from the UI at all, and
those are the ones an operator is most likely to be looking for.

Order both the limited subquery and the JSON aggregate by `id DESC` in
the PostgreSQL and SQLite queries so the limit keeps the newest rows.
SQLite makes no promise that `json_group_array` preserves the order of
an ordered subquery, so both drivers also sort the decoded rows by ID
descending through one shared helper, which makes the final order
deterministic and identical across databases.

The shared driver suite now inserts its rows in ascending order and
expects them back descending at every limit, and the list endpoint test
asserts that the most recently inserted job comes first.
The dead letter list filters IDs with a predicate of the form "no IDs
requested OR `id IN (subquery)`". Neither database can use that as an
index condition, so looking up a single job by ID walks the entire
table: on a 200k row table PostgreSQL scans the whole primary key and
discards every row but one with a filter, and SQLite reports a `SCAN`.
The cost grows with the size of the dead letter table even though the
request names the exact rows it wants.

Add a `JobDeadLetterListByIDs` query to both drivers in which the ID
predicate stands alone as a top-level condition, and have the drivers
choose it whenever the filters include IDs. `JobDeadLetterList` drops
its ID predicate and keeps serving every other request. On PostgreSQL
the requested IDs collapse into an array (`id = ANY (ARRAY(...))`),
which the planner evaluates once and uses as an index condition on the
primary key under both custom and generic plans, so the result doesn't
depend on how the statement's parameters are bound. On SQLite the plain
`id IN (...)` becomes a `SEARCH` on the integer primary key.

The remaining filters keep their existing form in both queries. The
shared driver suite gains cases that combine IDs with another filter
and that look up an ID with no matching row.
A dead letter request such as `?kinds=%00` passes validation and
reaches the database. The filters travel as JSON, where the NUL byte
becomes `\u0000`, and PostgreSQL refuses that in a `jsonb` cast, so the
request fails with a 500. The same applies to invalid UTF-8, which
PostgreSQL rejects in text parameters like the autocomplete `match`.
SQLite accepts all of these, so the two databases also disagree about
whether the request is valid.

Check `kinds`, `queues`, and `tags` on the list request and `match` and
`exclude` on the autocomplete request while extracting them from the
query string, and return a bad request error naming the parameter when
a value contains a NUL byte or isn't valid UTF-8. Rejecting the input
before it reaches a query makes the response a 400 on every database.
Other control characters and non-ASCII text remain valid.

Tests cover each parameter with lone and embedded NUL bytes and with
invalid UTF-8 sequences, and send requests through mounted endpoints
against PostgreSQL to assert the 400 status.
Dead letter autocomplete behaves differently depending on the database.
SQLite's `lower()` folds only ASCII letters and orders values by their
bytes, while PostgreSQL follows the collation of the database and its
columns. With a linguistic collation, a `match` of `é` also returns
`École`, and suggestions come back in dictionary order, which changes
which values survive the limit as well as how they are sorted.

Pin the PostgreSQL autocomplete query to `COLLATE "C"` for the case
folding of both the candidate and `match`, for the exclusion check, and
for the ordering before the limit and inside the aggregate. That gives
ASCII-only folding and byte ordering, the same as SQLite. Both drivers
also sort the decoded suggestions, since SQLite doesn't guarantee the
order of `json_group_array`.

The list query has a related difference: PostgreSQL renders
`finalized_at` in the session's time zone and SQLite renders it in UTC.
The shared normalization that sorts decoded jobs now also converts
`finalized_at` to UTC, so both drivers return the same representation
of the same instant. One difference remains and is documented rather
than changed: SQLite reports milliseconds where PostgreSQL reports
microseconds.

The shared driver suite gains an autocomplete group with mixed-case and
non-ASCII values. On PostgreSQL it gives the columns an ICU collation
when one is available, so the cases fail without the explicit collation
even when the test database itself uses "C".
Several dead letter behaviors have no test that would notice them
breaking. Nothing exercises the endpoints when the dead letter table is
missing. The size-limit cases use zero-valued slices such as
`make([]int64, 1001)`, which fail on element validation (an ID of zero,
an empty string) before the size limit is ever consulted, so the limits
themselves are untested. The shared driver suite never calls
autocomplete with an invalid facet or an out-of-range max, and the
not-found and unique-conflict cases assert only an error message, not
the status a client receives.

Add a test that drops the table from the test schema and asserts that
list, autocomplete, get, and retry each return the bad request error
naming the migration. Replace the size-limit cases with table-driven
validation tests whose slices hold valid elements, covering the passing
boundary and the first failing size for IDs, kinds, queues, tags, and
priorities, the per-value length limit, and the autocomplete `match`
and `exclude` limits. Each failing case asserts the specific validation
message, so a case can't pass for the wrong reason again.

Extend the shared driver suite with the autocomplete guards for invalid
facets and for max values outside 1 through 100, and assert that the
not-found cases are a 404 `NotFound` error and that the unique conflict
is an `APIError` with status 409.
Comment thread internal/riveruidriver/riveruidriverselect/riveruidriverselect.go Outdated
The name of the dead letter table is spelled out twice: once where the
extensions report whether dead letter jobs are available, and once in
the guard that each dead letter endpoint runs. The feature flag and the
guard have to agree, or the UI offers a page whose requests all fail.

Add `HasDeadLetterTable` alongside the endpoints, mirroring
`HasWorkflowV2Tables`, and use it for both the extension flag and the
guard so there is a single definition of what "available" means.
The dead letter list limit of 200, its default of 20, the autocomplete
limit of 100, and the facet names `job_kind` and `queue_name` appear as
bare literals in the handler and again in each driver's guards. The
copies have to change together, and nothing ties them to one another.

Define the limits as constants and the facet as a `JobDeadLetterFacet`
type in the driver interface package, and move the two drivers'
identical guards into `Validate` methods on the params so both reject
the same inputs with the same errors. The handler takes its default and
maximum from the constants and carries the typed facet through to the
driver.

Two places can't reference the constants and are checked instead.
Validation struct tags must be literals, so a comment points at the
constants and a test compares the tags against them. The queries embed
the job kind facet as a literal, so a test in each driver asserts that
the generated query still uses the constant's value.
The dead letter list response embeds the driver's result struct and
shadows its `ID` field to serialize it as a string. The driver's JSON
tags exist to decode the payload its query returns, so embedding them
makes an internal detail the definition of the public API: renaming a
key in the query would silently rename it in the response.

Declare the response item as its own struct with its own JSON tags and
map the driver result onto it. The wire format is unchanged: `id` as a
string, then `attempt`, `finalized_at`, `kind`, `max_attempts`, and
`queue`. A serialization test pins that exact output, including a null
`finalized_at`.
@bgentry
bgentry force-pushed the bg-dead-letter-ui-fixes branch from 4b8b5e5 to aa2a8f3 Compare October 24, 2026 19:00
@bgentry
bgentry marked this pull request as ready for review October 24, 2026 19:09
@bgentry
bgentry merged commit 0120686 into brandur-dead-letters-in-ui Oct 24, 2026
13 checks passed
@bgentry
bgentry deleted the bg-dead-letter-ui-fixes branch October 24, 2026 19:16
brandur added a commit that referenced this pull request Oct 24, 2026
* Bring dead letter jobs to River UI

We got a request in #706 that dead letter jobs should be in the UI, and
that is indeed very true. It's often quite convenient to pull up a UI
and retry a job from there instead of trying to write a one-off script
to do it programmatically.

Here, dead letters are added as a new entry on the left sidebar under
"Queues". I also thought about it integrating it directly into the
"Jobs" pane, but we'd need some new view in there as well, and I don't
think it's any better of a fit.

Like jobs, dead letters get a "filter" textbox at the top that lets you
select a variety of attributes to filter the view underneath in pursuit
of finding specific job(s).

The new pane is shown only for the Pro binary: in OSS, navigation is
hidden, access to the pane is blocked, and the API endpoints aren't
mounted.

Fixes #706.

* fix dead letter browsing and queries (#712)

Reuse the Jobs layouts for consistent dead letter list and detail pages,
with resilient filters, refresh errors, and not-found states. Show newest
dead letters first, use primary-key ID lookups, and align query validation
and autocomplete across Postgres and SQLite. Cancel superseded search
suggestions and share common UI components.

* Disable automatic refresh on dead letter jobs

There's some concern around query performance for dead letter jobs. Just
so that we can get this shipped out without too much ado, here we
disable automatic refreshes for dead letter jobs. Automatic refresh for
normal jobs continues as normal. Users can enable dead letter refresh
with the toggle (it's just off by default).

---------

Co-authored-by: Blake Gentry <bgentry@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants