Repository navigation
Conversation
The reachability log puts timely pointstamp count deltas into the differential diff. Progress updates arrive out of order across workers, so the accumulated multiplicity of a row can be negative, and under steady traffic it usually is. Operators that require positive multiplicities fail on it: a distinct reports a non-positive multiplicity, and a fast-path peek reports retractions for a row that does not exist. Each failure also logs a customer-data error. Remove PUBLIC_SELECT from mz_dataflow_operator_reachability_raw so that users cannot query it directly. Builtin views read it as the owner. The views over it aggregate with an accumulable count(*), which accepts negative totals. Projection pushdown removes that aggregate when a query ignores the count column, for example count(*) over the view, and the remaining reduce renders as a distinct over the raw log. Add a HAVING clause on the aggregate to mz_dataflow_operator_reachability_per_worker so that the aggregate survives. Groups with a zero total emit no row, so the filter does not change results. Extend rbac_mcp_agent.slt to assert that an unprivileged role is denied on the raw relation and can still read the views. Add introspection/reachability.slt, which pins the accumulable count(*) in the plans of count-free queries on both views. The cancellation testdrive now checks the raw relation over the mz_system connection. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The leftover check on mz_dataflow_operator_reachability_raw ran over the mz_system connection, which does not retry, and failed when the cancelled dataflow's reachability rows were not yet retracted. Expose the log through a view owned by mz_system and granted to the test user, so the check is a retried fast-path query like its siblings. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ggevay
left a comment
ggevay
left a comment
There was a problem hiding this comment.
LGTM, a few comments on the description and comment wording.
The Verification bullet for divergent-dataflow-cancellation.td still says it checks the raw relation over the mz_system connection. Since the second commit it reads through a view owned by mz_system. The PR body becomes the squash commit message, so this would land on main as written.
| // The diff carries a timely pointstamp count, which can accumulate | ||
| // negative, so this is not a valid relation for direct queries. Builtin | ||
| // views read it as the owner. | ||
| access: vec![], |
There was a problem hiding this comment.
Two groups can still query this relation: superusers, which in Cloud includes every organization admin (User::kind() maps is_external_admin() to Superuser), and everyone on deployments with RBAC checks off (a Materialize resource created without enableRbac: true, which is the CRD default, and the emulator). An admin running the incident's SELECT DISTINCT * still gets the error and still trips the alert. Could the PR description say which half covers whom (the HAVING clause fixes the views everywhere, the privilege change covers non-admins where RBAC checks are on)?
| # which can accumulate negative, so direct queries are limited to the system | ||
| # user. The views over it aggregate with an accumulable reduce and stay public. |
There was a problem hiding this comment.
nit: superusers, not only the system user. In Cloud that includes organization admins.
| $ postgres-execute connection=postgres://mz_system:materialize@${testdrive.materialize-internal-sql-addr} | ||
| ALTER SYSTEM SET enable_introspection_subscribes = false | ||
|
|
||
| # The raw reachability log is readable only by the system user. A view owned by |
There was a problem hiding this comment.
nit: superusers, not only the system user. In Cloud that includes organization admins.
Motivation
mz_dataflow_operator_reachability_rawcarries timely pointstamp count deltas in its diff. Progress updates arrive out of order across workers, so accumulations go negative, and on a multi-worker replica under steady traffic they nearly always are. Any distinct or fast-path peek over the relation then fails with an internal error and logs a customer-data error, which pages as potential data corruption. The public views fail the same way whenever a query ignores theircountcolumn, because projection pushdown removes the aggregate and leaves a distinct over the raw log.Related: DB-23, database-issues#5387.
Description
mz_dataflow_operator_reachability_rawdropsPUBLIC_SELECT. Builtin views keep reading it as the owner, following themz_catalog_rawpattern.mz_dataflow_operator_reachability_per_workergainsHAVING count(*) <> 0, so its accumulablecount(*)survives projection pushdown. It accepts negative totals, and groups with a zero total emit no row. The negative counts stay visible in the views'countcolumn.This mitigates the errors without changing how the log encodes counts.
Verification
rbac_mcp_agent.sltasserts that an unprivileged role is denied on the raw relation and can read both views.introspection/reachability.sltpins the accumulablecount(*)in the plans of count-free queries on both views. It fails without theHAVINGclause.divergent-dataflow-cancellation.tdchecks the raw relation over themz_systemconnection.Release notes
This release will restrict direct queries on
mz_introspection.mz_dataflow_operator_reachability_rawto superusers. Querymz_dataflow_operator_reachabilityormz_dataflow_operator_reachability_per_workerinstead.🤖 Generated with Claude Code