[mgree]

Motivation

Closes https://linear.app/materializeinc/issue/SQL-717/safe-cast-try-cast-function-for-handling-bad-data-on-ingest.

Description

A TRY_CAST(<expr> AS <type>) evaluates <expr> once and attempts to cast it to <type>. While casts can fail (e.g., not every string is an integer!), TRY_CAST will detect failures and produce NULL instead.

Advice for reviewers

There are just a few core design decisions:

• We don't support casts between weird internals (e.g., regclass, mz_aclitem) that are implemented in SQL.
• A failure anywhere in a nested structure yields a global NULL.
• We just build one new UnaryFunc wrapper, with some mild shenanigans in HIR to deconstruct and reconstruct casts.

I've maintained semantic commits, so it will be notably easier to go commit-by-commit here.

As follow-up work, we should extend some testing artifacts:

Follow-up PRs we should do:

1. output-consistency tests
2. parallel-workload (try_cast({} as T) templates for a few fallible targets in expression.py)
3. RQG
4. SQLancer and SQLancer++

I did not implement DEFAULT ... ON CONVERSION ERROR, which is a nice feature to have. It would not be a hard extension to make.

Verification

Extensive testing for each cast: good values stay good, bad values turn into NULL. This testing is in SLTs, proptests, and concrete unit tests with chosen values.

[def-]

QA LLM Review

1. MEDIUM -- TRY_CAST over a ROW(...) literal yields a record with NULL fields instead of a whole-value NULL

src/sql/src/plan/typeconv.rs:1561

Threading mode into the LiteralRecord arm of plan_coerce_with_failure_mode wraps each field coercion in its own TryCast, and those coercions sit inside the RecordCreate -- outside the whole-value record cast. So TRY_CAST(ROW('100000','a') AS rec2) evaluates to (,a) rather than NULL, which is exactly the "value with NULL elements" that doc/user/content/sql/functions/cast.md and the header of test/sqllogictest/try_cast.slt promise cannot happen.

Details

The planned HIR for CREATE TYPE rec2 AS (a int2, b text) shows the split directly. With a typed field, the conversion is inside the wrapper and the whole value goes NULL (this is the case try_cast.slt tests):

TryCast(CastRecord1ToRecord2 { cast_exprs: [CastInt32ToInt16(#0), #0] })(
  RecordCreate(100000, 'a'))                       -- TRY_CAST(ROW(100000,'a') AS rec2)  => NULL

With an untyped string literal, the conversion has already happened during coercion, and the record cast it is nested in is an identity:

TryCast(CastRecord1ToRecord2 { cast_exprs: [#0, #0] })(
  RecordCreate(TryCast(CastStringToInt16)('100000'), 'a'))   -- => (,a)

Two spellings of the same logical value therefore disagree. Leaving coercions as vec![SqlScalarType::String; arity] under NullFallback (i.e. not pushing the target field types down, the same reasoning as the Expr::Array/List/Map carve-out in query.rs) puts every field conversion back inside CastRecord1ToRecord2, where the wrapper covers it.

2. LOW -- TRY_CAST cannot convert a MAP[...] constructor that CAST converts fine

src/sql/src/plan/query.rs:4344

Dropping the target-type hint for MAP constructors under NullFallback types the literal by its own values, and there is no map-to-map entry in VALID_CASTS to convert it afterwards, so any map constructor whose natural value type differs from the target fails to plan. CAST(MAP['k' => '1'] AS map[text=>int4]) plans; TRY_CAST(MAP['k' => '1'] AS map[text=>int4]) fails with TRY_CAST does not support casting from map[text=>text] to map[text=>integer].

Details

ARRAY/LIST are unaffected because CastArrayToArray/CastList1ToList2 exist to do the elementwise conversion after the fact; MAP has no equivalent. The new docs state that the whole-value-NULL rule "also applies to ARRAY, LIST, and MAP constructors written directly inside TRY_CAST", which is not true for MAP -- either the hint needs to stay for maps (accepting element-level strictness, as the empty-constructor case already does) or the docs and the error message should say the form is unsupported.

[antiguru]

Looks good overall, just some questions:

• Can we eliminate casts that cannot fail, or downgrade them to regular casts in MSE::reduce?
• Could we make it an invariant that the inner expression of a try_cast is always could_error? Otherwise, there's not much point in try casting.

[mgree]

Looks good overall, just some questions:

• Can we eliminate casts that cannot fail, or downgrade them to regular casts in MSE::reduce?

We already turn TRY_CAST of a .could_error() == false cast into a regular cast when generating casts in HIR.

• Could we make it an invariant that the inner expression of a try_cast is always could_error? Otherwise, there's not much point in try casting.

We could make it an invariant---we already generate things this way, but I'm not 100% certain that nobody messes with things on the way to LIR. Worth adding assertions?

[ggevay]

LGTM, just minor comments.

(While rebasing, please also try dropping the "char" skip in the cast proptest in catalog.rs, and with it SqlScalarType::contains (its only caller), and update the DB-201 note in the try_cast.slt header, since those casts no longer panic.)

[ggevay]

The Parameter arm below keeps CAST's typing under NullFallback: an untyped parameter takes the target type, so a bad bound value fails at bind instead of becoming NULL. With psycopg, SELECT TRY_CAST(%s AS int) with 'abc' fails with unable to decode parameter: invalid input syntax for type integer. That seems right, since the parameter's type applies to the whole statement, but the docs could say so and suggest TRY_CAST($1::text AS int), which returns NULL. Through this arm, TRY_CAST(ROW($1, 'a') AS rec2) types $1 as text instead, so a parameter inside a ROW behaves differently from a bare one.

[ggevay]

This also turns EvalError::Internal into NULL, which hides planner bugs: before #38899, TRY_CAST('7'::"char" AS int8) returned NULL where CAST reported internal error: invalid input type. Returning the error would break could_error() == false, but a soft_panic_no_log! on Internal before returning NULL would keep such bugs visible in CI without logging per row in production. No cast impl returns Internal for bad input, so it can't fire on user data.

[ggevay]

This reads exactly like InvalidCast with name = "TRY_CAST": TRY_CAST(1 AS int4range) (no such cast) and TRY_CAST('t' AS regclass) (the cast exists, but TRY_CAST can't wrap it) both print TRY_CAST does not support casting from .... Saying that the cast exists but only CAST supports it would tell users what to do.

[ggevay]

The flag defaults off and its error says private preview, so per doc/developer/project-lifecycle.md this should be {{< private-preview />}} until the flag is on everywhere. Also, version-added: v26.43 in sql_funcs.yml is stale: this isn't in v26.43, v26.44 or v26.45. (Sorry for the late review!)

[mgree]

I went ahead and turned this on by default, as the flag is an escape hatch. (Code for TRY_CAST should only be triggered by its syntax, anyway.)

[ggevay]

CAST has no map-to-map cast either, so this isn't a TRY_CAST exception (also "casts" -> "cast"). An actual exception that the list doesn't mention is the dropped type hint: CAST(ARRAY[s, 2] AS int4[]) plans, but TRY_CAST(ARRAY[s, 2] AS int4[]) fails with ARRAY types text and integer cannot be matched, so constructor elements must agree on a type on their own.

[ggevay]

Nit: nothing pins that an infallible TRY_CAST stays unwrapped and non-nullable. The structural test in catalog.rs uses UnaryFunc::try_cast as its own oracle, so an always-wrap mutant passes. The assertion you offered antiguru would also catch it if it runs before constant folding, since the matrix's infallible rows are constants. Otherwise, a column like TRY_CAST(s = '' AS int4) in this MV, expecting nullable = false, would.

[antiguru]

Posted by Claude Code on behalf of @antiguru. This review is based on reading the diff. I did not build or run anything.

On TryCast wrapping another function

TryCast { inner: Box<UnaryFunc<E>> } wraps a function, not a scalar expression, so it never captures an argument expression. It holds expressions only through the payloads of casts like CastArrayToArray, CastList1ToList2, and CastRecord1ToRecord2, which already exist and already go through UnaryFunc<E>. What the PR adds is a recursive type (UnaryFunc contains TryCast contains UnaryFunc) plus the FirstColumn shim, which works around the &impl Eval signature of LazyUnaryFunc::eval. The alternatives look worse:

• An error-swallowing try(expr) at the expression level would also catch errors raised by the argument, and this PR deliberately lets those propagate.
• An error-mode flag on every fallible cast would multiply the variants and the registry entries.
• A TryCast { cast_expr: E } evaluated against column 0, like CastList1ToList2, would remove the HIR stage extraction. It would also nest more expressions inside functions, and properties such as propagates_nulls and the output type would have to be derived from E instead of read off the wrapped function.

Wrapping a function is the narrowest of these options, so I think it is justified. The cost is that the optimizer can no longer see through the cast: inverse goes away, the wrapper is never monotone or eliminable, and rules that match on specific cast variants won't fire. That costs optimizations but not correctness.

Blocking

1. enable_try_cast defaults to true (src/sql/src/session/vars/definitions.rs). Our convention is that a new flag defaults off in production and on in tests, and mzcompose/__init__.py already turns it on for tests. Once users create views that use TRY_CAST, the expressions are stored in the catalog and in LIR, so turning the feature off later is no longer possible. As written, the flag gates nothing.
2. Empty collection literals no longer plan (plan_cast in src/sql/src/plan/query.rs). Under NullFallback, ARRAY and LIST literals are planned without the type hint, so plan_array bails with "cannot determine type of empty array". As a result, TRY_CAST(ARRAY[] AS int4[]) and TRY_CAST(LIST[] AS int4 list) fail where CAST succeeds. An empty literal has no element cast that could fail, so it can keep the hint. MAP literals are rejected outright, the empty MAP[] included, and the docs should say so. No SLT covers either case.

Strong suggestions

1. Make TryCast.inner private. As a pub field, it lets anyone wrap an infallible function or another TryCast, and the try_cast[infallible] registry sample exists only because the type allows that. Keeping UnaryFunc::try_cast as the only constructor would enforce the invariant. Related: output_sql_type always forces the output to be nullable, even when the inner function cannot error and introduces_nulls is false. These two properties should agree.
2. Keep one CastFailureMode. It is defined in both sql-parser and mz_expr, but nothing in mz_expr uses it. It is a planner concept, so mz_sql could use the parser's enum (or own one in typeconv) and drop the mz_expr copy and its re-export.
3. Document the template requirement for stage extraction. extract_unary_stages_for_try_cast runs each template on a Datum::Dummy sentinel and pattern-matches the result into a chain of unary calls. A template that produces any other shape makes TRY_CAST fail with "cannot fall back to NULL", even when the strict cast cannot error. The test in catalog.rs that covers every type pair pins this down today, but the doc comment on CastTemplate::new should state the requirement.
4. Add an EXPLAIN test to the SLT. It would pin how try_cast[...] renders and show the per-stage wrapping on the path that routes through text, for example text to int8 to int4.
5. Move the new adapter tests to their own file. They add about 370 lines to src/adapter/src/catalog.rs, which is already 3,600 lines long. The PR also adds the proptest and mz-repr/proptest dev-dependencies to the adapter crate.

Nits

• The TryCastUnsupported message is "…; use CAST instead". The suggestion belongs in a hint, and the primary message should not use a semicolon to join two clauses.
• UnaryFunc::try_cast only wraps conditionally, which the name doesn't convey. A name like try_cast_if_fallible would.
• soft_panic_no_log turns EvalError::Internal into NULL in production without logging. That seems to be the intent, but it leaves no trace when it happens.

The snapshot and registry updates look correct. Adding a variant needs no LIR_VERSION bump.

[antiguru]

Posted by Claude Code on behalf of @antiguru.

This follows up on the FirstColumn shim in try_cast.rs.

An eager-only TryCast would not remove the shim without losing coverage. Several fallible casts implement LazyUnaryFunc directly: CastStringToArray, CastStringToList, CastStringToMap, CastStringToRange, CastStringToInt2Vector, CastArrayToArray, CastList1ToList2, CastRecord1ToRecord2, CastArrayToJsonb, and CastListToJsonb. They are lazy only because they carry a cast_expr: E and need temp_storage. Restricting TryCast to eager functions would reject TRY_CAST('{1,x}' AS int[]), element failures in containers, and text to range, and parsing malformed text is the ingest use case behind this feature.

Every LazyUnaryFunc::eval in the tree first evaluates its argument with let a = a.eval(datums, temp_storage)? and does not use datums after that. The regexp functions do the same under the name haystack. None of them is lazy in the argument, so the trait could take the evaluated datum:

fn eval_datum<'a>(&'a self, a: Datum<'a>, temp_storage: &'a RowArena)
    -> Result<Datum<'a>, EvalError>;

With this signature, the derived UnaryFunc::eval evaluates a and then calls eval_datum, and the blanket impl for EagerUnaryFunc maps onto it unchanged. TryCast can then call self.inner.eval_datum(a, temp_storage) directly, which removes FirstColumn and the fake one-column row. The cost is a mechanical change to about 20 lazy impls, which could land as a separate preparatory PR. The change also makes argument strictness part of the trait contract: all unary functions satisfy it today, but nothing enforces it yet.

[ggevay]

(v26.45 was cut on 10-02, so this is now v26.46 at the earliest.)

[ggevay]

With the "char" skip gone from the proptest, this has no callers left except its own test, so both can go.

[ggevay]

On the two blockers in antiguru's review:

• Defaulting enable_try_cast on seems fine to me: turning it off later still blocks new uses, while existing views, indexes and MVs with TRY_CAST keep working after a restart. (The exception is an INSERT relying on a column DEFAULT that uses TRY_CAST, which fails while the flag is off.)
• (TRY_CAST(ARRAY[] AS int4[]) failing is pinned by try_cast.slt:112-113 and documented. But keeping the type hint for empty constructors, as suggested, seems right, since no element cast can fail. TRY_CAST(LIST[] AS int4 list) has no test.)

Edit: Opened a PR for adjusting AGENTS.md's feature flag guidelines: #39718

[mgree]

fn eval_datum<'a>(&'a self, a: Datum<'a>, temp_storage: &'a RowArena)
    -> Result<Datum<'a>, EvalError>;

With this signature, the derived UnaryFunc::eval evaluates a and then calls eval_datum, and the blanket impl for EagerUnaryFunc maps onto it unchanged. TryCast can then call self.inner.eval_datum(a, temp_storage) directly, which removes FirstColumn and the fake one-column row. The cost is a mechanical change to about 20 lazy impls, which could land as a separate preparatory PR. The change also makes argument strictness part of the trait contract: all unary functions satisfy it today, but nothing enforces it yet.

Let's do this after your scalar func rewrite lands.