xlang: the snake_case conversion of field names is not specified, and the runtimes do not agree #4187

Search before asking

• I had searched in the issues and found no similar issues.

Version

All

Component(s)

Other

Minimal reproduce step

Problem

The xlang specification says that field names are "converted to snake_case" (docs/specification/xlang_serialization_spec.md, lines 882 and 1732), and line 1800 refers to "the snake_case field name". It does not define the conversion. Each runtime has its own conversion, and the results are different for acronyms, an acronym followed by a digit, a leading capital letter, and non-ASCII capital letters. Rust and Python write the declared field name without a conversion.

Java, Kotlin, Scala, JS, Dart, C#, Swift, and C++ agree for lowerCamelCase names that have only single ASCII capital letters: they all write createdAt as created_at. A Go field must be exported, so Go writes created_at for CreatedAt. Rust and Python write the declared name, so they match only for a field declared as created_at.

Field name written to TypeDef/TypeMeta (apache/main at a11d257)

Every cell comes from a probe program that ran the runtime code at a11d257 and, where possible, encoded and decoded a real TypeDef/TypeMeta.

1. Go writes only exported fields, so a name that starts with a lowercase letter does not reach a Go TypeDef. The SnakeCase helper gives created_at, foo_u_r_l, x2_y, and fooÖl for these names.
2. The Rust runtime writes the raw name (probe of TypeMeta::new with FieldInfo::new), but #[derive(ForyStruct)] does not compile for a field name with an ASCII capital letter (error E0063, "missing field"). The derive converts identifiers to snake_case at object/util.rs:410 and :422.
3. fooÖl is not a valid Dart identifier.
4. Rust and Python stop with "Unsupported character for LOWER_UPPER_DIGIT_SPECIAL encoding". Java writes the name as UTF-8.

Some runtimes use a second conversion for the field sort order and the struct fingerprint, and for some inputs it gives a different result from their written name:

In C#, Swift, and C++, the second conversion gave the same result as the written name for every input.

Conversion functions

• Java: StringUtils.lowerCamelToLowerUnderscore (StringUtils.java:259), applied at TypeDefEncoder.java:277. The probe built an xlang TypeDef, wrote each field with TypeDefEncoder.writeFieldsInfo, and found the decoded names in TypeDef.getEncoded(). Kotlin and Scala use this fory-core path and were not probed separately.
• JavaScript: lowerCamelToLowerUnderscore (TypeMeta.ts:979) writes the name (TypeMeta.ts:917). toSnakeCase (TypeMeta.ts:1059) gives the sort key, the fingerprint, and the read match. The probe encoded TypeMeta.fromTypeInfo(...).toBytes() and decoded the names.
• Go: SnakeCase (go/fory/util.go:60), applied at field_spec.go:427 and written at type_def.go:509. Unexported fields are skipped at type_def.go:494. The probe ran buildTypeDef and decodeTypeDef.
• Dart: _toSnakeCase (fory_generator.dart:5739, called at :1984). The runtime writes the generated identifier unchanged (type_resolver.dart:1181). The probe ran build_runner and read TypeDef.encoded.
• C#: the generator's ToSnakeCase (ForyModelGenerator.Analysis.cs:2803, called at :1490) and the runtime's LowerCamelToLowerUnderscore (TypeMeta.cs:81, applied at :333). Both use char.IsUpper, so they convert non-ASCII capital letters. The probe compiled a [ForyStruct] class with the source generator and round-tripped TypeMeta.Encode and Decode. A type with both createdAt and CreatedAt fails the build with FORY016 ("duplicate structural field identity 'created_at'").
• Swift: lowerCamelToLowerUnderscore (TypeMeta.swift:1123, applied at :267). The probe compiled Sources/Fory and round-tripped TypeMeta.encode and decode. The macro's toSnakeCase (ForyObjectMacro.swift:3698) gives the sort order; the probe ran a verbatim copy of it, because the macro build needs swift-syntax.
• C++: to_snake_case (cpp/fory/util/string_util.h:287), applied at type_resolver.h:1183. The probe built fory_serialization, ran FieldInfoBuilder::build(), and round-tripped TypeMeta::to_bytes and from_bytes. The sort order uses normalize_field_name (type_resolver.cc:1540), which calls a second, file-local to_snake_case (type_resolver.cc:1479).
• Rust: the derive passes the raw identifier to FieldInfo::new_with_id (rust/fory-derive/src/object/field_codec.rs:201). The probe ran TypeMeta::new with FieldInfo::new and decoded the result. to_snake_case (string_util.rs:110) gives the sort order and the fingerprint.
• Python: the raw dataclass attribute (meta/typedef.py:1302), written by typedef_encoder.write_field_info (typedef_encoder.py:247), which the probe ran with a recording buffer. _to_snake_case (struct.py:1183) gives the sort order and the fingerprint. The probe did not run build_field_infos or the full encode_typedef, because pyfory does not import without its native extension.

Proposal

Define one exact conversion in the xlang specification, apply it in every runtime for the written name, the sort order, and the fingerprint, and add known-answer cases (for example, the names in the table above) to the cross-language tests. A change to the conversion changes the wire names of some fields, so the specification must also say how existing data with such names is handled.

Related

We found no existing issue for this problem. The row-format schema-evolution work in #3714 uses the Java field-name conversion for its wire names and documents the portable subset, ordinary lowerCamelCase names, until this issue is resolved.

What did you expect to see?

xlang schema should always produce the same normalized names

What did you see instead?

different languages produce different names for the same schema

Anything Else?

Issue diagnosed with Claude Opus 5.5

Are you willing to submit a PR?

• I'm willing to submit a PR!