parse-bench: investigate Rust vs Go AST divergence on C# Mono fixtures #109

Summary

When running parse-bench with --ast-check, several C# fixtures produce byte-different parse trees between rust-antlr and go-antlr, even though both sides complete without error nodes. Kotlin, Java, and Trino already match under the same gate.

This blocks enabling --ast-check for the full default language set and makes Rust-vs-Go throughput numbers on those fixtures unfair (different trees ⇒ different work).

Affected fixtures

Under tools/parse-bench/fixtures/csharp/:

Observed symptom

Example from mono-codegen.cs (Rust has more children under using_directives than Go):

 Rule(compilation_unit, children=3)
-  Rule(using_directives, children=9)
+  Rule(using_directives, children=6)

Rust includes additional using directives (and later additional statements/members) that Go’s tree omits. Diffs also show extra using IKVM... / related regions on the Rust side.

Neither dump contains Err(...) nodes — both parses “succeed”, they just build different CSTs.

Likely cause (hypothesis)

C# grammars-v4 ships target-specific support bases:

• Go: csharp/v7/Go/CSharpLexerBase.go, CSharpParserBase.go
• Python: similar
• Rust: generated from .interp (+ whatever preprocessor/channel handling the runtime/lexer path implements)

The Mono fixtures use preprocessor conditionals (#if / #endif and related). Go’s CSharpLexerBase implements preprocessor directive handling; the Rust path may be:

1. not skipping the same #if regions Go skips, or
2. treating hidden/preprocessor channels differently, or
3. diverging on some other support-base behavior (not necessarily pure ATN prediction).

This needs confirmation with a minimal #if repro and side-by-side token streams / tree dumps.

How to reproduce

# Setup: ANTLR 4.13.2 jar + grammars-v4 sparse checkout including csharp/v7
# (see AGENTS.md / tools/parse-bench/README.md)

python3 tools/parse-bench/run.py   --languages csharp   --runtimes rust-antlr,go-antlr   --ast-check   --quick   --work-dir target/parse-bench-csharp-ast

Dumps land under:

target/parse-bench-csharp-ast/ast-dumps/csharp/*.rust.txt
target/parse-bench-csharp-ast/ast-dumps/csharp/*.go.txt

Investigation plan

1. Minimal repro — strip a Mono fixture to the smallest #if / using pattern that still diverges.
2. Token-stream parity — dump default-channel (and hidden) tokens from Rust vs Go for that repro; check whether Go drops preprocessor-disabled regions that Rust keeps.
3. Support-base gap — compare CSharpLexerBase (Go) preprocessor mode/#if evaluation with Rust lexer generation / channel handling for the same grammar.
4. Decide the fix surface:
   • Rust lexer/runtime channel + preprocessor support for C#, or
   • checked-in Rust support module analogous to Go/Python bases, or
   • fixture normalization (only if the divergence is fixture-specific and not a real product gap).
5. Once fixed, extend CI --ast-check languages to include csharp.

Acceptance criteria

• Root cause documented (token stream vs parse decision vs support base).
• mono-*.cs fixtures pass --ast-check against Go (byte-identical dumps, no error nodes), or fixtures are explicitly justified as non-comparable with a tracked exception.
• CI AST parity step can include csharp without special-casing.

Related

• parse-bench --ast-check (Rust/Go tree dump gate)
• Recent fix for Java precpred / > vs >> left-recursive loop-enter fast path (unrelated mechanism; C# looks preprocessor/support-base shaped)