EBNF as source? #333

Activity

1. tinovyatkin commented on Aug 11, 2026

   

   tinovyatkin

   on Aug 11, 2026

   ContributorAuthor

   More actions

   Feasibility verdict

   Generic .ebnf input with automatic dialect detection and fully heuristic EBNF-to-ANTLR conversion is not feasible as a reliable production feature. A strict importer for one named EBNF dialect/profile is feasible, but it is a substantial frontend/transpiler feature rather than a syntax rewrite.

   I would not add bare .ebnf as a peer of .g4 yet, and I do not see evidence for claiming coverage of "most" or a "significant part" of grammars in the wild. A defensible first scope would name the dialect explicitly, reject unsupported semantics, and make the generated .g4 inspectable.

   What the linked project demonstrates

   I checked AleksanderKruk/ebnf2antlr at 4517a19.

   • Its input grammar accepts only an optional numeric production label, a restricted NAME ::= expression form, |, -, postfix */+/?, parentheses, and quoted literals. Names cannot contain _ or -; W3C character classes and #x... code points are not supported.
   • The converter takes parse-tree text and performs four textual rewrites: remove [n], replace ::= with :, append ;, and change double quotes to single quotes. It does not derive a lexer/parser partition, whitespace policy, keyword priority, start rule, case-sensitivity policy, source map, or ANTLR representation for language difference.
   • The only test is assertEquals(1, 1). The repository also has no declared license, so its code cannot currently be reused safely.

   I built it and exercised it:

   • Its own 218-production XQuery sample converts to a file, but both official ANTLR 4.13.2 and the current antlr4-rust-gen reject the result at W3C A - B expressions. The sample ends before XQuery's terminal productions, and every retained production name is uppercase; ANTLR treats uppercase rule names as lexer rules, so this does not produce an XQuery parser.
   • The current Fluent EBNF produces many syntax diagnostics and malformed output. The converter still exits successfully and writes the file.
   • The current Vue I18n EBNF fails at its opening (* ... *) comment and emits only the ANTLR grammar header.
   • A small diagram-oriented grammar converts syntactically, but its lexer-like productions are intentionally comments, so ANTLR correctly reports undefined rules. This is common for EBNF used as documentation rather than as a complete executable grammar.

   This project is a useful proof that one small notation subset can be parsed and rewritten. It is not evidence that general EBNF conversion is solved.

   Why heuristics are insufficient

   "EBNF" is a family of notations and semantic conventions, not one interchange format:

   • W3C EBNF includes character code points, character classes, and language difference A - B.
   • RFC 5234 ABNF instead has /, incremental =/, prefix repetition such as 1*2element, case-insensitive strings by default, core rules, and prose values.
   • A small, non-random GitHub sample of files named .ebnf already includes regex literals and EOF (Fluent), mixed ISO/W3C notation (Vue I18n), embedded semantic actions (gocc), and preprocessor directives plus custom lookahead annotations (Oberon).

   Even after parsing the notation, the source often does not determine:

   • which productions should be lexer tokens, lexer fragments, or parser rules;
   • whitespace/comment skipping and lexical modes;
   • keyword-versus-identifier priority and contextual keywords;
   • root rule and EOF behavior;
   • case sensitivity;
   • how prose constraints, semantic predicates, indentation, or external productions are implemented;
   • what parse-tree/API shape is intended when several ANTLR rewrites recognize the same language.

   The linked XQuery case is especially instructive: the normative specification has a separate lexical structure, longest-match rules, terminal-delimitation rules, and extra-grammatical constraints required for syntactically valid sentences. The EBNF production text alone is deliberately incomplete.

   Fit in this repository

   At current main (65cdc825), the CLI advertises ROOT.g4, and the loader directly invokes the G4 frontend and enforces G4 naming/import behavior. There is a reusable-looking boundary after syntax lowering, but the internal GrammarUnit and its child nodes carry G4 SyntaxId/SourceSpan provenance throughout.

   Therefore the lowest-risk shape, if research justifies it, is a dedicated dialect frontend/transpiler that emits:

   1. auditable .g4;
   2. a source map back to the EBNF;
   3. strict, non-zero failure for unsupported or under-specified constructs;
   4. explicit options/annotations for root, lexer partition, whitespace, and case policy.

   A native second frontend feeding GrammarUnit directly would require a broader source/provenance and loader refactor. Dialect-specific rules should stay out of the generic grammar compiler either way.

   Recommended next research step

   Split the question into a corpus study and a single-profile prototype:

   1. Build a licensed corpus and classify each file by actual dialect and whether it is a complete executable grammar.
   2. Pick one explicit profile, for example W3C EBNF plus required lexical annotations. Do not auto-detect.
   3. Define unsupported constructs and fail closed.
   4. Validate generated G4 with both official ANTLR and this generator, then differential-test accepted/rejected strings against an independent implementation where one exists.

   So the verdict is no for generic heuristic .ebnf support as written; yes, conditionally, for an explicit dialect/profile after corpus evidence.