Decode \uXXXX and \UXXXXXXXX in string literals; add <<'…' and <<"…"

[rjose]

Problem

Models write Unicode escapes in Forthic strings: \u2014 for a dash, \u0027 for an apostrophe, \uD83D\uDE00 for an emoji, and the Python form \U0001F600. Forthic kept each one as text, so a user saw \u2014 in a reply. The bigb host repairs some of them before it parses the program (BIG-8657, bigb#7596), but that repair cannot decode ', " or \: it cannot know if a literal will be parsed again as code, where a decoded quote can close the inner literal. It also covers only one host path.

Also, a marked string had to be triple-quoted. <<"streamed reply" tokenized as unknown words, so a reply in a regular string could not stream (BIG-8660 needs it).

Options and choices

• Where to decode: in the host, before parsing, or in the tokenizer. Tokenizer. It decodes once for each literal it parses, and decoded text is content: it is never scanned again for a delimiter. In a code string, each level decodes its own escapes, as in any language. Every Forthic caller gets it.
• Long form: \UXXXXXXXX (Python; about 5 prod programs each week), \u{…} (JavaScript), both, or neither. \U only (Rino).
• Lone surrogate: keep the escape text, decode the lone UTF-16 unit, U+FFFD, or an error. Keep the text (Rino). This is the rule for every unknown escape, and a string never holds invalid UTF-16.
• <<'…': supported, next to <<"…" (Rino). Same escapes and holdback as the triple forms.
• Streaming holdback: scan the tail of the open string in get_string_value() for a partial escape, or stop gathering at an escape that the input ends inside. Stop gathering. The open string then never holds undecided text, one rule covers both widths, and the old backslash holdback goes away. Only the triple-quote delimiter holdback stays. One visible effect: a settled \\ at the end of a chunk now reaches the sink one chunk earlier than in 0.21.0, which held every trailing backslash.
• Token end_pos: it was start_pos plus the decoded length, so it fell short after every escape (\n by one, a surrogate pair by ten). A closed string now records where its content ends in the source.

Key changes

• src/forthic/tokenizer.ts
  • read_escape(index) returns the escape's text and length, null (the backslash is content), or PENDING (the input ends first: \, a partial \u/\U, or a high surrogate that a low surrogate escape can still follow). read_hex() reads the digits.
  • gather_escape() serves both gather loops. On PENDING the loop stops, so a streaming chunk returns the open string without the undecided escape, and run() throws UnterminatedStringError as before.
  • is_string_redirect_start() accepts << glued to any quote. transition_from_GATHER_STRING takes is_string_redirect, sets string_redirect_open, and marks its token. It no longer advances before the escape check, like the triple-quote loop.
  • get_string_value() holds back only a trailing run of the triple-quote delimiter.
  • Both string loops set token_end_pos at the closing delimiter, and get_token_location() uses it for end_pos. The error underline in errors.ts uses end_pos.

Breaking changes

• A literal that is not raw now decodes \uXXXX and \UXXXXXXXX. A program that needs the text \u2014 writes \\u2014 or r'\u2014'.
• << glued to a quote is now a marked string, not a word.
• Ships in 0.22 with U1 (Restore every binding through export_state / import_state #57, Land U1.2-U1.4 on main: parallel MAP, error recovery, marked-string streaming #61).

Tests

• src/forthic/tests/unit/core/tokenizer.test.ts: "Unicode escapes (\uXXXX, \UXXXXXXXX)": every non-raw width and marked strings, hex case, surrogate pairs, lone surrogates, incomplete escapes, \U limits, \u{…} and raw literals unchanged, \\u, decoded quotes are content. "Marked single-width strings": both quotes, escapes, << "x" with a space, unterminated.
• src/forthic/tests/unit/core/tokenizer.test.ts: "String token positions after escapes": end_pos after escapes and without them.
• src/forthic/tests/integration/string_escapes.test.ts: "Unicode escapes in code strings": RUN and MAP code strings; a decoded quote in the outer literal closes the inner one (UnknownWordError).
• src/forthic/tests/integration/string_redirect.test.ts: "escapes stream as their decoded text": four programs, every cut; the sink text is always a prefix of the final string and equals it at the end.

Why other files changed

• CHANGELOG.md: two breaking entries under [Unreleased], with a note that regex and JSON want a raw literal.
• scripts/generate-docs.ts: the prompt template. It names the new escapes and has a short "Unicode escapes" section. Its two code → expected claims run in npm run smoke.
• docs/forthic-prompt.md: regenerated from the template.

Limits

• An error inside code that RUN or MAP runs reports a position in the decoded code text. After an escape in the outer literal, that position does not match the source. \n has the same limit in 0.21.0.

Closes BIG-8659.

🤖 Generated with Claude Code