Skip to content

Preserve bytes containing backslash-u sequences in JSON serialization - #624

Open
rioyu123 wants to merge 1 commit into
google:mainfrom
rioyu123:codex/fiddle-bytes-serialization
Open

rioyu123 wants to merge 1 commit into
google:mainfrom
rioyu123:codex/fiddle-bytes-serialization

Conversation

@rioyu123

Copy link
Copy Markdown

Serializing a bytes value converted it to a string with raw_unicode_escape, which treats literal \uXXXX / \UXXXXXXXX byte sequences as escapes. As a result, br'\u00ff' came back as b'\xff', and some invalid sequences such as br'\u123' or br'\uZZZZ' raised UnicodeDecodeError during dump_json.

Serialization now uses latin-1, which maps each byte to one character. The reader is unchanged and still uses raw_unicode_escape, so the schema stays the same and existing documents load as before. For bytes containing backslash-u sequences, the stored string now keeps those characters as written (for example br'\u1234' is stored as the six characters \u1234). Both the old and new forms read back as the same bytes.

Adds regression coverage for:

  • round-trips through Config → dump_json → load_json → build for empty, ASCII, all 256 byte values, and five escape-like inputs;
  • loading an old-format document containing U+1234 and checking that it still reads as br'\u1234'.

Testing (local, Linux, Python 3.12):

  • pytest --ignore=fiddle/_src/extensions/seqio_test.py -q fiddle: 1117 passed, 4 skipped, 330 subtests passed.
  • pytype -j 4 fiddle/_src/experimental/serialization.py: no errors.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant