Skip to content

Add streaming JSON adapters and regression contracts for all engines - #5

Open
Easton97-Jens wants to merge 28 commits into
digitalwave:mainfrom
Easton97-Jens:codex/streaming-parsers-and-regression-suite
Open

Easton97-Jens wants to merge 28 commits into
digitalwave:mainfrom
Easton97-Jens:codex/streaming-parsers-and-regression-suite

Conversation

@Easton97-Jens

Copy link
Copy Markdown
Contributor

jsoncons previously parsed and discarded a complete DOM without producing the argument output needed for a ModSecurity comparison. Several adapters used fixed file buffers, inconsistent path/limit handling, unsafe numerical conversions or incomplete validation; the old runner could report success when output comparison was skipped.

This PR brings the complete reviewed implementation from Easton97-Jens/jsonbench.

Relationship to #4: this consolidated PR supersedes the original jsoncons SAX PR. Its head dd603ee is an ancestor of the submitted 715ae64 branch. All 12 original changed files are included, with 29 additional files (41 total); nine original files receive further corrections, while the public jsoncons header, shell entry point and RSS helper are unchanged. Original valid/invalid tests remain, and repaired YAJL defects now require equality. Once this replacement exists, #4 can be closed as superseded; these are not independent patches.

  • jsoncons uses its public streaming reader/visitor API, without constructing a DOM. YAJL streams and finalizes bounded chunks; RapidJSON and nlohmann use public SAX streams. Paths, scalar counts, silent validation, Unicode/NUL handling, errors and resource limits are checked. Numerical text is preserved where native events permit it; unsupported ranges fail explicitly instead of rounding silently.
  • JSON-C, Jansson, cJSON, JsonCpp and yyjson retain native DOM grammar validation with checked native ranges and resource handling. simdjson traverses public OnDemand values with an independent 512-container recursion ceiling. DOM and padded whole-input memory remain explicit. Thread numeric locales are restored on every exit path.
  • All available engines run the same deterministic CLI regression matrix with explicit golden names and values. Tests cover valid/invalid JSON, Unicode and malformed UTF-8, numerical boundaries, root/nested arrays, repeated keys, empty containers, depth/argument/string limits, large documents, silence, reuse and read/output errors. Missing output, crashes, timeouts and required failures cannot become compatibility passes. Individual-engine selection and optional dependencies remain supported.
  • Six API/path harnesses run through make check, with private fixtures and shared RAII/C allocator helpers. Identical native C++ emission and DOM initialization are shared. The CLI validates iteration counts before allocation; Sonar reporting matches the actual C++17 build. No analysis rules or required assertions are excluded.

Validation on the exact submitted source tree:

  • ./autogen.sh, ./configure and make succeed with ten engines; make check passes all six programs and 5,779 assertions.
  • tests/run_tests.sh JSONCONS: 275 passed, 0 failed, 0 skipped, 2 separately checked spelling differences, exit 0.
  • tests/run_tests.sh: 2,409 passed, 44 failed, 0 skipped, 2 differences, exit 1. These required failures remain visible for native exponent/integer ranges, duplicate or NUL keys, and cJSON decoded NUL limitations; they are documented per engine in docs/adapter-stack.md. jsoncons, YAJL and yyjson have no required CLI failures.
  • ASan/UBSan with leak detection reproduces the matrix. All API suites pass; final edited JsonCpp code was rechecked with 331 API assertions and its 240-pass/2-duplicate-failure CLI subset. Independent sanitizer reports and leftover fixture directories: zero. All 63 final source hashes match the private tested tree.
  • Glaze could not be enabled: configure found no usable glaze/glaze.hpp with its tested modern C++ flags. Its runtime is not claimed as tested; optional handling remains intact. The source archive includes all new helpers.

Compatibility and analysis limits:

jsoncons normalizes integer -0 to 0 and removes exponent +; nlohmann also normalizes integer -0. Those spellings have explicit expected results. YAJL path/NUL/root-number-at-EOF defects are corrected rather than reproduced. Native representation failures remain assertions. Four const advisories on rj_parse_buffer, rj_parse_file, rj_parser_cleanup and nl_parser_cleanup are retained by explicit decision to preserve exact public C function types and typed callbacks; they are documented, not suppressed or marked accepted.

Fork PRs Easton97-Jens#11, Easton97-Jens#12 and Easton97-Jens#13 have been merged and passed their final Sonar gates, all with zero hotspots and zero new-code duplication. #11 reports zero new issues; #12/#13 report the same four retained const advisories. The subsequent fork-main analysis also passes but reports 11 new issues and supplies no inline annotations through GitHub. A complete server issue inventory is therefore not certified. Cloud line coverage is not imported and reports 0%; assertion counts do not establish a coverage percentage.

See tests/README.md for reproducible checks, docs/jsoncons-sax.md for SAX details, and docs/adapter-stack.md for per-engine limitations and future ModSecurity integration recommendations. Its measured DOM/SAX/YAJL report is explicitly the earlier e6fe3a3 snapshot, before Sonar corrections; no unmeasured performance claim is made. No other engines are removed, no JSON Schema dependency is added, and ModSecurity itself is unchanged.

Reference: owasp-modsecurity/ModSecurity#3308 (comment)

Überschneidung mit Upstream-PR #4

Geprüft: #4 und der veröffentlichte Gesamtbranch Easton97-Jens:codex/streaming-parsers-and-regression-suite.

Der Gesamtbranch ersetzt #4 inhaltlich. Empfehlung: einen Gesamt-PR als Nachfolger anlegen und #4 erst danach als ersetzt schließen. Alternativ #4 zuerst mergen und anschließend nur dessen Ergänzungen einreichen. Beide Änderungen nicht als unabhängige Patches behandeln.

Der Gesamtbranch ist veröffentlicht; ein zusätzlicher Upstream-PR konnte noch nicht angelegt werden. GitHub antwortet auf die Erstellung mit HTTP 403, Resource not accessible by integration. #4 bleibt daher offen. Die vollständige vorbereitete PR-Beschreibung steht in jsonbench-upstream-pr.md.

jens Steinbacher and others added 28 commits October 13, 2026 12:42
Use streaming jsoncons/YAJL and fix CLI and Sonar findings
Stream RapidJSON and nlohmann SAX with checked RAII and limits
…rage

Validate native DOM and OnDemand adapters and address Sonar findings
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant