From dd603ee24e82f1ab488e372a34c18ab304bdcc93 Mon Sep 17 00:00:00 2001 From: jens Steinbacher Date: Tue, 13 Oct 2026 12:42:59 +0000 Subject: [PATCH] Use streaming jsoncons SAX parsing and test all JSON engines --- README.md | 27 +++ docs/jsoncons-sax.md | 123 ++++++++++ docs/validation-results.md | 161 +++++++++++++ src/jsonbench.c | 42 +++- src/jsonconsparser.cpp | 399 ++++++++++++++++++++++++++++---- src/jsonconsparser.h | 9 + tests/README.md | 78 +++++++ tests/benchmark_jsoncons.py | 287 +++++++++++++++++++++++ tests/benchmark_rusage.c | 47 ++++ tests/jsoncons_api_tests.cpp | 316 +++++++++++++++++++++++++ tests/regression_tests.py | 431 +++++++++++++++++++++++++++++++++++ tests/run_tests.sh | 190 +-------------- 12 files changed, 1873 insertions(+), 237 deletions(-) create mode 100644 docs/jsoncons-sax.md create mode 100644 docs/validation-results.md create mode 100644 tests/README.md create mode 100644 tests/benchmark_jsoncons.py create mode 100644 tests/benchmark_rusage.c create mode 100644 tests/jsoncons_api_tests.cpp create mode 100755 tests/regression_tests.py diff --git a/README.md b/README.md index 6aee8dd..87fde65 100644 --- a/README.md +++ b/README.md @@ -179,6 +179,33 @@ tests/run_tests.sh # all available engines tests/run_tests.sh SIMDJSON # single engine ``` +The runner requires Python 3 (standard library only). Every selected engine +runs the same regression matrix: explicit argument output, valid/invalid JSON, +UTF-8, numeric ranges, size/limit boundaries, silent mode and repeated passes. +Failures and unsupported-limit skips are reported separately; missing expected +output is a failure. A JSONCONS-only run also compares applicable cases to YAJL +when it is installed. Other engines' known defects remain visible as failures. + +The jsoncons adapter uses streaming SAX events, strict JSON syntax and lossless +number events. See [the implementation and compatibility notes](docs/jsoncons-sax.md) +and [validation results](docs/validation-results.md). + +To exercise the jsoncons C interface separately after configuring the project +(use the same include flags as your normal build): + +```bash +g++ -std=c++17 ${CPPFLAGS:-} tests/jsoncons_api_tests.cpp src/jsonconsparser.cpp -o /tmp/jsoncons_api_tests +/tmp/jsoncons_api_tests +``` + +For memory-safety validation, add `-fsanitize=address,undefined +-fno-omit-frame-pointer` to that compile command. For repeatable DOM/SAX/YAJL +measurements, preserve a pre-change executable and run: + +```bash +python3 tests/benchmark_jsoncons.py --baseline /path/to/jsonbench-before --binary src/jsonbench +``` + --- ## TODO diff --git a/docs/jsoncons-sax.md b/docs/jsoncons-sax.md new file mode 100644 index 0000000..c6545cb --- /dev/null +++ b/docs/jsoncons-sax.md @@ -0,0 +1,123 @@ +# jsoncons SAX evaluation + +This change implements the parser evaluation requested in +[ModSecurity issue #3308](https://github.com/owasp-modsecurity/ModSecurity/issues/3308#issuecomment-6142615388). +It does not change ModSecurity or claim integration compatibility with its complete request-body processor. + +## Original behavior and implementation + +The former adapter read an entire file into an `ostringstream`, constructed a +complete `jsoncons::json` DOM, and discarded it. It produced no arguments and +returned unsupported for depth and argument limits. A single-engine test used +itself as reference; missing output and unsupported limits could therefore +appear successful. The combined runner could terminate during `diff | head` +before reporting all failures. + +The new adapter uses public `basic_json_reader`/`json_visitor` APIs (verified with +jsoncons 1.3.2), a streaming binary `ifstream`, and event callbacks. No JSON DOM +or whole-file string is constructed. `read(ec)` includes end-of-input validation +and rejects trailing content. A strict parsing handler rejects comments (the +library's default handler accepts them). Syntax errors include line/column; +I/O, output, allocation and limit failures propagate across the existing C API +without leaking C++ exceptions. Streams and visitor state use RAII. + +The expanded tests exposed a jsoncons 1.3.2 decoding defect: a lone escaped low +surrogate is silently dropped, and a high surrogate followed by a non-low code +unit can be converted into the wrong character. A small public streaming-source +wrapper validates Unicode escape pairing across input buffers before forwarding +them. It performs no JSON grammar parsing or whole-file pass. Raw UTF-8 and the +remaining JSON syntax are validated by jsoncons. + +Each scalar produces one argument, in event order, including duplicate keys; +null produces an empty value. Empty containers produce none. Limits count root +containers as depth 1 and root scalars as depth 0. A depth/argument limit of N +allows exactly N, then rejects the next container/scalar. Direct C callers can +set zero; CLI `-d`/`-a` require decimal integers in 1..UINT_MAX. Every parse call +starts with fresh state and retains the configured limits. Silent mode suppresses +argument writes while enforcing all parsing, count and size checks. + +## Argument names and intentional differences + +The customary YAJL names are retained: `{"a":{"b":1}}` produces `a.b: 1`, +`{"a":[1,2]}` produces `a.a: 1` and `a.a: 2`, and `[1,2]` uses `array.array`. +Array indexes are not added. Nested arrays repeat the enclosing array key. +Duplicate object keys remain separate arguments. + +A structural stack records prefix lengths and parent keys. The shared +`PathHandler` is unsuitable here: it infers parent boundaries from literal dots, +has an off-by-one output-buffer check, and cannot reliably restore parent keys +containing dots. It and the other engines remain unchanged. + +Intentional corrections and numeric normalizations are explicit regression cases: + +- `{"a.b":{"c":1},"d":2}` ends with `d: 2`; YAJL incorrectly emits `a.d: 2`. +- Mixed root arrays, including `[{"x":1},2]`, restore the containing array key: + both adapters produce `array.array.x: 1` then `array.array: 2` in this case. +- Native integers use exact signed/unsigned 64-bit conversions. Oversized + integers arrive as `bigint` string events, preserving all digits, without + conversion to floating point or arbitrary-precision arithmetic. +- `lossless_number(true)` delivers decimal/scientific numbers as `bigdec` + string events. Decimals, exponent magnitude and trailing decimal zeros remain + exact, including `1e9999` and `1e-9999`. These are argument strings, not claims + that a downstream binary float can represent their range. +- jsoncons normalizes integer `-0` to `0` and removes an explicit positive + exponent sign (`1e+03` becomes `1e03`). Negative-zero decimal `-0.0` remains. + Byte compatibility is only asserted for cases without these differences. +- Decoded embedded NULs are written by length instead of truncated like YAJL's + `%s` output. Raw newline/colon/dot characters retain the CLI's ambiguous text + convention; consumers requiring unambiguous framing should use length-aware + callbacks rather than parse this display format. +- jsoncons finalizes documents and rejects truncated input. YAJL's existing + file adapter does not call `yajl_complete_parse`, so it can accept truncation. + Root numbers without a trailing delimiter also lose their argument in YAJL; + jsoncons emits it correctly. These are two separately tested EOF differences. +- The public jsoncons source adaptor accepts a UTF-8 BOM; YAJL rejects it. + This was checked separately and is permitted by RFC 8259's parser guidance. + +## Memory and safety limits + +Keys, full argument paths and decoded values must be shorter than the configured +`JSON_STRING_SIZE` (8192 bytes by default), in both output and silent modes. +Checked dynamic strings avoid fixed-buffer writes. jsoncons buffers an individual +token before visiting it, so its public temporary allocator is additionally +bounded: each token/state allocation may use at most +`4 * JSON_STRING_SIZE + 16384` bytes (48 KiB by default). Excessive strings, +numbers and state growth fail explicitly, including unterminated large tokens. +This budget allows buffer growth and ordinary escaped strings within the value +limit; it is a resource ceiling in addition to semantic limits. + +Memory grows with retained path/depth state and bounded tokenizer storage, rather +than with total document size. There is no fixed whole-file size limit in this +adapter. Configured depth above INT_MAX is capped by jsoncons's int-based depth +option; the temporary state allocation ceiling can reject far smaller extreme +depths too. Deep recursion is avoided by the event reader and explicit stack. + +The other engines have different syntax, range, output and resource behavior. +The regression runner reports those differences as failures where they violate +the required contract; it does not repair their implementations or label absent +argument output as compatible. Missing optional engines remain optional. + +## Verification and integration guidance + +Run the repository build and engine matrix documented in the README. The C API +harness independently checks invalid parameters, setter boundaries, zero limits, +cleanup, error ownership and reuse after failed parses. The benchmark helper +compares a preserved pre-change executable against the SAX executable and YAJL; +its deterministic inputs and raw samples make the measurement reproducible. +See `validation-results.md` for measured results and commands. + +Before adopting jsoncons in ModSecurity: + +1. Pin and test the jsoncons version and strict/lossless options. Test its public + incremental `json_parser` API with actual body chunks and an explicit final + parse step; this adapter currently exposes only a file entry point. +2. Send visitor arguments into ModSecurity's length-aware argument API instead + of stdout. Retain duplicate order, null semantics and both limits. +3. Agree on array names, dotted/empty keys, embedded NULs and numeric lexical + normalizations; replay a real request corpus with explicit expectations. +4. Match body-size, decoded-token, depth and argument budgets to ModSecurity's + configuration. Handle partial events on late parse failure consistently with + request-body error handling; streaming output is not transactional. +5. Measure throughput, allocation, latency and peak memory on representative + concurrent requests, malformed bodies and production compiler/toolchains. + SAX parsing here does not establish streaming JSON-schema validation. diff --git a/docs/validation-results.md b/docs/validation-results.md new file mode 100644 index 0000000..e3faf17 --- /dev/null +++ b/docs/validation-results.md @@ -0,0 +1,161 @@ +# Validation results + +Validated in `/workspace/digitalwave-jsonbench` based on upstream commit +`a89f2e93fb7f1415095d6fb6c1bb5fab6bdc8e6d`. The previous onboarding fork in +`/workspace/jsonbench` was left unchanged. Platform: Debian 13 amd64, GCC 14.2, +jsoncons 1.3.2. Dependencies/tool activation reside outside the checkout under +`/workspace/.jsonbench-tools`; no lockfile or dependency declaration was changed. +Glaze was unavailable and remains optional/unbuilt/unrun. Ten other engines +were discovered and exercised. + +## Build and checks + +From the target checkout, with the existing tool activation sourced: + +```sh +source /workspace/.jsonbench-tools/activate.sh +./autogen.sh +./configure +make -j2 +tests/run_tests.sh JSONCONS +tests/run_tests.sh +``` + +Autotools generation, default configuration and build succeeded. The final +changed parser was compiled with no new warnings. `git diff --check` and shell +syntax checks passed. No ModSecurity source or other parser implementation was +modified. Changes to `jsonbench.c` check unsigned CLI limits and propagate errors +from jsoncons' non-timed output pass with `-n > 1`. + +- JSONCONS selection: **260 passed, 0 failed, 0 skipped**; **6 checked behavioral + differences**, reported separately. Exit 0. 28 explicit cases matched YAJL + byte for byte. Existing fixtures also matched their YAJL reference output. +- Entire ten-engine matrix: **1563 passed, 629 failed, 156 skipped**, plus the + same 6 checked differences. Exit 1, as required. All skips are observed + unsupported-limit warnings; no missing argument output is labelled compatible. +- 56 parser invocations crashed in the full matrix; they were failures, the + runner completed its summary, and core dumps were disabled. No timeouts. +- Opt-in timing (`JSONBENCH_TIMING_RUNS=2`) also passed for JSONCONS; timing + errors would fail the run rather than being ignored. +- A preserved pre-change DOM executable fails **107 required assertions** under + the final JSONCONS matrix (133 passed, 26 unsupported-limit skips, exit 1). + This confirms the new tests detect the former missing-output/limit behavior. +- A separate source copy configured with `--with-jsoncons=no` built successfully + and retained nine engines. Selecting JSONCONS there fails clearly as unavailable. + +The shared matrix contains 34 valid and 41 invalid explicit fixtures, plus +constructed exact/exceeded depth and argument boundaries, string/path boundaries, +2 MiB hostile tokens, a document above the legacy file-size ceiling, invalid CLI +limits, repeated passes and I/O errors. Fixtures are deterministic temporary +files. Golden expectations compare complete bytes, preserving NULs/newlines and +real `Time` keys. Negative checks require a diagnostic and ordinary failure: +signals/timeouts never count as successful rejection or unsupported skips. + +## Results per engine in the full matrix + +The separate JSONCONS/YAJL block adds 28 compatibility passes and 6 checked +DIFFs; it is not included in individual rows below. + +| Engine | Passed | Failed | Skipped | +|---|---:|---:|---:| +| YAJL | 191 | 41 | 0 | +| RAPIDJSON | 191 | 41 | 0 | +| NLOHMANNJSON | 209 | 23 | 0 | +| JSONC | 93 | 113 | 26 | +| JANSSON | 135 | 71 | 26 | +| CJSON | 122 | 84 | 26 | +| JSONCPP | 106 | 100 | 26 | +| JSONCONS | 232 | 0 | 0 | +| SIMDJSON | 108 | 98 | 26 | +| YYJSON | 148 | 58 | 26 | + +The non-jsoncons failures expose existing adapter behavior: numeric reformatting +or loss of range, missing argument output in DOM wrappers, dropped NULs, +incorrect dotted-key restoration, accepted malformed/UTF-8 inputs, missing +finalization, unenforced limits, fixed file buffers and unsafe oversized-key +handling. Individual failures and child exit codes are captured in +`/tmp/jsonbench-root-all-tests.log`. A full replay against the preserved pre-change executable also failed; every +non-jsoncons failure label in the current matrix was present in that baseline. +These counts describe contract checks, not 629 independent bugs. Other engines were retained and their assertions remain +strict; their defects were not patched as part of this jsoncons migration. + +## Memory-safety and API evidence + +A standalone C API harness compiled with `-Wall -Wextra -Werror` passed **220 +checks**. It also passed all 220 under AddressSanitizer + UndefinedBehaviorSanitizer +and leak detection, with no diagnostics: + +```sh +g++ -std=c++17 -Wall -Wextra -Werror -g -O1 \ + -fno-omit-frame-pointer -fsanitize=address,undefined $CPPFLAGS \ + tests/jsoncons_api_tests.cpp src/jsonconsparser.cpp -o /tmp/jsoncons-api-tests +ASAN_OPTIONS=detect_leaks=1:halt_on_error=1 \ +UBSAN_OPTIONS=halt_on_error=1 /tmp/jsoncons-api-tests +``` + +The complete CLI matrix was additionally run against a separately built +JSONCONS-only ASan/UBSan executable: **226 passed, 0 failed, 40 skipped**, exit 0, +no sanitizer diagnostics. Those 40 skips are YAJL comparisons unavailable in +that isolated build, rather than untested jsoncons functionality. The original +all-engine executable was not replaced for this check. Sanitizer build flags: +`-g -O1 -fno-omit-frame-pointer -fno-pie -fsanitize=address,undefined` and linking +with `-no-pie -fsanitize=address,undefined`. + +Independent source review found no remaining material lifetime, allocation, +conversion or buffer issue in the changed implementation. A separate 34-case +Unicode probe verified valid/invalid pairs at 15 offsets around the 16 KiB +buffer boundary, escaped backslashes/quotes, U+10000 and U+10FFFF. Resource +ceilings, partial output on late failures and version-specific surrogate +validation remain explicit integration considerations. + +## Performance measurement + +```sh +python3 tests/benchmark_jsoncons.py \ + --baseline /workspace/.jsonbench-tools/jsonbench-dom-baseline \ + --binary src/jsonbench --output /tmp/jsonbench-final-benchmark.json +``` + +One warmup then five fresh-process samples per measured variant; table timing is +CLI parser mean ± sample standard deviation, in milliseconds. Memory is mean +child-process peak RSS in KiB, measured with a compiled Linux `wait4` helper. +Input: root array of identical short three-field objects, silent output, argument +limit set exactly to the generated leaf count. Effective production optimization +was O2. Binary/config/input hashes and raw samples are in the JSON report; hashes +were checked unchanged during the measurement. + +| Actual input bytes | Variant | Parser ms, mean ± stddev | Mean peak RSS, KiB | +|---:|---|---:|---:| +| 409,564 | jsoncons_dom | 4.477 ± 0.648 | 5330 | +| 409,564 | jsoncons_sax | 2.932 ± 0.079 | 2671 | +| 409,564 | yajl | 1.498 ± 0.208 | 2734 | +| 5,242,854 | yajl | Unrun: configured file limit | — | +| 5,242,854 | jsoncons_dom | 69.876 ± 8.417 | 38976 | +| 5,242,854 | jsoncons_sax | 41.717 ± 5.489 | 2673 | +| 33,554,368 | yajl | Unrun: configured file limit | — | +| 33,554,368 | jsoncons_dom | 363.273 ± 21.097 | 228209 | +| 33,554,368 | jsoncons_sax | 240.755 ± 22.905 | 2649 | + +On this synthetic cached-file workload SAX used substantially less peak memory +than the previous DOM implementation and lower parser time. At 32 MiB the +measured values were approximately 363 ms/223 MiB for DOM and 241 ms/2.6 MiB for +SAX. YAJL was faster on the common smaller fixture; its configured 512 KiB file +buffer prevented larger comparisons. The old DOM adapter merely parsed/discarded +its tree, while SAX also performs argument-path/count/size checks and the Unicode +escape guard. No general ModSecurity throughput or concurrency claim follows. + +## Changed and added files + +Modified: `src/jsonconsparser.cpp`, `src/jsonconsparser.h`, `src/jsonbench.c`, +`tests/run_tests.sh`, `README.md`. + +Added: `tests/regression_tests.py`, `tests/jsoncons_api_tests.cpp`, +`tests/benchmark_jsoncons.py`, `tests/benchmark_rusage.c`, `tests/README.md`, +`docs/jsoncons-sax.md`, `docs/validation-results.md`. + +Generated local Autotools artifacts (`build-aux/`, `config.h`, configure backups, +and ignored object/build files) are not source changes and should not be committed. +See [the SAX notes](jsoncons-sax.md) for the six explicit YAJL differences, +optional BOM behavior, remaining resource/API limits and ModSecurity integration +recommendations. This remains a file-parser evaluation, not an implemented +ModSecurity chunked-body or schema-validation integration. diff --git a/src/jsonbench.c b/src/jsonbench.c index 8a2d06c..12fdbdf 100644 --- a/src/jsonbench.c +++ b/src/jsonbench.c @@ -18,6 +18,7 @@ #include #include #include +#include #ifndef LIMIT_ARG_NUM #define LIMIT_ARG_NUM 500 @@ -57,6 +58,23 @@ static void print_stats(const double *times, int n) { static char available_engines[16][20] = {""}; static int engine_count = 0; +/* Do not let atoi overflow, accept a prefix, or turn negatives into unsigned + * limits. The parser interfaces take double, which exactly represents UINT_MAX. + */ +static int parse_limit(const char *text, unsigned int *value) { + if (text == NULL || *text == '\0') return 0; + for (const char *p = text; *p; ++p) { + if (*p < '0' || *p > '9') return 0; + } + errno = 0; + char *end = NULL; + unsigned long parsed = strtoul(text, &end, 10); + if (errno == ERANGE || *end != '\0' || parsed == 0 || parsed > UINT_MAX) + return 0; + *value = (unsigned int)parsed; + return 1; +} + static void showhelp(void) { printf("Use: jsonbench [OPTIONS]\n\n"); printf("OPTIONS:\n"); @@ -154,18 +172,16 @@ int main(int argc, char ** argv) { } break; case 'a': - arg_limit = atoi(optarg); - if (arg_limit == 0 || arg_limit > (unsigned int)UINT_MAX) { - fprintf(stderr, "Ohh... Try to pass for '-a' an integer between 0 and %u\n", + if (!parse_limit(optarg, &arg_limit)) { + fprintf(stderr, "Pass for '-a' an integer between 1 and %u\n", (unsigned int)UINT_MAX); free(jsonengine); return EXIT_FAILURE; } break; case 'd': - depth_limit = atoi(optarg); - if (depth_limit == 0 || depth_limit > (unsigned int)UINT_MAX) { - fprintf(stderr, "Ohh... Try to pass for '-d' an integer between 0 and %u\n", + if (!parse_limit(optarg, &depth_limit)) { + fprintf(stderr, "Pass for '-d' an integer between 1 and %u\n", (unsigned int)UINT_MAX); free(jsonengine); return EXIT_FAILURE; @@ -759,13 +775,21 @@ int main(int argc, char ** argv) { if (n > 1 && !silence) { jsoncons_parser *json = NULL; - jsoncons_json_init(&json, &error_msg); - if (json != NULL) { + rc = jsoncons_json_init(&json, &error_msg); + if (rc == 0 && json != NULL) { jsoncons_set_max_depth(json, depth_limit); jsoncons_set_max_arg_num(json, arg_limit); jsoncons_set_silence(json, 0); - jsoncons_parse_file(json, jsonfile, &error_msg); + rc = jsoncons_parse_file(json, jsonfile, &error_msg); jsoncons_json_cleanup(json); + } else { + rc = 1; + } + if (rc != 0) { + if (error_msg) { fprintf(stderr, "Error: %s\n", error_msg); free(error_msg); } + free(times); + free(jsonengine); + return EXIT_FAILURE; } } } diff --git a/src/jsonconsparser.cpp b/src/jsonconsparser.cpp index 6f35840..34e7fe6 100644 --- a/src/jsonconsparser.cpp +++ b/src/jsonconsparser.cpp @@ -2,84 +2,387 @@ #if HAVE_JSONCONS +#include +#include +#include +#include #include -#include +#include +#include #include -#include +#include #include -#include +#include +#include +#include + +#include +#include +#include struct jsoncons_parser { - int silence; + unsigned int max_depth = 500; + unsigned int max_args = 500; + bool silence = false; }; -extern "C" int jsoncons_json_init(jsoncons_parser **parser, char **error_msg) { - if (parser == nullptr || error_msg == nullptr) { - return -1; +namespace { + +// jsoncons 1.3.2 silently drops lone escaped low surrogates and does not check +// that the second code unit of a pair is a low surrogate. Validate just this +// lexical boundary while forwarding its public stream source, including pairs +// split over buffers. All JSON grammar and raw UTF-8 validation stay in jsoncons. +class UnicodeSource { + enum class State { outside, text, escape, hex, pair_slash, pair_u }; + jsoncons::stream_source source_; + State state_ = State::outside; + unsigned int codepoint_ = 0; + int digits_ = 0; + bool low_required_ = false; + std::size_t line_ = 1; + std::size_t column_ = 0; + + [[noreturn]] void invalid() const { + throw std::invalid_argument("Invalid Unicode surrogate escape at line " + + std::to_string(line_) + ", column " + std::to_string(column_)); } - try { - *parser = new jsoncons_parser{}; - } catch (const std::bad_alloc&) { - *parser = nullptr; - *error_msg = strdup("Unable to allocate jsoncons parser"); - return 1; + void check(unsigned char c) { + if (c == '\n') { ++line_; column_ = 0; } else { ++column_; } + switch (state_) { + case State::outside: + if (c == '"') state_ = State::text; + break; + case State::text: + if (c == '"') state_ = State::outside; + else if (c == '\\') state_ = State::escape; + break; + case State::escape: + if (c == 'u') { + state_ = State::hex; + codepoint_ = 0; + digits_ = 0; + } else { + state_ = State::text; + } + break; + case State::hex: { + unsigned int digit; + if (c >= '0' && c <= '9') digit = c - '0'; + else if (c >= 'a' && c <= 'f') digit = c - 'a' + 10; + else if (c >= 'A' && c <= 'F') digit = c - 'A' + 10; + else invalid(); + codepoint_ = codepoint_ * 16 + digit; + if (++digits_ != 4) break; + if (low_required_) { + if (codepoint_ < 0xdc00 || codepoint_ > 0xdfff) invalid(); + low_required_ = false; + state_ = State::text; + } else if (codepoint_ >= 0xd800 && codepoint_ <= 0xdbff) { + low_required_ = true; + state_ = State::pair_slash; + } else { + if (codepoint_ >= 0xdc00 && codepoint_ <= 0xdfff) invalid(); + state_ = State::text; + } + break; + } + case State::pair_slash: + if (c != '\\') invalid(); + state_ = State::pair_u; + break; + case State::pair_u: + if (c != 'u') invalid(); + state_ = State::hex; + codepoint_ = 0; + digits_ = 0; + break; + } } - *error_msg = nullptr; - return 0; -} + public: + using value_type = char; + explicit UnicodeSource(std::istream& input) : source_(input) {} + UnicodeSource(UnicodeSource&&) = default; + bool eof() const { return source_.eof(); } + bool is_error() const { return source_.is_error(); } + jsoncons::span read_buffer() { + const auto data = source_.read_buffer(); + for (const unsigned char c : data) check(c); + return data; + } +}; -extern "C" int jsoncons_parse_file(jsoncons_parser *parser, const char *filename, char **error_msg) { - (void)parser; - if (filename == nullptr || error_msg == nullptr) { - return -1; +// jsoncons materializes one decoded token before delivering its event. Bound +// its temporary allocations as well as the visitor lengths, so an unterminated +// multi-megabyte string/number cannot consume document-sized memory first. +// Leave room for geometric growth and the parser's structural state stack. +template +class TemporaryAllocator { + public: + using value_type = T; + TemporaryAllocator() = default; + template TemporaryAllocator(const TemporaryAllocator&) {} + T* allocate(std::size_t count) { + constexpr std::size_t budget = + static_cast(JSON_STRING_SIZE) * 4 + 16384; + if (count > budget / sizeof(T)) + throw std::length_error("jsoncons token/state allocation limit exceeded"); + return std::allocator{}.allocate(count); + } + void deallocate(T* data, std::size_t count) noexcept { + std::allocator{}.deallocate(data, count); } + template bool operator==(const TemporaryAllocator&) const { return true; } + template bool operator!=(const TemporaryAllocator&) const { return false; } +}; - std::ifstream input(filename); - if (!input.is_open()) { - *error_msg = strdup("Unable to open JSON file"); - return 2; +// Keep structural boundaries rather than finding them by searching for dots. +// PathHandler cannot distinguish literal dots/empty keys from nesting, and its +// fixed buffers have unsafe boundary checks. This stack follows the same usual +// dotted naming convention without copying those limitations. +class ArgumentVisitor final : public jsoncons::default_json_visitor { + struct Frame { + std::size_t prefix_length; + std::string key; + }; + + const jsoncons_parser& settings_; + std::vector stack_; + std::string prefix_; + std::string key_; + unsigned int arguments_ = 0; + const char* error_ = nullptr; + + void fail(const char* message, std::error_code& ec) { + error_ = message; + // Visitor return values alone do not abort jsoncons::json_reader. + ec = std::make_error_code(std::errc::invalid_argument); } - std::ostringstream oss; - oss << input.rdbuf(); + void begin(bool array, std::error_code& ec) { + if (stack_.size() >= settings_.max_depth) { + fail("JSON depth limit exceeded", ec); + return; + } + const bool root_array = array && stack_.empty(); + const std::size_t length = root_array ? 5 : + prefix_.size() + (prefix_.empty() ? 0 : 1) + key_.size(); + if (length >= JSON_STRING_SIZE) { + fail("Argument name too long", ec); + return; + } + stack_.push_back({prefix_.size(), key_}); + if (root_array) { + prefix_ = "array"; + key_ = "array"; + } else { + if (!prefix_.empty()) prefix_ += '.'; + prefix_ += key_; + } + } - try { - auto parsed = jsoncons::json::parse(oss.str()); - (void)parsed; - } catch (const std::exception &e) { - *error_msg = strdup(e.what()); - return 2; + void end() { + // The JSON reader guarantees properly paired begin/end events. + prefix_.resize(stack_.back().prefix_length); + key_ = std::move(stack_.back().key); + stack_.pop_back(); } - return 0; -} + void argument(const char* value, std::size_t length, std::error_code& ec) { + if (arguments_ >= settings_.max_args) { + fail("Argument number limit exceeded", ec); + return; + } + const std::size_t name_length = prefix_.size() + + (prefix_.empty() ? 0 : 1) + key_.size(); + // Silent parsing must enforce the same validation and size limits. + if (name_length >= JSON_STRING_SIZE) { + fail("Argument name too long", ec); + return; + } + if (length >= JSON_STRING_SIZE) { + fail("Argument value too long", ec); + return; + } + ++arguments_; + if (settings_.silence) return; + + // Length-aware writes preserve decoded embedded NULs and Unicode. + bool ok = true; + if (!prefix_.empty()) { + ok = std::fwrite(prefix_.data(), 1, prefix_.size(), stdout) == prefix_.size(); + ok = (std::fputc('.', stdout) != EOF) && ok; + } + ok = (std::fwrite(key_.data(), 1, key_.size(), stdout) == key_.size()) && ok; + ok = (std::fwrite(": ", 1, 2, stdout) == 2) && ok; + ok = (std::fwrite(value, 1, length, stdout) == length) && ok; + ok = (std::fputc('\n', stdout) != EOF) && ok; + if (!ok) fail("Unable to write JSON argument output", ec); + } + + JSONCONS_VISITOR_RETURN_TYPE visit_begin_object(jsoncons::semantic_tag, + const jsoncons::ser_context&, std::error_code& ec) override { + begin(false, ec); + JSONCONS_VISITOR_RETURN; + } + JSONCONS_VISITOR_RETURN_TYPE visit_begin_array(jsoncons::semantic_tag, + const jsoncons::ser_context&, std::error_code& ec) override { + begin(true, ec); + JSONCONS_VISITOR_RETURN; + } + JSONCONS_VISITOR_RETURN_TYPE visit_end_object(const jsoncons::ser_context&, + std::error_code&) override { + end(); + JSONCONS_VISITOR_RETURN; + } + JSONCONS_VISITOR_RETURN_TYPE visit_end_array(const jsoncons::ser_context&, + std::error_code&) override { + end(); + JSONCONS_VISITOR_RETURN; + } + JSONCONS_VISITOR_RETURN_TYPE visit_key(const string_view_type& name, + const jsoncons::ser_context&, std::error_code& ec) override { + if (name.size() >= JSON_STRING_SIZE) { + fail("Argument name too long", ec); + } else { + key_.assign(name.data(), name.size()); + } + JSONCONS_VISITOR_RETURN; + } + JSONCONS_VISITOR_RETURN_TYPE visit_null(jsoncons::semantic_tag, + const jsoncons::ser_context&, std::error_code& ec) override { + argument("", 0, ec); + JSONCONS_VISITOR_RETURN; + } + JSONCONS_VISITOR_RETURN_TYPE visit_bool(bool value, jsoncons::semantic_tag, + const jsoncons::ser_context&, std::error_code& ec) override { + argument(value ? "true" : "false", value ? 4 : 5, ec); + JSONCONS_VISITOR_RETURN; + } + JSONCONS_VISITOR_RETURN_TYPE visit_string(const string_view_type& value, + jsoncons::semantic_tag, const jsoncons::ser_context&, + std::error_code& ec) override { + // lossless_number delivers decimals and oversized integers as tagged + // strings, so no conversion through double (or truncation) is needed. + argument(value.data(), value.size(), ec); + JSONCONS_VISITOR_RETURN; + } + JSONCONS_VISITOR_RETURN_TYPE visit_int64(int64_t value, jsoncons::semantic_tag, + const jsoncons::ser_context&, std::error_code& ec) override { + const std::string text = std::to_string(value); + argument(text.data(), text.size(), ec); + JSONCONS_VISITOR_RETURN; + } + JSONCONS_VISITOR_RETURN_TYPE visit_uint64(uint64_t value, jsoncons::semantic_tag, + const jsoncons::ser_context&, std::error_code& ec) override { + const std::string text = std::to_string(value); + argument(text.data(), text.size(), ec); + JSONCONS_VISITOR_RETURN; + } + JSONCONS_VISITOR_RETURN_TYPE visit_double(double, jsoncons::semantic_tag, + const jsoncons::ser_context&, std::error_code& ec) override { + // Defensive: read options must prevent binary floating conversion. + fail("Unexpected non-lossless JSON number event", ec); + JSONCONS_VISITOR_RETURN; + } + + public: + explicit ArgumentVisitor(const jsoncons_parser& settings) : settings_(settings) {} + const char* error() const { return error_; } +}; -extern "C" int jsoncons_set_max_depth(jsoncons_parser *parser, double max_depth) { - if (parser == nullptr) { +int set_limit(jsoncons_parser* parser, double value, unsigned int* target) { + if (parser == nullptr || !std::isfinite(value) || value < 0 || + value > std::numeric_limits::max() || std::floor(value) != value) { return -1; } - (void)max_depth; - return 1; + *target = static_cast(value); + return 0; } -extern "C" int jsoncons_set_max_arg_num(jsoncons_parser *parser, double max_arg_num) { - if (parser == nullptr) { - return -1; +} // namespace + +extern "C" int jsoncons_json_init(jsoncons_parser** parser, char** error_msg) { + if (error_msg == nullptr) return -1; + *error_msg = nullptr; + if (parser == nullptr) return -1; + *parser = nullptr; + try { + *parser = new jsoncons_parser{}; + return 0; + } catch (const std::exception& e) { + *error_msg = strdup(e.what()); + return 1; + } catch (...) { + *error_msg = strdup("Unable to initialize jsoncons parser"); + return 1; } - (void)max_arg_num; - return 1; } -extern "C" int jsoncons_set_silence(jsoncons_parser *parser, int silence) { - if (parser != nullptr) { - parser->silence = silence; +extern "C" int jsoncons_parse_file(jsoncons_parser* parser, const char* filename, + char** error_msg) { + if (error_msg == nullptr) return -1; + *error_msg = nullptr; + if (parser == nullptr || filename == nullptr) return -1; + + // Keep all potentially throwing stream/library/visitor operations inside + // the C boundary; stream, reader and visitor release storage on all exits. + try { + std::ifstream input(filename, std::ios::binary); + if (!input.is_open()) { + *error_msg = strdup("Unable to open JSON file"); + return 2; + } + ArgumentVisitor visitor(*parser); + jsoncons::json_options options; + options.lossless_number(true).err_handler(jsoncons::strict_json_parsing()); + // jsoncons uses int for its own nesting limit. For larger requested + // values its representable ceiling is an additional safety limit. + options.max_nesting_depth(static_cast( + parser->max_depth > static_cast(INT_MAX) ? + INT_MAX : parser->max_depth)); + jsoncons::basic_json_reader> reader(input, visitor, options); + std::error_code ec; + reader.read(ec); // Includes finalization and checks trailing content. + if (ec) { + std::string message = visitor.error() ? visitor.error() : ec.message(); + message += " at line " + std::to_string(reader.line()) + + ", column " + std::to_string(reader.column()); + *error_msg = strdup(message.c_str()); + return 2; + } + if (input.bad()) { + *error_msg = strdup("Error reading JSON file"); + return 2; + } + if (!parser->silence && std::fflush(stdout) != 0) { + *error_msg = strdup("Unable to write JSON argument output"); + return 2; + } + return 0; + } catch (const std::exception& e) { + *error_msg = strdup(e.what()); + return 2; + } catch (...) { + *error_msg = strdup("Unexpected jsoncons parser failure"); + return 2; } - return 0; } -extern "C" int jsoncons_json_cleanup(jsoncons_parser *parser) { +extern "C" int jsoncons_set_max_depth(jsoncons_parser* parser, double max_depth) { + return set_limit(parser, max_depth, parser ? &parser->max_depth : nullptr); +} +extern "C" int jsoncons_set_max_arg_num(jsoncons_parser* parser, double max_args) { + return set_limit(parser, max_args, parser ? &parser->max_args : nullptr); +} +extern "C" int jsoncons_set_silence(jsoncons_parser* parser, int silence) { + if (parser == nullptr) return -1; + parser->silence = silence != 0; + return 0; +} +extern "C" int jsoncons_json_cleanup(jsoncons_parser* parser) { delete parser; return 0; } diff --git a/src/jsonconsparser.h b/src/jsonconsparser.h index b497901..20d375e 100644 --- a/src/jsonconsparser.h +++ b/src/jsonconsparser.h @@ -11,6 +11,15 @@ extern "C" { typedef struct jsoncons_parser jsoncons_parser; +/* Zero denotes success; nonzero denotes failure. Parse/init errors, when + * available, are malloc-owned strings that the caller must free. All C entry + * points contain C++ exceptions. Cleanup accepts NULL; parsing starts fresh + * on every call while retaining the configured limits and silent setting. + * Limits are nonnegative integral doubles <= UINT_MAX. A zero limit allows + * no containers (depth) or no scalar arguments (argument count), respectively. + * CLI limits must be positive. Root containers have depth 1, root scalars 0. + * Null is an empty argument value; every scalar counts, including null/bool. + */ int jsoncons_json_init(jsoncons_parser **parser, char **error_msg); int jsoncons_parse_file(jsoncons_parser *parser, const char *filename, char **error_msg); int jsoncons_set_max_depth(jsoncons_parser *parser, double max_depth); diff --git a/tests/README.md b/tests/README.md new file mode 100644 index 0000000..877cbea --- /dev/null +++ b/tests/README.md @@ -0,0 +1,78 @@ +# Regression tests + +Build using the repository's usual `./autogen.sh`, `./configure`, and `make` +commands, then run: + +```sh +tests/run_tests.sh # every available engine, the same required matrix +tests/run_tests.sh JSONCONS # one engine; still compare with YAJL if available +``` + +The existing shell entry point uses `regression_tests.py`, a Python 3 standard +library helper. It writes deterministic temporary fixtures and deletes them +afterwards. No external test framework is required. All parser subprocesses +have a 15 second timeout and disabled core dumps. A parser crash or timeout is +a failure, rather than successful rejection of invalid JSON. + +The matrix applies to **every selected engine**. It checks exact argument names, +byte values, repeated keys, root primitives, nested arrays and objects, empty +containers, booleans/null, Unicode, embedded NUL, escaped characters, numeric +boundaries and precision, malformed JSON/UTF-8, depth and argument boundaries, +long paths/keys/values, huge tokens, a document larger than the legacy file +buffer, silent mode, repeated benchmark iterations, CLI limit validation, file +errors and output errors. Existing fixtures and cross-engine comparisons remain +included. New expected outputs are explicit data in the helper, not generated +by the parser under test. Argument counts are also tested by running exactly at +the leaf count and one below it. Boolean and null leaves count as arguments; +empty containers do not. + +A parse-only adapter that emits no arguments **fails** output tests. An adapter +that explicitly warns that a particular limit is unsupported can skip that +limit check; it still runs parsing, output, numeric and malformed-input tests. +Unavailable YAJL causes a clearly labelled comparison skip. Single-engine runs +never compare the engine with itself. Only byte equality after checking the +expected outputs is reported as compatibility. Known differences are separately +reported as `DIFF`, not as compatibility passes. Full-engine runs expose existing +adapter defects and return nonzero when any required assertion fails. + +The required numeric contract preserves exact integer precision (including +values outside signed/unsigned 64-bit ranges) and decimal/exponent precision. +jsoncons' public lossless-number API normalizes integer `-0` to `0` and removes +the optional exponent `+`; those two differences have explicit expected results. +Other engines are checked against the original numerical spelling. Known YAJL +path-restoration, embedded-NUL and root-number-at-EOF differences are measured +separately; when YAJL itself is selected it must still satisfy the intended safe +path/output contract and fails those affected tests. No output-wide normalization +is applied to comparisons. The helper removes only the CLI timing record at its +defined position, preserving real blank lines and `Time` argument data. + +To test another built executable or reproduce baseline failures: + +```sh +JSONBENCH_BINARY=/absolute/path/to/jsonbench tests/run_tests.sh JSONCONS +``` + +Timing is opt-in, so regression runs do not spend time benchmarking every small +fixture. Timing failures also affect the test exit status: + +```sh +JSONBENCH_TIMING_RUNS=20 tests/run_tests.sh JSONCONS +``` + +`jsoncons_api_tests.cpp` separately tests the C-compatible API, including null +parameters, limit validation, zero-limit semantics, cleanup, error ownership, +parser reuse after success/failure, and settings preserved after invalid setters. +After configuring with JSONCONS enabled, compile and run it with the same header +paths used for the main build: + +```sh +g++ -std=c++17 -Wall -Wextra -Werror $CPPFLAGS \ + tests/jsoncons_api_tests.cpp src/jsonconsparser.cpp -o /tmp/jsoncons-api-tests +/tmp/jsoncons-api-tests +``` + +`CPPFLAGS` must include the installed jsoncons include path if it is outside the +compiler's standard paths. Add `-fsanitize=address,undefined +-fno-omit-frame-pointer` to this command to run the harness with sanitizers. +The API harness returns nonzero on failure and returns 77 if configured without +JSONCONS; that is unavailable coverage, not a passing test. diff --git a/tests/benchmark_jsoncons.py b/tests/benchmark_jsoncons.py new file mode 100644 index 0000000..9815ec3 --- /dev/null +++ b/tests/benchmark_jsoncons.py @@ -0,0 +1,287 @@ +#!/usr/bin/env python3 +"""Measure JSONCONS SAX, an optional preserved DOM binary, and available YAJL. + +Example: + python3 tests/benchmark_jsoncons.py --baseline /path/to/old/jsonbench \ + --binary src/jsonbench --output /tmp/jsoncons-benchmark.json + +The synthetic workload is a root array of short, three-field objects. It is +intended for a reproducible local comparison, not a ModSecurity performance +claim. The original jsoncons DOM adapter performs no argument/path/limit work. +""" + +import argparse +import hashlib +import json +import os +from pathlib import Path +import platform +import re +import shlex +import signal +import statistics +import subprocess +import sys +import tempfile +import time + + +REPOSITORY = Path(__file__).resolve().parent.parent +ITEM = b'{"name":"abcdefghijklmnopqrstuvwxyz0123456789","id":123456789,"active":true}' +UINT_MAX = 4294967295 + + +def positive_int(text): + value = int(text) + if value <= 0: + raise argparse.ArgumentTypeError("must be a positive integer") + return value + + +def size_bytes(text): + match = re.fullmatch(r"([0-9]+)(B|K|KB|KiB|M|MB|MiB|G|GB|GiB)?", text, re.IGNORECASE) + if not match: + raise argparse.ArgumentTypeError("use bytes or a size such as 400KiB, 5MiB, or 32MiB") + suffix = (match.group(2) or "B").upper() + power = {"B": 0, "K": 1, "KB": 1, "KIB": 1, + "M": 2, "MB": 2, "MIB": 2, "G": 3, "GB": 3, "GIB": 3}[suffix] + value = int(match.group(1)) * 1024 ** power + if value < len(ITEM) + 2: + raise argparse.ArgumentTypeError("size must fit at least one benchmark object") + if ((value - 1) // (len(ITEM) + 1)) * 3 > UINT_MAX: + raise argparse.ArgumentTypeError("generated argument count would exceed the CLI limit") + return value + + +def file_hash(path): + digest = hashlib.sha256() + with path.open("rb") as stream: + for block in iter(lambda: stream.read(1024 * 1024), b""): + digest.update(block) + return digest.hexdigest() + + +def executable(parser, path, label): + path = path.expanduser().resolve() + if not path.is_file() or not os.access(path, os.X_OK): + hint = ("preserve the original executable before rebuilding, then pass --baseline PATH" + if label == "baseline" else "run ./autogen.sh, ./configure, and make first") + parser.error(f"{label} executable is missing or not executable: {path}; {hint}") + return path + + +def available_engines(binary): + result = subprocess.run([str(binary), "-h"], stdin=subprocess.DEVNULL, + capture_output=True, text=True, check=False) + if result.returncode: + raise RuntimeError(f"{binary} -h failed: {result.stderr.strip()}") + return set(re.findall(r"^\s*- ([A-Z][A-Z0-9]+)\s*$", result.stdout, re.MULTILINE)) + + +def configured_buffer(path): + if not path.is_file(): + raise RuntimeError(f"configured header missing: {path}; run ./configure or pass --config PATH") + text = path.read_text() + match = re.search(r"^\s*#\s*define\s+FILE_BUFFER_SIZE\s+([0-9]+)[uUlL]*\s*$", text, re.MULTILINE) + if not match or int(match.group(1)) < 2: + raise RuntimeError(f"cannot determine FILE_BUFFER_SIZE from {path}") + return int(match.group(1)) + + +def dataset(directory, target): + count = (target - 1) // (len(ITEM) + 1) + path = directory / f"input-{target}.json" + digest = hashlib.sha256() + with path.open("wb") as stream: + def write(block): + stream.write(block) + digest.update(block) + + write(b"[" + ITEM) + remaining = count - 1 + block = (b"," + ITEM) * 4096 + while remaining >= 4096: + write(block) + remaining -= 4096 + write((b"," + ITEM) * remaining) + write(b"]") + return path, {"target_bytes": target, "bytes": path.stat().st_size, + "objects": count, "arguments": count * 3, "sha256": digest.hexdigest()} + + +def invoke(helper, binary, engine, path, argument_count, timeout): + command = [str(helper), str(binary), "-s", "-a", str(argument_count), + "-n", "1", "-e", engine, str(path)] + start = time.perf_counter() + process = subprocess.Popen(command, stdin=subprocess.DEVNULL, + stdout=subprocess.PIPE, stderr=subprocess.PIPE, + start_new_session=True) + try: + stdout, stderr = process.communicate(timeout=timeout) + except subprocess.TimeoutExpired: + try: + os.killpg(process.pid, signal.SIGKILL) + except ProcessLookupError: + pass + stdout, stderr = process.communicate() + return {"status": "failed", "reason": f"timeout after {timeout} seconds", + "exit_code": process.returncode, + "stdout": stdout.decode(errors="replace"), "stderr": stderr.decode(errors="replace")} + except BaseException: + # Ctrl-C must also stop the helper and its parser subprocess. + try: + os.killpg(process.pid, signal.SIGKILL) + except ProcessLookupError: + pass + process.communicate() + raise + wall = time.perf_counter() - start + stdout = stdout.decode(errors="replace") + stderr = stderr.decode(errors="replace") + timing = re.search(r"Time:\s+([0-9.eE+-]+)\s+sec", stdout) + usage = re.search(r"^JSONBENCH_RUSAGE (\{.*\})$", stderr, re.MULTILINE) + if process.returncode or not timing or not usage: + return {"status": "failed", "exit_code": process.returncode, + "reason": "parser failed or required timing/resource output is missing", + "stdout": stdout, "stderr": stderr} + result = {"status": "ok", "parser_seconds": float(timing.group(1)), "wall_seconds": wall} + result.update(json.loads(usage.group(1))) + diagnostics = re.sub(r"^JSONBENCH_RUSAGE .*\n?", "", stderr, flags=re.MULTILINE).strip() + if diagnostics: + result["stderr"] = diagnostics + return result + + +def summary(samples): + result = {} + for metric in ("parser_seconds", "wall_seconds", "peak_rss_kib"): + values = [sample[metric] for sample in samples] + result[metric] = {"mean": statistics.mean(values), + "sample_stdev": statistics.stdev(values) if len(values) > 1 else 0, + "min": min(values), "max": max(values)} + return result + + +def main(): + parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) + parser.add_argument("--binary", type=Path, default=REPOSITORY / "src/jsonbench") + parser.add_argument("--baseline", type=Path, help="preserved executable containing the original DOM implementation") + parser.add_argument("--config", type=Path, default=REPOSITORY / "config.h", + help="configured header used to build --binary, for YAJL's file buffer limit") + parser.add_argument("--samples", type=positive_int, default=5) + parser.add_argument("--warmups", type=positive_int, default=1) + parser.add_argument("--sizes", type=size_bytes, nargs="+", default=None) + parser.add_argument("--timeout", type=positive_int, default=60, help="seconds per process") + parser.add_argument("--output", type=Path, help="write JSON report here instead of stdout") + args = parser.parse_args() + if platform.system() != "Linux": + parser.error("the peak-RSS helper currently requires Linux (ru_maxrss is reported in KiB)") + + binary = executable(parser, args.binary, "SAX") + baseline = executable(parser, args.baseline, "baseline") if args.baseline else None + config = args.config.expanduser().resolve() + try: + file_buffer_size = configured_buffer(config) + engines = available_engines(binary) + if "JSONCONS" not in engines: + raise RuntimeError(f"JSONCONS is unavailable in {binary}; configure with jsoncons headers installed") + if baseline and "JSONCONS" not in available_engines(baseline): + raise RuntimeError(f"JSONCONS is unavailable in the baseline executable {baseline}") + except (OSError, RuntimeError) as error: + parser.error(str(error)) + + common_size = min(400 * 1024, file_buffer_size - 1) + sizes = args.sizes or [max(common_size, len(ITEM) + 2), 5 * 1024 * 1024, 32 * 1024 * 1024] + compiler = shlex.split(os.environ.get("CC", "cc")) + if not compiler: + parser.error("CC is empty; set CC to a C compiler or leave it unset to use cc") + + artifacts = {"binary": {"path": str(binary), "sha256": file_hash(binary)}, + "config": {"path": str(config), "sha256": file_hash(config)}} + if baseline: + artifacts["baseline"] = {"path": str(baseline), "sha256": file_hash(baseline)} + report = {"schema_version": 1, "platform": platform.platform(), "artifacts": artifacts, + "method": {"samples": args.samples, "warmups": args.warmups, + "cli": "-s -a GENERATED_ARGUMENT_COUNT -n 1 -e ENGINE INPUT", + "input": "root array of identical objects: short ASCII string, exact integer, boolean", + "timing": "CLI parser elapsed seconds; independent fresh processes after warmups", + "rss": "Linux wait4 child peak RSS in KiB via a small compiled C launcher", + "file_buffer_size": file_buffer_size, + "yajl_maximum_file_bytes": file_buffer_size - 1, + "compiler": compiler, + "caveat": "Original DOM adapter only parses/discards a DOM. SAX also builds argument paths and enforces argument/size limits. Synthetic cached-file workload; no ModSecurity performance conclusion."}, + "inputs": [], "runs": []} + failed = False + with tempfile.TemporaryDirectory(prefix="jsonbench-benchmark-") as temp: + directory = Path(temp) + helper = directory / "benchmark-rusage" + compile_command = compiler + ["-O2", "-std=c11", "-Wall", "-Wextra", + str(Path(__file__).with_name("benchmark_rusage.c")), "-o", str(helper)] + try: + compile_result = subprocess.run(compile_command, capture_output=True, text=True, check=False) + except OSError as error: + parser.error(f"cannot run C compiler {compiler[0]}: {error}") + if compile_result.returncode: + parser.error("cannot compile peak-RSS helper: " + compile_result.stderr.strip()) + + for target in sizes: + path, info = dataset(directory, target) + report["inputs"].append(info) + variants = [("jsoncons_sax", binary, "JSONCONS")] + if baseline: + variants.insert(0, ("jsoncons_dom", baseline, "JSONCONS")) + else: + report["runs"].append({"variant": "jsoncons_dom", "input_bytes": info["bytes"], + "status": "skipped", "reason": "no baseline supplied; pass --baseline /path/to/preserved/executable"}) + if "YAJL" not in engines: + report["runs"].append({"variant": "yajl", "input_bytes": info["bytes"], + "status": "skipped", "reason": "YAJL is unavailable in the configured binary"}) + elif info["bytes"] > file_buffer_size - 1: + report["runs"].append({"variant": "yajl", "input_bytes": info["bytes"], + "status": "skipped", "reason": f"input exceeds configured YAJL file limit of {file_buffer_size - 1} bytes"}) + else: + variants.append(("yajl", binary, "YAJL")) + + for name, target_binary, engine in variants: + run = {"variant": name, "input_bytes": info["bytes"], "status": "ok", "warmups": [], "samples": []} + for _ in range(args.warmups): + sample = invoke(helper, target_binary, engine, path, info["arguments"], args.timeout) + run["warmups"].append(sample) + if sample["status"] != "ok": + run["status"] = "failed" + break + if run["status"] == "ok": + for _ in range(args.samples): + sample = invoke(helper, target_binary, engine, path, info["arguments"], args.timeout) + run["samples"].append(sample) + if sample["status"] != "ok": + run["status"] = "failed" + break + if run["status"] == "ok": + run["summary"] = summary(run["samples"]) + metrics = run["summary"] + print(f"{name}: {info['bytes']} bytes, " + f"{metrics['parser_seconds']['mean'] * 1000:.3f} +/- " + f"{metrics['parser_seconds']['sample_stdev'] * 1000:.3f} ms, " + f"{metrics['peak_rss_kib']['mean']:.0f} KiB mean peak RSS", file=sys.stderr) + else: + failed = True + print(f"FAILED {name} for {info['bytes']} bytes: inspect JSON report diagnostics", file=sys.stderr) + report["runs"].append(run) + + for label, artifact in artifacts.items(): + if file_hash(Path(artifact["path"])) != artifact["sha256"]: + failed = True + report.setdefault("errors", []).append(f"{label} changed during measurement; results are invalid") + report["status"] = "failed" if failed else "ok" + output = json.dumps(report, indent=2) + "\n" + if args.output: + args.output.write_text(output) + print(f"JSON report: {args.output}", file=sys.stderr) + else: + sys.stdout.write(output) + return 1 if failed else 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tests/benchmark_rusage.c b/tests/benchmark_rusage.c new file mode 100644 index 0000000..bf42813 --- /dev/null +++ b/tests/benchmark_rusage.c @@ -0,0 +1,47 @@ +#define _DEFAULT_SOURCE + +#include +#include +#include +#include +#include +#include + +/* Measure a target forked by a small, freshly exec'd process. Directly using + * Python's child rusage can include the Python parent's resident-memory floor. + * Linux reports ru_maxrss in KiB; benchmark_jsoncons.py checks the platform. + */ +int main(int argc, char **argv) { + if (argc < 2) { + fprintf(stderr, "Usage: benchmark_rusage EXECUTABLE [ARGUMENTS...]\n"); + return 2; + } + + pid_t child = fork(); + if (child < 0) { + perror("fork"); + return 2; + } + if (child == 0) { + execv(argv[1], argv + 1); + perror("execv"); + _exit(127); + } + + int status; + struct rusage usage; + while (wait4(child, &status, 0, &usage) < 0) { + if (errno == EINTR) continue; + perror("wait4"); + return 2; + } + double user = (double)usage.ru_utime.tv_sec + usage.ru_utime.tv_usec / 1000000.0; + double system = (double)usage.ru_stime.tv_sec + usage.ru_stime.tv_usec / 1000000.0; + fprintf(stderr, + "JSONBENCH_RUSAGE {\"peak_rss_kib\":%ld,\"user_seconds\":%.9f," + "\"system_seconds\":%.9f}\n", + usage.ru_maxrss, user, system); + + if (WIFEXITED(status)) return WEXITSTATUS(status); + return WIFSIGNALED(status) ? 128 + WTERMSIG(status) : 2; +} diff --git a/tests/jsoncons_api_tests.cpp b/tests/jsoncons_api_tests.cpp new file mode 100644 index 0000000..e474f1e --- /dev/null +++ b/tests/jsoncons_api_tests.cpp @@ -0,0 +1,316 @@ +// Direct regression checks for the C interface, including parser reuse and +// limits that cannot be represented by the CLI's positive-integer options. +#include "../src/jsonconsparser.h" + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include + +#if HAVE_JSONCONS +namespace { + +class TemporaryInput { + std::string path_; + + public: + explicit TemporaryInput(const std::string& text) { + char name[] = "/tmp/jsonbench-jsoncons-api-XXXXXX"; + const int fd = mkstemp(name); + if (fd < 0) throw std::runtime_error("mkstemp failed"); + path_ = name; + std::size_t written = 0; + while (written < text.size()) { + const ssize_t count = write(fd, text.data() + written, text.size() - written); + if (count < 0 && errno == EINTR) continue; + if (count <= 0) { + close(fd); + unlink(path_.c_str()); + throw std::runtime_error("temporary input write failed"); + } + written += static_cast(count); + } + if (close(fd) != 0) { + unlink(path_.c_str()); + throw std::runtime_error("temporary input close failed"); + } + } + ~TemporaryInput() { unlink(path_.c_str()); } + TemporaryInput(const TemporaryInput&) = delete; + TemporaryInput& operator=(const TemporaryInput&) = delete; + const char* path() const { return path_.c_str(); } +}; + +class CapturedOutput { + FILE* file_ = nullptr; + int saved_fd_ = -1; + + public: + CapturedOutput() { + if (std::fflush(stdout) != 0) throw std::runtime_error("stdout flush failed"); + file_ = std::tmpfile(); + if (file_ == nullptr) throw std::runtime_error("stdout temporary file failed"); + saved_fd_ = dup(STDOUT_FILENO); + if (saved_fd_ < 0 || dup2(fileno(file_), STDOUT_FILENO) < 0) { + if (saved_fd_ >= 0) close(saved_fd_); + std::fclose(file_); + throw std::runtime_error("stdout capture failed"); + } + } + ~CapturedOutput() { + std::fflush(stdout); + dup2(saved_fd_, STDOUT_FILENO); + close(saved_fd_); + std::fclose(file_); + } + CapturedOutput(const CapturedOutput&) = delete; + CapturedOutput& operator=(const CapturedOutput&) = delete; + std::string read() { + if (std::fflush(stdout) != 0 || std::fseek(file_, 0, SEEK_SET) != 0) + throw std::runtime_error("captured output rewind failed"); + std::string text; + char buffer[1024]; + std::size_t count; + while ((count = std::fread(buffer, 1, sizeof buffer, file_)) != 0) + text.append(buffer, count); + if (std::ferror(file_)) throw std::runtime_error("captured output read failed"); + return text; + } +}; + +// Start each error slot with a caller-owned allocation. The API must reset +// the pointer, not free the caller's old value. Free returned diagnostics with +// free(), as required by the C interface, on success and failure paths alike. +class ErrorSlot { + char* original_; + + public: + char* value; + ErrorSlot() : original_(static_cast(std::malloc(1))), value(original_) { + if (original_ == nullptr) throw std::bad_alloc(); + *original_ = '\0'; + } + ~ErrorSlot() { + if (value != original_) std::free(value); + std::free(original_); + } + ErrorSlot(const ErrorSlot&) = delete; + ErrorSlot& operator=(const ErrorSlot&) = delete; + bool replaced() const { return value != original_; } +}; + +class Parser { + public: + jsoncons_parser* value = nullptr; + Parser() { + ErrorSlot error; + const int rc = jsoncons_json_init(&value, &error.value); + if (rc != 0 || value == nullptr || error.value != nullptr) { + jsoncons_json_cleanup(value); + throw std::runtime_error("parser initialization failed"); + } + } + ~Parser() { jsoncons_json_cleanup(value); } + Parser(const Parser&) = delete; + Parser& operator=(const Parser&) = delete; +}; + +struct ParseResult { + int status; + bool error_replaced; + bool error_is_null; + std::string error; + std::string output; +}; + +ParseResult parse_path(jsoncons_parser* parser, const char* path) { + ErrorSlot error; + CapturedOutput output; + const int status = jsoncons_parse_file(parser, path, &error.value); + return {status, error.replaced(), error.value == nullptr, + error.value != nullptr ? error.value : "", output.read()}; +} + +ParseResult parse(jsoncons_parser* parser, const std::string& text) { + TemporaryInput input(text); + return parse_path(parser, input.path()); +} + +class Checks { + unsigned int total_ = 0; + unsigned int failures_ = 0; + + public: + void check(const std::string& name, bool passed) { + ++total_; + if (!passed) { + ++failures_; + std::fprintf(stderr, "FAIL: %s\n", name.c_str()); + } + } + void success(const std::string& name, const ParseResult& result, + const std::string& expected = "") { + check(name + ": succeeds", result.status == 0); + check(name + ": clears previous error", result.error_replaced && result.error_is_null); + check(name + ": exact output", result.output == expected); + if (result.status != 0) + std::fprintf(stderr, " diagnostic: %s\n", result.error.c_str()); + } + void failure(const std::string& name, const ParseResult& result) { + check(name + ": fails", result.status != 0); + check(name + ": returns fresh malloc-owned diagnostic", + result.error_replaced && !result.error_is_null && !result.error.empty()); + } + int finish() const { + std::printf("jsoncons C API: %u checks passed, %u failed\n", + total_ - failures_, failures_); + return failures_ == 0 ? 0 : 1; + } +}; + +void invalid_arguments(Checks& checks) { + { + ErrorSlot error; + checks.check("init rejects null parser slot", jsoncons_json_init(nullptr, &error.value) != 0); + checks.check("failed init clears previous error", error.replaced() && error.value == nullptr); + } + jsoncons_parser* null_parser = nullptr; + checks.check("init rejects null error slot", jsoncons_json_init(&null_parser, nullptr) != 0); + checks.check("failed init leaves null parser", null_parser == nullptr); + checks.check("init rejects both null slots", jsoncons_json_init(nullptr, nullptr) != 0); + checks.check("cleanup accepts null", jsoncons_json_cleanup(nullptr) == 0); + checks.check("depth setter rejects null parser", jsoncons_set_max_depth(nullptr, 1) != 0); + checks.check("argument setter rejects null parser", jsoncons_set_max_arg_num(nullptr, 1) != 0); + checks.check("silence setter rejects null parser", jsoncons_set_silence(nullptr, 1) != 0); + + Parser parser; + TemporaryInput input("null"); + const ParseResult no_parser = parse_path(nullptr, input.path()); + checks.check("parse rejects null parser", no_parser.status != 0); + checks.check("invalid parser clears previous error", no_parser.error_replaced && no_parser.error_is_null); + const ParseResult no_filename = parse_path(parser.value, nullptr); + checks.check("parse rejects null filename", no_filename.status != 0); + checks.check("invalid filename clears previous error", no_filename.error_replaced && no_filename.error_is_null); + checks.check("parse rejects null error slot", + jsoncons_parse_file(parser.value, input.path(), nullptr) != 0); + checks.failure("missing file", parse_path(parser.value, (std::string(input.path()) + ".missing").c_str())); + checks.success("parse remains usable after invalid calls", parse(parser.value, "null"), ": \n"); +} + +void limit_setters(Checks& checks) { + Parser parser; + checks.check("set depth to two", jsoncons_set_max_depth(parser.value, 2) == 0); + checks.check("set arguments to two", jsoncons_set_max_arg_num(parser.value, 2) == 0); + checks.check("enable silence", jsoncons_set_silence(parser.value, 1) == 0); + const double upper = static_cast(UINT_MAX); + const std::vector> invalid = { + {"NaN", std::numeric_limits::quiet_NaN()}, + {"positive infinity", std::numeric_limits::infinity()}, + {"negative infinity", -std::numeric_limits::infinity()}, + {"negative integer", -1.0}, {"negative fraction", -0.5}, + {"fraction below one", 0.5}, {"positive fraction", 1.5}, + {"above UINT_MAX", upper + 1.0}, {"DBL_MAX", std::numeric_limits::max()} + }; + for (const auto& value : invalid) { + checks.check("depth rejects " + value.first, + jsoncons_set_max_depth(parser.value, value.second) != 0); + checks.check("argument count rejects " + value.first, + jsoncons_set_max_arg_num(parser.value, value.second) != 0); + checks.success("rejected " + value.first + " preserves limits", + parse(parser.value, "{\"a\":{\"b\":null,\"c\":true}}")); + checks.failure("rejected " + value.first + " preserves depth ceiling", + parse(parser.value, "[[[null]]]")); + checks.failure("rejected " + value.first + " preserves count ceiling", + parse(parser.value, "[null,true,false]")); + } + checks.check("depth accepts UINT_MAX", jsoncons_set_max_depth(parser.value, upper) == 0); + checks.check("argument count accepts UINT_MAX", jsoncons_set_max_arg_num(parser.value, upper) == 0); + checks.success("maximum representable limits permit ordinary JSON", + parse(parser.value, "{\"a\":[null,true,false]}")); +} + +void zero_limits(Checks& checks) { + Parser parser; + checks.check("set depth to zero", jsoncons_set_max_depth(parser.value, 0) == 0); + checks.check("enable silence for zero depth", jsoncons_set_silence(parser.value, 1) == 0); + for (const auto& text : {"null", "true", "false", "1", "\"value\""}) + checks.success(std::string("zero depth allows root ") + text, parse(parser.value, text)); + checks.failure("zero depth rejects object", parse(parser.value, "{}")); + checks.failure("zero depth rejects array", parse(parser.value, "[]")); + checks.check("restore depth for zero arguments", jsoncons_set_max_depth(parser.value, 3) == 0); + checks.check("set arguments to zero", jsoncons_set_max_arg_num(parser.value, 0) == 0); + for (const auto& text : {"{}", "[]", "{\"a\":[{},[]]}"}) + checks.success(std::string("zero arguments allows ") + text, parse(parser.value, text)); + for (const auto& text : {"null", "true", "false", "0", "\"\"", "{\"a\":null}", "[false]"}) + checks.failure(std::string("zero arguments rejects ") + text, parse(parser.value, text)); + checks.check("negative zero is an integral zero limit", jsoncons_set_max_depth(parser.value, -0.0) == 0); + checks.failure("negative-zero depth has zero semantics", parse(parser.value, "{}")); +} + +void boundaries_and_reuse(Checks& checks) { + Parser parser; + checks.check("set exact depth boundary", jsoncons_set_max_depth(parser.value, 3) == 0); + checks.check("set exact count boundary", jsoncons_set_max_arg_num(parser.value, 2) == 0); + checks.check("silence accepts any nonzero value", jsoncons_set_silence(parser.value, -7) == 0); + checks.success("array exactly at depth", parse(parser.value, "[[[null]]]")); + checks.failure("array above depth", parse(parser.value, "[[[[null]]]]")); + checks.success("object exactly at depth", parse(parser.value, "{\"a\":{\"b\":{\"c\":true}}}")); + checks.failure("object above depth", parse(parser.value, "{\"a\":{\"b\":{\"c\":{}}}}")); + checks.success("null and bool exactly at count", parse(parser.value, "{\"first\":null,\"second\":true}")); + checks.failure("null and bool above count", parse(parser.value, "{\"first\":null,\"second\":true,\"third\":false}")); + checks.success("empty containers consume no arguments", parse(parser.value, "{\"a\":{},\"b\":[],\"c\":null,\"d\":false}")); + checks.success("success resets count and path", parse(parser.value, "{\"new\":null,\"fresh\":false}")); + const ParseResult malformed = parse(parser.value, "{\"leaked\":{\"a\":true,\"b\":"); + checks.failure("nested truncated input", malformed); + checks.check("syntax error reports line", malformed.error.find("line ") != std::string::npos); + checks.check("syntax error reports column", malformed.error.find("column ") != std::string::npos); + checks.check("silent error produces no arguments", malformed.output.empty()); + checks.success("syntax failure resets count and path", parse(parser.value, "{\"fresh\":true}")); + checks.check("disable silence after reused parser", jsoncons_set_silence(parser.value, 0) == 0); + checks.success("output has fresh path after nested failure", parse(parser.value, "{\"fresh\":true}"), "fresh: true\n"); + checks.success("successful root array resets afterward", parse(parser.value, "[null,false]"), "array.array: \narray.array: false\n"); + checks.success("root scalar has fresh empty path", parse(parser.value, "2"), ": 2\n"); + const ParseResult count_error = parse(parser.value, "{\"a\":null,\"b\":true,\"c\":false}"); + checks.failure("retains count limit when unsilenced", count_error); + checks.check("count limit prints exactly permitted prefix", count_error.output == "a: \nb: true\n"); + checks.success("count failure resets parser", parse(parser.value, "{\"after\":false}"), "after: false\n"); + checks.check("tighten depth retained across calls", jsoncons_set_max_depth(parser.value, 1) == 0); + checks.failure("retains depth limit across calls", parse(parser.value, "{\"too\":[]}")); + checks.success("depth failure resets parser", parse(parser.value, "{\"after\":null}"), "after: \n"); + checks.check("reenable silence", jsoncons_set_silence(parser.value, 2) == 0); + checks.success("silence persists on successful reuse", parse(parser.value, "{\"quiet\":false}")); + checks.failure("silence still validates malformed JSON", parse(parser.value, "{\"quiet\":true,}")); + checks.success("silence persists after failed reuse", parse(parser.value, "{\"quiet\":true}")); +} + +} // namespace + +int main() { + try { + Checks checks; + invalid_arguments(checks); + limit_setters(checks); + zero_limits(checks); + boundaries_and_reuse(checks); + return checks.finish(); + } catch (const std::exception& error) { + std::fprintf(stderr, "FAIL: C API test setup: %s\n", error.what()); + return 1; + } +} +#else +int main() { + std::fprintf(stderr, "jsoncons C API tests require an enabled JSONCONS engine\n"); + return 77; +} +#endif diff --git a/tests/regression_tests.py b/tests/regression_tests.py new file mode 100755 index 0000000..303a960 --- /dev/null +++ b/tests/regression_tests.py @@ -0,0 +1,431 @@ +#!/usr/bin/env python3 +"""Deterministic CLI regression checks, without an external test framework.""" + +import difflib +import os +from pathlib import Path +import re +import resource +import subprocess +import sys +import tempfile +from dataclasses import dataclass +from typing import Optional + + +TESTS_DIR = Path(__file__).resolve().parent +# Remove only the final CLI timing trailer. Removing arbitrary Time: lines or +# blank lines would lose real argument data. Byte comparisons retain NUL/UTF-8. +TIMING_TEXT = rb"\nTime: (?:[0-9]+\.[0-9]+ sec|mean=[0-9]+\.[0-9]+ sec stddev=[0-9]+\.[0-9]+ sec \(N=[0-9]+\))\n\n" +TIMING_SUFFIX = re.compile(TIMING_TEXT + rb"\Z") +TIMING_PREFIX = re.compile(rb"\A" + TIMING_TEXT) + + +@dataclass +class Case: + name: str + document: bytes + expected: bytes + count: int + yajl_observed: Optional[bytes] = None + numeric_difference: bool = False + + def output_for(self, engine): + # YAJL preserves -0 and exponent '+'. jsoncons' public lossless API + # normalizes those spellings; other engines still must preserve value. + if self.numeric_difference and engine != "JSONCONS": + return self.yajl_observed + return self.expected + + +def utf8(text): + return text.encode("utf-8") + + +VALID_CASES = [ + Case("empty-object", b"{}", b"", 0), + Case("empty-array", b"[]", b"", 0), + Case("empty-containers", b'{"a":{},"b":[],"c":[{},[]]}', b"", 0), + Case("scalars", b'{"s":"text","t":true,"f":false,"n":null,"i":-42,"d":1.25}', + b"s: text\nt: true\nf: false\nn: \ni: -42\nd: 1.25\n", 6), + Case("root-string", b'"hello"', b": hello\n", 1), + Case("root-integer", b"42\n", b": 42\n", 1), + Case("root-integer-eof", b"42", b": 42\n", 1, b""), + Case("root-decimal", b"1.25\n", b": 1.25\n", 1), + Case("root-decimal-eof", b"1.25", b": 1.25\n", 1, b""), + Case("root-true", b"true", b": true\n", 1), + Case("root-false", b"false", b": false\n", 1), + Case("root-null", b"null", b": \n", 1), + Case("root-array", b'[1,true,null,"s"]', + b"array.array: 1\narray.array: true\narray.array: \narray.array: s\n", 4), + Case("nested-arrays", b"[[0,1],[2,3]]", + b"array.array.array: 0\narray.array.array: 1\narray.array.array: 2\narray.array.array: 3\n", 4), + Case("nested-objects", b'{"a":{"b":{"c":1}},"after":2}', b"a.b.c: 1\nafter: 2\n", 2), + Case("mixed-arrays", b'{"a":[1,[2,3],4],"b":{"c":5},"end":6}', + b"a.a: 1\na.a.a: 2\na.a.a: 3\na.a: 4\nb.c: 5\nend: 6\n", 6), + Case("array-objects", b'{"rows":[{"a":1,"b":2},{"a":3}],"after":4}', + b"rows.rows.a: 1\nrows.rows.b: 2\nrows.rows.a: 3\nafter: 4\n", 4), + Case("array-parent-key", b'[{"a":1},2,{"b":3},4]', + b"array.array.a: 1\narray.array: 2\narray.array.b: 3\narray.array: 4\n", 4), + Case("array-after-empty", b'{"a":[{"x":[]},1,{},2,[],3]}', + b"a.a: 1\na.a: 2\na.a: 3\n", 3), + Case("repeated-keys", b'{"k":1,"k":2,"nest":{"k":3,"k":4}}', + b"k: 1\nk: 2\nnest.k: 3\nnest.k: 4\n", 4), + Case("empty-keys", b'{"":{"":1},"last":2}', b": 1\nlast: 2\n", 2), + Case("dotted-keys", b'{"a.b":{"c":1},"d":2}', + b"a.b.c: 1\nd: 2\n", 2, b"a.b.c: 1\na.d: 2\n"), + Case("unicode", utf8('{"Grüße":"日本語 😀","escaped":"\\u20ac\\ud83d\\ude00"}'), + utf8("Grüße: 日本語 😀\nescaped: €😀\n"), 2), + Case("literal-unicode-escape", b'{"literal":"\\\\udc00"}', b"literal: \\udc00\n", 1), + Case("surrogate-across-buffer", b" " * 16379 + b'"\\ud83d\\ude00"', utf8(": 😀\n"), 1), + Case("escaped-backslash-across-buffer", b" " * 16382 + b'"\\\\udc00"', b": \\udc00\n", 1), + Case("utf8-across-buffer", b" " * 16381 + utf8('"😀"'), utf8(": 😀\n"), 1), + Case("escaped-strings", b'{"s":"quote:\\" slash:\\/ backslash:\\\\ tab:\\t line:\\n end"}', + b's: quote:" slash:/ backslash:\\ tab:\t line:\n end\n', 1), + Case("embedded-nul", b'{"a\\u0000b":"x\\u0000y"}', b"a\0b: x\0y\n", 1, b"a: x\n"), + Case("timing-like-value", b'{"Time":"0.000000000 sec","blank":"\\n"}', + b"Time: 0.000000000 sec\nblank: \n\n", 2), + Case("integer-boundaries", + b'{"min":-9223372036854775808,"max":9223372036854775807,"u":18446744073709551615,"beyond":18446744073709551616,"negative":-9223372036854775809}', + b"min: -9223372036854775808\nmax: 9223372036854775807\nu: 18446744073709551615\nbeyond: 18446744073709551616\nnegative: -9223372036854775809\n", 5), + Case("negative-zero", b'{"zero":0,"negative":-0,"decimal":-0.0}', + b"zero: 0\nnegative: 0\ndecimal: -0.0\n", 3, + b"zero: 0\nnegative: -0\ndecimal: -0.0\n", True), + Case("decimal-precision", + b'{"decimal":1.2300,"precise":0.123456789012345678901234567890,"tiny":1e-10000,"huge":1E9999}', + b"decimal: 1.2300\nprecise: 0.123456789012345678901234567890\ntiny: 1e-10000\nhuge: 1E9999\n", 4), + Case("exponents", b'{"plus":1e+03,"capital":1E-03,"zeros":1e-0003,"huge":1E+9999}', + b"plus: 1e03\ncapital: 1E-03\nzeros: 1e-0003\nhuge: 1E9999\n", 4, + b"plus: 1e+03\ncapital: 1E-03\nzeros: 1e-0003\nhuge: 1E+9999\n", True), +] + +INVALID_CASES = [ + ("empty-input", b""), ("whitespace-only", b" \t\r\n"), + ("truncated-object", b'{"a":1'), ("truncated-array", b"[1,2"), + ("truncated-string", b'{"s":"text'), ("truncated-literal", b"[tru"), + ("missing-closing-brace", b'{"a":{"b":1}'), ("missing-closing-bracket", b'{"a":[1,2}'), + ("missing-colon", b'{"a" 1}'), ("missing-comma", b"[1 2]"), ("wrong-separator", b'{"a"=1}'), + ("trailing-array-comma", b"[1,]"), ("trailing-object-comma", b'{"a":1,}'), + ("invalid-escape", b'"\\q"'), ("short-unicode-escape", b'"\\u123"'), + ("unpaired-high-surrogate", b'"\\ud800"'), ("unpaired-low-surrogate", b'"\\udc00"'), + ("high-surrogate-with-nonsurrogate", b'"\\ud800\\u0041"'), + ("high-surrogate-with-high-surrogate", b'"\\ud800\\ud800"'), + ("low-surrogate-across-buffer", b" " * 16379 + b'"\\udc00"'), + ("invalid-surrogate-pair-across-buffer", b" " * 16377 + b'"\\ud800\\u0041"'), + ("unescaped-newline", b'"a\nb"'), ("utf8-continuation", b'"\x80"'), + ("utf8-overlong", b'"\xc0\xaf"'), ("utf8-truncated", b'"\xe2\x82"'), + ("utf8-surrogate", b'"\xed\xa0\x80"'), ("utf8-out-of-range", b'"\xf4\x90\x80\x80"'), + ("utf8-invalid-key", b'{"\xff":1}'), ("extra-document", b"{} []"), + ("extra-text", b'{"a":1}xyz'), ("extra-nul", b"{}\0"), + ("number-leading-zero", b"[01]"), ("number-leading-plus", b"[+1]"), + ("number-missing-integer", b"[.1]"), ("number-missing-fraction", b"[1.]"), + ("number-missing-exponent", b"[1e]"), ("number-missing-exponent-value", b"[1e+]"), + ("number-double-sign", b"[--1]"), ("number-nan", b"[NaN]"), + ("number-infinity", b"[Infinity]"), ("comment", b"{/* comment */}"), +] + + +def disable_core_dumps(): + resource.setrlimit(resource.RLIMIT_CORE, (0, 0)) + + +class Suite: + def __init__(self, binary, selected, available, temp): + self.binary, self.selected, self.available, self.temp = binary, selected, available, Path(temp) + self.passed = self.failed = self.skipped = self.differences = 0 + + def report(self, status, label): + print(f"{status:<5} {label}", flush=True) + field = {"PASS": "passed", "FAIL": "failed", "SKIP": "skipped", "DIFF": "differences"}[status] + setattr(self, field, getattr(self, field) + 1) + + def fixture(self, name, document): + path = self.temp / f"{name}.json" + path.write_bytes(document) + return path + + def run(self, engine, path, options=(), stdout=subprocess.PIPE): + command = [str(self.binary), "-e", engine, *map(str, options), str(path)] + try: + return subprocess.run(command, stdout=stdout, stderr=subprocess.PIPE, timeout=15, + preexec_fn=disable_core_dumps, check=False) + except subprocess.TimeoutExpired as error: + return subprocess.CompletedProcess(command, 124, error.stdout or b"", (error.stderr or b"") + b"\nTest timed out\n") + + @staticmethod + def arguments(result): + iterations = 1 + for index, arg in enumerate(result.args[:-1]): + if arg == "-n": + iterations = int(result.args[index + 1]) + # For -n>1 the CLI prints timing before the final output pass. + if iterations > 1: + match = TIMING_PREFIX.search(result.stdout) + return result.stdout[match.end():] if match else None + match = TIMING_SUFFIX.search(result.stdout) + return result.stdout[:match.start()] if match else None + + @staticmethod + def diagnostic(result, expected=None, actual=None): + print(f" exit status: {result.returncode}; command: {' '.join(result.args)}") + if result.stderr: + print(" stderr: " + repr(result.stderr[:1200])) + if expected is not None and actual is not None: + print(f" expected {len(expected)} output bytes, got {len(actual)}") + lines = difflib.unified_diff(expected.decode("utf-8", "backslashreplace").splitlines(True), + actual.decode("utf-8", "backslashreplace").splitlines(True), + fromfile="expected", tofile="actual") + for index, line in enumerate(lines): + if index == 12: + print(" ... diff truncated ...") + break + print(" " + repr(line.rstrip("\n")[:300])) + elif result.stdout: + print(" stdout: " + repr(result.stdout[:1200])) + + def success(self, engine, name, path, expected=None, options=(), silent=False): + label = f"[{engine}] {name}" + options = ["-a", "9999", *options, *(["-s"] if silent else [])] + result = self.run(engine, path, options) + actual = self.arguments(result) + if result.returncode != 0 or actual is None: + self.report("FAIL", label + (" (missing timing/output trailer)" if actual is None else "")) + self.diagnostic(result) + return None + if silent: + expected = b"" + if expected is not None and actual != expected: + self.report("FAIL", label + " (argument output mismatch)") + self.diagnostic(result, expected, actual) + return None + self.report("PASS", label) + return actual + + def rejected(self, engine, name, path, options=(), diagnostic=None, silent=True): + result = self.run(engine, path, ["-a", "9999", *options, *(["-s"] if silent else [])]) + self.rejected_result(engine, name, result, diagnostic) + + def rejected_result(self, engine, name, result, diagnostic=None): + # Signals/timeouts are failures, not successful validation. Unsupported + # warnings alone do not establish that malformed input was rejected. + errors = [line for line in result.stderr.splitlines() if b"does not support" not in line] + good = 0 < result.returncode < 124 and bool(b"\n".join(errors).strip()) + if diagnostic is not None: + good = good and diagnostic.lower() in result.stderr.lower() + if good: + self.report("PASS", f"[{engine}] {name}") + else: + self.report("FAIL", f"[{engine}] {name} (expected rejection with a diagnostic)") + self.diagnostic(result) + + def limit_rejected(self, engine, name, path, options, diagnostic=None, silent=True): + result = self.run(engine, path, ["-a", "9999", *options, *(["-s"] if silent else [])]) + warning = b"does not support max depth limit" if "depth" in name else b"does not support max arg num limit" + if 0 <= result.returncode < 124 and warning in result.stderr: + self.report("SKIP", f"[{engine}] {name} (adapter reports unsupported limit; no enforcement claim)") + else: + self.rejected_result(engine, name, result, diagnostic) + + def baseline(self): + fixtures = ["test01.json", "test02.json", "test03.json", "test04.json", "test06.json", "512KB.json"] + reference = {} + if "YAJL" in self.available: + for fixture in fixtures: + result = self.run("YAJL", TESTS_DIR / fixture, ["-a", "9999"]) + output = self.arguments(result) + if result.returncode == 0 and output: + reference[fixture] = output + else: + self.report("FAIL", f"[YAJL] baseline reference: {fixture}") + self.diagnostic(result) + for engine in self.selected: + for fixture in fixtures: + path = TESTS_DIR / fixture + self.success(engine, f"existing valid parse/silent: {fixture}", path, silent=True) + if fixture in reference: + self.success(engine, f"existing argument output vs YAJL: {fixture}", path, reference[fixture]) + else: + self.report("SKIP", f"[{engine}] output comparison: {fixture} (YAJL unavailable; compatibility untested)") + self.rejected(engine, "reject existing invalid fixture: test05.json", TESTS_DIR / "test05.json") + self.limit_rejected(engine, "existing depth limit (-d 1)", TESTS_DIR / "test06.json", ["-d", "1"]) + self.limit_rejected(engine, "existing argument limit (-a 1)", TESTS_DIR / "test06.json", ["-a", "1"]) + + def matrix(self): + for engine in self.selected: + for case in VALID_CASES: + path = self.fixture(case.name, case.document) + self.success(engine, f"golden arguments ({case.count}): {case.name}", path, + case.output_for(engine), ["-a", str(max(1, case.count))]) + self.success(engine, f"silent validation: {case.name}", path, + options=["-a", str(max(1, case.count))], silent=True) + if case.count > 1: + self.limit_rejected(engine, f"argument count exceeds {case.count - 1}: {case.name}", + path, ["-a", str(case.count - 1)]) + for name, document in INVALID_CASES: + path = self.fixture(name, document) + self.rejected(engine, f"invalid JSON (silent): {name}", path) + self.rejected(engine, f"invalid JSON (output): {name}", path, silent=False) + self.boundaries(engine) + self.large_document(engine) + self.operational(engine) + self.compatibility() + + def compatibility(self): + if "JSONCONS" not in self.selected: + return + for case in VALID_CASES: + if "YAJL" not in self.available: + self.report("SKIP", f"[JSONCONS/YAJL] {case.name} (YAJL unavailable; compatibility untested)") + continue + path = self.fixture(case.name, case.document) + jsoncons = self.run("JSONCONS", path, ["-a", "9999"]) + yajl = self.run("YAJL", path, ["-a", "9999"]) + jsoncons_output, yajl_output = self.arguments(jsoncons), self.arguments(yajl) + observed = case.expected if case.yajl_observed is None else case.yajl_observed + if jsoncons.returncode != 0 or jsoncons_output != case.expected or yajl.returncode != 0 or yajl_output != observed: + self.report("FAIL", f"[JSONCONS/YAJL] comparison prerequisite: {case.name}") + self.diagnostic(jsoncons, case.expected, jsoncons_output) + self.diagnostic(yajl, observed, yajl_output) + elif jsoncons_output == yajl_output: + self.report("PASS", f"[JSONCONS/YAJL] byte-exact compatibility: {case.name}") + else: + self.report("DIFF", f"[JSONCONS/YAJL] {case.name} (both outputs checked; documented difference, no compatibility claim)") + + def config_value(self, name, fallback): + config = self.binary.parent.parent / "config.h" + if not config.is_file(): + config = TESTS_DIR.parent / "config.h" + if config.is_file(): + found = re.search(r"^#define\s+" + name + r"\s+([0-9]+)", config.read_text(), re.MULTILINE) + if found: + return int(found.group(1)) + return fallback + + def boundaries(self, engine): + depth = 64 + for kind, document, expected in [ + ("arrays", b"[" * depth + b"1" + b"]" * depth, + b".".join([b"array"] * (depth + 1)) + b": 1\n"), + ("objects", b'{"k":' * depth + b"1" + b"}" * depth, + b".".join([b"k"] * depth) + b": 1\n")]: + path = self.fixture("deep-" + kind, document) + self.success(engine, f"{kind} exactly at depth 64", path, expected, ["-d", str(depth)]) + self.limit_rejected(engine, f"{kind} exceed depth 63", path, ["-d", str(depth - 1)]) + default_depth = self.config_value("LIMIT_DEPTH", 500) + exact = self.fixture("default-depth", b"[" * default_depth + b"0" + b"]" * default_depth) + self.success(engine, f"default depth accepts {default_depth} containers", exact, + b".".join([b"array"] * (default_depth + 1)) + b": 0\n") + path = self.fixture("excessive-depth", b"[" * (default_depth + 1) + b"0" + b"]" * (default_depth + 1)) + self.limit_rejected(engine, f"default depth rejects {default_depth + 1} containers", path, []) + for root in [b"{}", b"[]", b"1\n"]: + self.success(engine, f"root at depth 1: {root!r}", self.fixture("depth-root", root), + b": 1\n" if root.startswith(b"1") else b"", ["-d", "1"]) + for name, document in [("booleans", b"[true,false]"), ("nulls", b"[null,null]")]: + self.limit_rejected(engine, f"{name} enforce argument limit with output", self.fixture(name, document), + ["-a", "1"], silent=False) + size = self.config_value("JSON_STRING_SIZE", 8192) + key, value, child = b"k" * (size - 1), b"v" * (size - 1), b"b" * (size - 3) + length_cases = [ + ("key", b'{"' + key + b'":1}', key + b": 1\n", b'{"' + key + b'k":1}'), + ("value", b'{"k":"' + value + b'"}', b"k: " + value + b"\n", b'{"k":"' + value + b'v"}'), + ("combined-path", b'{"a":{"' + child + b'":1}}', b"a." + child + b": 1\n", b'{"a":{"' + child + b'b":1}}')] + for name, document, expected, oversized in length_cases: + for silence in (False, True): + mode = "silent" if silence else "output" + self.success(engine, f"{name} length {size - 1} ({mode})", self.fixture(name, document), expected, silent=silence) + self.rejected(engine, f"{name} length {size} rejected ({mode})", self.fixture(name + "-long", oversized), silent=silence) + giant = b"x" * (2 * 1024 * 1024) + for name, document in [("huge-string", b'{"k":"' + giant + b'"}'), + ("huge-number", b'{"k":1' + b"0" * len(giant) + b"}"), + ("huge-unterminated-string", b'{"k":"' + giant)]: + self.rejected(engine, f"bounded hostile token (silent): {name}", self.fixture(name, document)) + self.rejected(engine, f"bounded hostile token (output): {name}", self.fixture(name, document), silent=False) + + def large_document(self, engine): + ceiling = self.config_value("FILE_BUFFER_SIZE", 10485760) + payload = b"x" * 512 + record = b'{"id":123,"pad":"' + payload + b'"}' + count = max(2000, (ceiling + 1024) // (len(record) + 1) + 1) + path = self.fixture("large-stream", b"[" + b",".join([record] * count) + b"]") + expected = (b"array.array.id: 123\narray.array.pad: " + payload + b"\n") * count + self.success(engine, f"large document ({path.stat().st_size} bytes, {count * 2} arguments)", path, + expected, ["-a", str(count * 2)]) + self.success(engine, "large document silent validation", path, options=["-a", str(count * 2)], silent=True) + self.limit_rejected(engine, "large document argument boundary", path, ["-a", str(count * 2 - 1)]) + + def operational(self, engine): + self.rejected(engine, "missing input file diagnostic", self.temp / "does-not-exist.json") + path = self.fixture("iterations", b'{"x":1,"x":2}') + self.success(engine, "three timed iterations print arguments once", path, b"x: 1\nx: 2\n", ["-n", "3", "-a", "2"]) + self.success(engine, "three silent iterations emit no arguments", path, options=["-n", "3", "-a", "2"], silent=True) + self.limit_rejected(engine, "timed iterations retain argument limit", path, ["-n", "3", "-a", "1"]) + for option in ("-d", "-a"): + for value in ("-1", "abc", "1x", "1.5", "4294967296", "0"): + self.rejected(engine, f"CLI rejects {option} {value}", path, [option, value]) + if Path("/dev/full").exists(): + for iterations in (1, 2): + with open("/dev/full", "wb") as output: + result = self.run(engine, path, ["-n", str(iterations)], stdout=output) + if 0 < result.returncode < 124 and result.stderr.strip(): + self.report("PASS", f"[{engine}] output error propagated (-n {iterations})") + else: + self.report("FAIL", f"[{engine}] output error ignored (-n {iterations})") + self.diagnostic(result) + else: + self.report("SKIP", f"[{engine}] output error test (/dev/full unavailable)") + + def timing(self): + try: + runs = int(os.environ.get("JSONBENCH_TIMING_RUNS", "0")) + except ValueError: + self.report("FAIL", "JSONBENCH_TIMING_RUNS must be a nonnegative integer") + return + if runs < 0: + self.report("FAIL", "JSONBENCH_TIMING_RUNS must be a nonnegative integer") + elif runs: + print(f"\n=== Timing: {runs} iterations per fixture, silent mode ===") + for fixture in ["test01.json", "test06.json", "512KB.json"]: + for engine in self.selected: + result = self.run(engine, TESTS_DIR / fixture, ["-s", "-a", "9999", "-n", str(runs)]) + if result.returncode != 0 or self.arguments(result) != b"": + self.report("FAIL", f"[{engine}] timing: {fixture}") + self.diagnostic(result) + else: + print(f"{engine:14} {fixture:12} {result.stdout.decode().strip()}") + + +def main(): + if len(sys.argv) not in (2, 3): + print("Usage: regression_tests.py BINARY [JSON_ENGINE]", file=sys.stderr) + return 1 + binary = Path(sys.argv[1]).resolve() + try: + result = subprocess.run([str(binary), "-h"], stdout=subprocess.PIPE, stderr=subprocess.PIPE, + timeout=15, preexec_fn=disable_core_dumps, check=False) + except (OSError, subprocess.TimeoutExpired) as error: + print(f"ERROR: cannot discover engines: {error}", file=sys.stderr) + return 1 + available = re.findall(r"^\s*-\s+([A-Z]+)\s*$", result.stdout.decode(), re.MULTILINE) + if result.returncode != 0 or not available: + print("ERROR: cannot discover engines from binary -h", file=sys.stderr) + return 1 + selected = available + if len(sys.argv) == 3: + if sys.argv[2] not in available: + print(f"ERROR: engine '{sys.argv[2]}' unavailable (available: {' '.join(available)})", file=sys.stderr) + return 1 + selected = [sys.argv[2]] + print(f"=== Available engines: {' '.join(available)} ===") + print(f"=== Selected engines: {' '.join(selected)} ===") + with tempfile.TemporaryDirectory(prefix="jsonbench-") as temp: + suite = Suite(binary, selected, available, temp) + suite.baseline() + suite.matrix() + suite.timing() + print(f"\n=== Results: {suite.passed} passed, {suite.failed} failed, {suite.skipped} skipped, " + f"{suite.differences} checked behavioral differences ===") + return 1 if suite.failed else 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tests/run_tests.sh b/tests/run_tests.sh index 316f940..3cb75ef 100755 --- a/tests/run_tests.sh +++ b/tests/run_tests.sh @@ -1,192 +1,22 @@ #!/usr/bin/env bash set -euo pipefail -BINARY="$(cd "$(dirname "$0")/.." && pwd)/src/jsonbench" TESTS_DIR="$(cd "$(dirname "$0")" && pwd)" -TMPDIR_LOCAL=$(mktemp -d "${TMPDIR:-/tmp}/jsonbench.XXXXXX") -trap 'rm -rf "$TMPDIR_LOCAL"' EXIT +BINARY="${JSONBENCH_BINARY:-$TESTS_DIR/../src/jsonbench}" if [[ ! -x "$BINARY" ]]; then - echo "ERROR: binary not found or not executable: $BINARY" - echo " Run './autogen.sh && ./configure && make' first" + echo "ERROR: binary not found or not executable: $BINARY" >&2 + echo " Run './autogen.sh && ./configure && make' first" >&2 exit 1 fi - -# Discover available engines from jsonbench -h output -ENGINES=() -while IFS= read -r line; do - if [[ "$line" =~ [[:space:]]-[[:space:]]([A-Z]+)$ ]]; then - ENGINES+=("${BASH_REMATCH[1]}") - fi -done < <("$BINARY" -h 2>&1 || true) - -if [[ ${#ENGINES[@]} -eq 0 ]]; then - echo "ERROR: no engines found in '$BINARY -h' output" +if ! command -v python3 >/dev/null 2>&1; then + echo "ERROR: Python 3 is required by the regression test helper" >&2 exit 1 fi - -# Optional: filter to a single engine -if [[ "${1:-}" != "" ]]; then - ENGINE_FILTER="$1" - FOUND=0 - for e in "${ENGINES[@]}"; do - [[ "$e" == "$ENGINE_FILTER" ]] && FOUND=1 - done - if [[ "$FOUND" -eq 0 ]]; then - echo "ERROR: engine '$ENGINE_FILTER' is not available (available: ${ENGINES[*]})" - exit 1 - fi - ENGINES=("$ENGINE_FILTER") +if [[ $# -gt 1 ]]; then + echo "Usage: $0 [JSON_ENGINE]" >&2 + exit 1 fi -PASS=0 -FAIL=0 -SKIP=0 - -# Valid JSON fixtures; test05.json is intentionally invalid and excluded here -VALID_FIXTURES=(test01.json test02.json test03.json test04.json test06.json 512KB.json) -INVALID_FIXTURES=(test05.json) - -# Use YAJL as the reference engine for output consistency; fall back to first available -REFERENCE_ENGINE="${ENGINES[0]}" -for e in "${ENGINES[@]}"; do - if [[ "$e" == "YAJL" ]]; then - REFERENCE_ENGINE="YAJL" - break - fi -done - -# Generate reference output for each valid fixture -echo "=== Reference engine: $REFERENCE_ENGINE ===" -for fixture in "${VALID_FIXTURES[@]}"; do - "$BINARY" -e "$REFERENCE_ENGINE" -a 9999 "$TESTS_DIR/$fixture" \ - > "$TMPDIR_LOCAL/ref_${fixture}.raw" 2>/dev/null - grep -Ev "^(Time:|$)" "$TMPDIR_LOCAL/ref_${fixture}.raw" \ - > "$TMPDIR_LOCAL/ref_${fixture}.out" || true -done -echo "" - -# --- Helpers --- - -pass() { echo "PASS $1"; PASS=$((PASS + 1)); } -fail() { echo "FAIL $1"; FAIL=$((FAIL + 1)); } -skip() { echo "SKIP $1"; SKIP=$((SKIP + 1)); } - -# Run a parse that must succeed (exit 0) -check_valid() { - local engine="$1" fixture="$2" - local label="[$engine] valid parse: $fixture" - local rc=0 - "$BINARY" -e "$engine" -s -a 9999 "$TESTS_DIR/$fixture" \ - > "$TMPDIR_LOCAL/out.txt" 2>&1 || rc=$? - if [[ "$rc" -eq 0 ]]; then pass "$label"; else fail "$label"; fi -} - -# Check that non-silent output matches the reference engine's output -check_consistency() { - local engine="$1" fixture="$2" - local label="[$engine] output consistency: $fixture" - local rc=0 - "$BINARY" -e "$engine" -a 9999 "$TESTS_DIR/$fixture" \ - > "$TMPDIR_LOCAL/cmp_${engine}_${fixture}.raw" 2>/dev/null || rc=$? - grep -Ev "^(Time:|$)" "$TMPDIR_LOCAL/cmp_${engine}_${fixture}.raw" \ - > "$TMPDIR_LOCAL/cmp_${engine}_${fixture}.out" || true - if [[ "$rc" -ne 0 ]]; then - fail "$label (parse failed)" - return - fi - if [[ ! -s "$TMPDIR_LOCAL/cmp_${engine}_${fixture}.out" ]]; then - skip "$label (engine produces no field output)" - return - fi - if diff -q "$TMPDIR_LOCAL/ref_${fixture}.out" \ - "$TMPDIR_LOCAL/cmp_${engine}_${fixture}.out" > /dev/null 2>&1; then - pass "$label" - else - fail "$label" - diff "$TMPDIR_LOCAL/ref_${fixture}.out" \ - "$TMPDIR_LOCAL/cmp_${engine}_${fixture}.out" | head -20 - fi -} - -# Run a parse that must fail (non-zero exit); handles "does not support" gracefully -# Usage: check_rejected engine label [extra args...] -check_rejected() { - local engine="$1" label="$2" - shift 2 - local rc=0 - "$BINARY" -e "$engine" -s "$@" \ - > "$TMPDIR_LOCAL/out.txt" 2>"$TMPDIR_LOCAL/err.txt" || rc=$? - if [[ "$rc" -ne 0 ]]; then - pass "[$engine] $label" - elif grep -q "does not support" "$TMPDIR_LOCAL/err.txt" 2>/dev/null; then - skip "[$engine] $label (engine does not support this limit)" - else - fail "[$engine] $label (expected non-zero exit, got 0)" - fi -} - -# --- Tests --- - -for engine in "${ENGINES[@]}"; do - # 1. Valid parse - for fixture in "${VALID_FIXTURES[@]}"; do - check_valid "$engine" "$fixture" - done - - # 2. Output consistency vs reference (skip self-comparison) - if [[ "$engine" != "$REFERENCE_ENGINE" ]]; then - for fixture in "${VALID_FIXTURES[@]}"; do - check_consistency "$engine" "$fixture" - done - fi - - # 3. Invalid JSON must be rejected - for fixture in "${INVALID_FIXTURES[@]}"; do - check_rejected "$engine" "rejects invalid JSON: $fixture" \ - "$TESTS_DIR/$fixture" - done - - # 4. Depth limit enforcement - check_rejected "$engine" "depth limit (-d 1) on test06.json" \ - -d 1 "$TESTS_DIR/test06.json" - - # 5. Arg limit enforcement - check_rejected "$engine" "arg limit (-a 1) on test06.json" \ - -a 1 "$TESTS_DIR/test06.json" -done - -# --- Summary --- -echo "" -echo "=== Results: $PASS passed, $FAIL failed, $SKIP skipped ===" - -# --- Timing table --- -BENCH_N=20 -echo "" -echo "=== Timing: mean of ${BENCH_N} runs (seconds, silent mode) ===" -echo "" - -printf "%-20s" "Fixture" -for engine in "${ENGINES[@]}"; do - printf "%12s" "$engine" -done -printf "\n" -printf "%-20s" "--------------------" -for engine in "${ENGINES[@]}"; do - printf "%12s" "------------" -done -printf "\n" - -for fixture in "${VALID_FIXTURES[@]}"; do - printf "%-20s" "$fixture" - for engine in "${ENGINES[@]}"; do - raw=$("$BINARY" -e "$engine" -s -a 9999 -n "$BENCH_N" \ - "$TESTS_DIR/$fixture" 2>/dev/null) || true - mean=$(printf '%s\n' "$raw" | grep "^Time:" | grep -oE "mean=[0-9.]+" | sed 's/mean=//') - printf "%12s" "${mean:-ERR}" - done - printf "\n" -done -echo "" - -[[ "$FAIL" -eq 0 ]] +# Standard-library helper, preserving the existing CLI and engine filter. +exec python3 "$TESTS_DIR/regression_tests.py" "$BINARY" "$@"