Skip to content

Remove the 500-token cap from the nested set operation lookahead - #2780

Merged
manticore-projects merged 1 commit into
JSQLParser:masterfrom
edeadington:fix/nested-set-operation-lookahead-limit
Oct 13, 2026
Merged

manticore-projects merged 1 commit into
JSQLParser:masterfrom
edeadington:fix/nested-set-operation-lookahead-limit

Conversation

@edeadington

@edeadington edeadington commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

isNestedSetOperationAhead() decides whether ((SELECT ...) UNION ...) is a parenthesised set operation or a parenthesised FROM item. It scans for the ) that closes the first branch but gives up after 500 tokens, so when that branch is longer the parser takes the FROM item path and fails on the set operator:

SELECT * FROM ((SELECT a FROM t WHERE <500+ tokens>) UNION ALL (SELECT a FROM t2)) AS u
-- Encountered: <K_UNION>

The same applies to x IN ((SELECT ...) UNION (SELECT ...)). 500 tokens is only a few lines of generated SQL.

The cap was most likely added to bound performance: the scan called getToken(i) for every position, and getToken(i) walks i tokens from the current position on each call, so an uncapped scan is quadratic.

This PR walks the token chain once using the existing skipBalancedBracketGroup() helper, so the scan is linear and needs no cap. It's also faster than the capped scan for short branches.

Median parse time for the query above, same build with only the lookahead swapped:

First-branch tokens Current (capped) getToken(i) scan without cap This PR
106 0.23 ms 0.20 ms 0.17 ms
406 0.75 ms 0.75 ms 0.41 ms
4,006 fails 42 ms 3.1 ms
20,006 fails 1,010 ms 15 ms
80,006 fails 16,139 ms 68 ms

On timeouts: with the quadratic scan, removing the cap would let one lookahead blow through the default 8 s Feature.timeOut on large inputs (16 s at 80k tokens). The linear scan stays well inside it. skipBalancedBracketGroup() also checks interrupted, so a parse that does time out stops mid-scan, where the old loop ran on to 500 regardless.

Two regression tests cover the FROM and IN cases.

isNestedSetOperationAhead() gave up after 500 tokens, so a parenthesised
set operation whose first branch is longer than that was parsed as a
parenthesised FROM item and failed on the set operator. Walk the token
chain once instead of calling getToken(i) per position, which removes
the need for a cap.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@edeadington
edeadington marked this pull request as ready for review October 12, 2026 21:59
@manticore-projects
manticore-projects merged commit d709c4e into JSQLParser:master Oct 13, 2026
18 of 19 checks passed
@manticore-projects

Copy link
Copy Markdown
Contributor

Thank you for this correction, much appreciated!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants