Repository navigation
org.yaml.snakeyaml.error.YAMLException: The incoming YAML document exceeds the limit: 3145728 code points. #7543
Description
Environment Information
jruby 9.3.9.0 (2.6.8) 2022-10-24 537cd1f OpenJDK 64-Bit Server VM 17.0.5+8-LTS on 17.0.5+8-LTS +jit [x86_64-darwin]
Darwin 21.6.0 Darwin Kernel Version 21.6.0: Sun Nov 6 23:31:16 PST 2022; root:xnu-8020.240.14~1/RELEASE_X86_64 x86_64
Expected Behavior
Using Psych::Parser to parse a large yaml file succeeds.
Actual Behavior
Using JRuby 9.3.9.0 parsing a large yaml file results in an exception:
org.yaml.snakeyaml.error.YAMLException: The incoming YAML document exceeds the limit: 3145728 code points.
Reverting to jruby-9.3.8.0 works.
Activity
chadlwilson commentedon Jan 8, 2023
This is due to #7388 and the 3MB code point limit now honored in SnakeYaml
1.32( https://bitbucket.org/snakeyaml/snakeyaml/wiki/Changes )Without looking at the specifics, suspect this might require ruby/psych#579 to workaround by setting
https://javadoc.io/static/org.yaml/snakeyaml/1.32/org/yaml/snakeyaml/LoaderOptions.html#setCodePointLimit-int-
donv commentedon Jan 9, 2023
Yeah, this looks right.
Any way to set the limit before ruby/psych#579 is done?
chadlwilson commentedon Jan 9, 2023
Apart from some hardcore runtime bytecode manipulation, I don't think so. Sadly the
LoaderOptionsdefaults appear hard coded and without any external way to change the default values JVM-wide (e.g static default holders, system properties).headius commentedon Jan 13, 2023
I have added some of these methods in ruby/psych#613. This is a unilateral exposure of these properties only in the JRuby version, so we should try to work with the maintainers of the C extension and see if we can have the same API for both.
headius commentedon Jan 26, 2023
I've requested review for my changes in ruby/psych#613. I would also like to release psych 5.1 to incorporate the new SnakeYAML Engine in ruby/psych#612, so this is a good time to do it.
Lock jruby version for tests
Lock jruby version for tests
updates-2023-01-31 (#106)
chadlwilson commentedon Feb 8, 2023
Looks like ability to control this was merged in ruby/psych#613, released in Psych
5.1.0which is part of JRuby9.4.1.0in #7626 so if that is all that is required on JRuby side, we might be able to update the milestone here (currently9.4.2.0) and close this unless it needs backport)?Also not sure if it needs some docs somewhere to show how to override them in normal use cases.
headius commentedon Feb 8, 2023
Oops, yup, this one should have been resolved as of 9.4.1.
headius commentedon Feb 8, 2023
There are no docs for the new features and no tests. Perhaps you could come up with some? I must admit I do not know exactly what YAML constructs the various settings apply to.
chadlwilson commentedon Feb 8, 2023
The irony is that my/our use case of JRuby actually doesn't rely on YAML parsing, Psych or SnakeYAML at all - it's just that we use
jruby-completeand I like reducing noise for the community from CVEs. Some other things I work on have direct SnakeYAML exposure so was familiar with some of the noise/risks in the area and interested in the overlap with JRuby world.I wonder if @donv has something on the test side.
Where do you suggest the docs would sit? Within Psych? Jruby itself somewhere?
donv commentedon Feb 8, 2023
Hi!
Documenting the usage here since I could not find it anywhere else yet.
I have switched to JRuby 9.4.1.0 in development and it seems to work fine.
headius commentedon Feb 8, 2023
@donv thanks for confirming!
headius commentedon Feb 8, 2023
@chadlwilson Well that is a good question. There is no logic in rdoc to generate documentation from the Java extension, and any way we don't have the same feature in the C extension so putting it in the general psych docs would not work anyway.
@hsbt @tenderlove How should we handle this? Maybe we can collaborate to get these same config options supported in the C extension?