java.lang.OutOfMemoryError error occurs when StringScanner keeps storing << and scan results in an array. (JRuby only, not CRuby)

java.lang.OutOfMemoryError error occurs when StringScanner keeps storing << and scan results in an array.
(JRuby only, not CRuby)

No Problem Code

string_scanner_scan_string.rb

require 'strscan'

s = StringScanner.new('')

N = 100000

a = []
N.times do
  s.string = s.rest + "1234567890"
  a << s.scan(/.*/)
end

$ ruby string_scanner_scan_string.rb 
$

Problem Code

string_scanner_scan.rb

require 'strscan'

s = StringScanner.new('')

N = 100000

a = []
N.times do
  s << "1234567890"
  a << s.scan(/.*/)
end

$ ruby string_scanner_scan.rb 
Error: Your application used more memory than the automatic cap of 3758MB.
Specify -J-Xmx####M to increase it (#### = cap size in MB).
Specify -w for full java.lang.OutOfMemoryError: Java heap space stack trace

Estimated memory requirements

100,000 x 10 [Byte] = 1,000,000 [Byte] = 1 [MB]

The above is required for Array and StringScanner.

1 [MB] x 2 = 2 [MB]

3758MB is not needed, so a memory leak is likely to occur.

Environment

$ ruby -v
jruby 9.4.5.0 (3.1.4) 2023-11-02 1abae2700f Java HotSpot(TM) 64-Bit Server VM 25.121-b13 on 1.8.0_121-b13 +jit [x86_64-darwin]
$ gem list strscan

*** LOCAL GEMS ***

strscan (3.0.8 java, default: 3.0.7 java)

I ran this with a JVM flag to dump the heap and have some information:

$ jruby -J-XX:+HeapDumpOnOutOfMemoryError memory_bug.rb 
java.lang.OutOfMemoryError: Java heap space
Dumping heap to java_pid28134.hprof ...
Heap dump file created [4081053622 bytes in 6.000 secs]

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "main"

Analyzing the heap, I see all the String objects in the Array, but each one is holding a huge buffer from the original string.

I believe the problem happens like this:

StringScanner gets appended "1234567890" each loop
StringScanner#scan returns a shared string based on that buffer
Next loop modifies the string, so the buffer is copied to a new buffer
Old buffer is retained by the String returned from scan, so memory usage grows very quickly.

This is a case where copy-on-write is failing us; as the string grows, so do the retained buffers, and we end up with exponentially more byte[] data in memory than you would expect.

I'm not sure why this doesn't affect CRuby.

I'm not sure why this doesn't affect CRuby.

Aha. The C code does not make a shared string for #scan. I will make a PR.

Do not make shared strings for extracted ranges

Do not make shared strings for extracted ranges (#84)

@hsbt

This is fixed.

@hsbt Could we get a release with this please?

@headius

@headius

Aha. The C code does not make a shared string for #scan. I will make a PR.

There seems to be a problem with StringScanner#check, is this one ok?

string_scanner_check.rb

require 'strscan'

s = StringScanner.new('')

N = 100000

a = []
N.times do
  s << "1234567890"
  a << s.check(/.*/)
end

$ ruby string_scanner_check.rb 
Error: Your application used more memory than the automatic cap of 3743MB.
Specify -J-Xmx####M to increase it (#### = cap size in MB).
Specify -w for full java.lang.OutOfMemoryError: Java heap space stack trace

I checked this before the merge of #84.

@naitoh

@naitoh Uh-oh! That one still seems to be broken. I thought I caught everywhere this should be a copied string but I must have missed check. I'll do another PR.

I can release a new version when we're ready.

@naitoh

@naitoh False alarm! This has the same effect under CRuby, because check is returning a longer and longer string each time. Both implementations will consume many GB of memory, but JRuby hits the JVM limit and raises an error.

@headius

@headius

@naitoh False alarm! This has the same effect under CRuby, because check is returning a longer and longer string each time. Both implementations will consume many GB of memory, but JRuby hits the JVM limit and raises an error.

Sorry.
I have confirmed that CRuby and JRuby work similarly in the case of StringScanner#check.

This seems to be a reasonable behavior in this case.

@kou

@kou
Is it possible to make this release?

Done!

Use `@scanner << readline` instead of `@scanner.string = @scanner.res…

Navigation Menu

Repository navigation

`java.lang.OutOfMemoryError` error occurs when StringScanner keeps storing `<<` and `scan` results in an array. (JRuby only, not CRuby) #83

No Problem Code

Problem Code

Estimated memory requirements

Environment

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

java.lang.OutOfMemoryError error occurs when StringScanner keeps storing << and scan results in an array. (JRuby only, not CRuby) #83

Description

No Problem Code

Problem Code

Estimated memory requirements

Environment

Activity

headius commented on Jan 20, 2024

headius commented on Jan 20, 2024

headius commented on Jan 20, 2024

naitoh commented on Jan 20, 2024

headius commented on Jan 20, 2024

kou commented on Jan 20, 2024

headius commented on Jan 20, 2024

naitoh commented on Jan 20, 2024

naitoh commented on Jan 21, 2024

kou commented on Jan 21, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

`java.lang.OutOfMemoryError` error occurs when StringScanner keeps storing `<<` and `scan` results in an array. (JRuby only, not CRuby) #83