fs.openAsBlob() does not work properly for files > 2GB

Version

v21.7.3 (and v20.12.2)

Platform

Darwin bender.local 23.4.0 Darwin Kernel Version 23.4.0: Fri Mar 15 00:10:42 PDT 2024; root:xnu-10063.101.17~1/RELEASE_ARM64_T6000 arm64

Subsystem

fs

What steps will reproduce the bug?

Steps to reproduce bug:

Generate a 3 GB file with dd if=/dev/random of=random.bin bs=1048576 count=3072
Launch node (v21.7.3 and v20.12.2 both repro for me)
const b = await fs.openAsBlob("random.bin")
b.slice(2**31-2, 2**31-1) <-- works
b.slice(2**31-1, 2**31) <-- returns a slice of zero length
b.slice(2**31, 2**31+1) <-- crashes

Full textual output of doing this given below.

Also, for files greater than 4GB, the size of the blob is wrong. It appears to be exactly 4*(2**30) less than the actual size. For example, for a 5*(2**30) byte file, the blob size is reported as 1073741824, which is (2**30). And, for a file that is exactly 4*(2**30), it reports a size of zero.

How often does it reproduce? Is there a required condition?

Always reproduces.

What is the expected behavior? Why is that the expected behavior?

Can operate on the Blob normally.

What do you see instead?

Described above. See output below.

Additional information

$ dd if=/dev/random of=random.bin bs=1048576 count=3072
3072+0 records in
3072+0 records out
3221225472 bytes transferred in 4.213743 secs (764457033 bytes/sec)
$ ls -l random.bin
-rw-r--r--  1 jbrandt  staff  3221225472 Apr 18 15:04 random.bin
$ node -v
v21.7.3
$ node
Welcome to Node.js v21.7.3.
Type ".help" for more information.
> const b = await fs.openAsBlob("random.bin")
undefined
> b
Blob { size: 3221225472, type: '' }
> b.slice(2**31-2, 2**31-1)
Blob { size: 1, type: '' }
> b.slice(2**31-1, 2**31)
Blob { size: 0, type: '' }
> b.slice(2**31, 2**31+1)

  #  node[96070]: static void node::Blob::ToSlice(const FunctionCallbackInfo<v8::Value> &) at ../src/node_blob.cc:247
  #  Assertion failed: args[0]->IsUint32()

----- Native stack trace -----

 1: 0x1041dfc0c node::Assert(node::AssertionInfo const&) [/Users/jbrandt/.nvm/versions/node/v21.7.3/bin/node]
 2: 0x10589b4a4 node::Blob::ToSlice(v8::FunctionCallbackInfo<v8::Value> const&) (.cold.2) [/Users/jbrandt/.nvm/versions/node/v21.7.3/bin/node]
 3: 0x1041a8994 node::Blob::ToSlice(v8::FunctionCallbackInfo<v8::Value> const&) [/Users/jbrandt/.nvm/versions/node/v21.7.3/bin/node]
 4: 0x104c32a38 Builtins_CallApiCallbackGeneric [/Users/jbrandt/.nvm/versions/node/v21.7.3/bin/node]
 5: 0x104c30b84 Builtins_InterpreterEntryTrampoline [/Users/jbrandt/.nvm/versions/node/v21.7.3/bin/node]
 6: 0x104c30b84 Builtins_InterpreterEntryTrampoline [/Users/jbrandt/.nvm/versions/node/v21.7.3/bin/node]
 7: 0x104c2e8ac Builtins_JSEntryTrampoline [/Users/jbrandt/.nvm/versions/node/v21.7.3/bin/node]
 8: 0x104c2e594 Builtins_JSEntry [/Users/jbrandt/.nvm/versions/node/v21.7.3/bin/node]
 9: 0x1044daef8 v8::internal::(anonymous namespace)::Invoke(v8::internal::Isolate*, v8::internal::(anonymous namespace)::InvokeParams const&) [/Users/jbrandt/.nvm/versions/node/v21.7.3/bin/node]
10: 0x1044db0e0 v8::internal::Execution::CallScript(v8::internal::Isolate*, v8::internal::Handle<v8::internal::JSFunction>, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>) [/Users/jbrandt/.nvm/versions/node/v21.7.3/bin/node]
11: 0x10439ef68 v8::Script::Run(v8::Local<v8::Context>, v8::Local<v8::Data>) [/Users/jbrandt/.nvm/versions/node/v21.7.3/bin/node]
12: 0x1041c8d1c node::contextify::ContextifyScript::EvalMachine(v8::Local<v8::Context>, node::Environment*, long long, bool, bool, bool, v8::MicrotaskQueue*, v8::FunctionCallbackInfo<v8::Value> const&) [/Users/jbrandt/.nvm/versions/node/v21.7.3/bin/node]
13: 0x1041c87fc node::contextify::ContextifyScript::RunInContext(v8::FunctionCallbackInfo<v8::Value> const&) [/Users/jbrandt/.nvm/versions/node/v21.7.3/bin/node]
14: 0x104c32a38 Builtins_CallApiCallbackGeneric [/Users/jbrandt/.nvm/versions/node/v21.7.3/bin/node]
15: 0x104c30b84 Builtins_InterpreterEntryTrampoline [/Users/jbrandt/.nvm/versions/node/v21.7.3/bin/node]
16: 0x104c30b84 Builtins_InterpreterEntryTrampoline [/Users/jbrandt/.nvm/versions/node/v21.7.3/bin/node]
17: 0x104c30b84 Builtins_InterpreterEntryTrampoline [/Users/jbrandt/.nvm/versions/node/v21.7.3/bin/node]
18: 0x104c30b84 Builtins_InterpreterEntryTrampoline [/Users/jbrandt/.nvm/versions/node/v21.7.3/bin/node]
19: 0x104c30b84 Builtins_InterpreterEntryTrampoline [/Users/jbrandt/.nvm/versions/node/v21.7.3/bin/node]
20: 0x10a6531d0 
21: 0x10a6ad6ac 
22: 0x104c30b84 Builtins_InterpreterEntryTrampoline [/Users/jbrandt/.nvm/versions/node/v21.7.3/bin/node]
23: 0x104c30b84 Builtins_InterpreterEntryTrampoline [/Users/jbrandt/.nvm/versions/node/v21.7.3/bin/node]
24: 0x10a66b508 
25: 0x10a6767a8 
26: 0x10a66cb7c 
27: 0x10a65307c 
28: 0x10a6ad6ac 
29: 0x10a6a73f0 
30: 0x104c69e04 Builtins_GeneratorPrototypeNext [/Users/jbrandt/.nvm/versions/node/v21.7.3/bin/node]
31: 0x10a65c654 
32: 0x10a65307c 
33: 0x10a6ad6ac 
34: 0x10a66d390 
35: 0x10a6754b8 
36: 0x10a66e24c 
37: 0x10a675b48 
38: 0x10a64f78c 
39: 0x104c2e8ac Builtins_JSEntryTrampoline [/Users/jbrandt/.nvm/versions/node/v21.7.3/bin/node]
40: 0x104c2e594 Builtins_JSEntry [/Users/jbrandt/.nvm/versions/node/v21.7.3/bin/node]
41: 0x1044daef8 v8::internal::(anonymous namespace)::Invoke(v8::internal::Isolate*, v8::internal::(anonymous namespace)::InvokeParams const&) [/Users/jbrandt/.nvm/versions/node/v21.7.3/bin/node]
42: 0x1044da478 v8::internal::Execution::Call(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, int, v8::internal::Handle<v8::internal::Object>*) [/Users/jbrandt/.nvm/versions/node/v21.7.3/bin/node]
43: 0x1043b2404 v8::Function::Call(v8::Local<v8::Context>, v8::Local<v8::Value>, int, v8::Local<v8::Value>*) [/Users/jbrandt/.nvm/versions/node/v21.7.3/bin/node]
44: 0x104100fc0 node::InternalMakeCallback(node::Environment*, v8::Local<v8::Object>, v8::Local<v8::Object>, v8::Local<v8::Function>, int, v8::Local<v8::Value>*, node::async_context) [/Users/jbrandt/.nvm/versions/node/v21.7.3/bin/node]
45: 0x104117c00 node::AsyncWrap::MakeCallback(v8::Local<v8::Function>, int, v8::Local<v8::Value>*) [/Users/jbrandt/.nvm/versions/node/v21.7.3/bin/node]
46: 0x1042af900 node::StreamBase::CallJSOnreadMethod(long, v8::Local<v8::ArrayBuffer>, unsigned long, node::StreamBase::StreamBaseJSChecks) [/Users/jbrandt/.nvm/versions/node/v21.7.3/bin/node]
47: 0x1042b0f7c node::EmitToJSStreamListener::OnStreamRead(long, uv_buf_t const&) [/Users/jbrandt/.nvm/versions/node/v21.7.3/bin/node]
48: 0x1042b526c node::LibuvStreamWrap::OnUvRead(long, uv_buf_t const*) [/Users/jbrandt/.nvm/versions/node/v21.7.3/bin/node]
49: 0x1042b59a0 node::LibuvStreamWrap::ReadStart()::$_1::__invoke(uv_stream_s*, long, uv_buf_t const*) [/Users/jbrandt/.nvm/versions/node/v21.7.3/bin/node]
50: 0x104c1a390 uv__stream_io [/Users/jbrandt/.nvm/versions/node/v21.7.3/bin/node]
51: 0x104c21da4 uv__io_poll [/Users/jbrandt/.nvm/versions/node/v21.7.3/bin/node]
52: 0x104c0fda4 uv_run [/Users/jbrandt/.nvm/versions/node/v21.7.3/bin/node]
53: 0x1041016f0 node::SpinEventLoopInternal(node::Environment*) [/Users/jbrandt/.nvm/versions/node/v21.7.3/bin/node]
54: 0x10421f754 node::NodeMainInstance::Run(node::ExitCode*, node::Environment*) [/Users/jbrandt/.nvm/versions/node/v21.7.3/bin/node]
55: 0x10421f4d4 node::NodeMainInstance::Run() [/Users/jbrandt/.nvm/versions/node/v21.7.3/bin/node]
56: 0x10419eb48 node::Start(int, char**) [/Users/jbrandt/.nvm/versions/node/v21.7.3/bin/node]
57: 0x19e7060e0 start [/usr/lib/dyld]

----- JavaScript stack trace -----

1: slice (node:internal/blob:265:21)
2: REPL5:1:3
3: runInThisContext (node:vm:136:12)
4: defaultEval (node:repl:598:22)
5: bound (node:domain:432:15)
6: runBound (node:domain:443:12)
7: onLine (node:repl:927:10)
8: emit (node:events:531:35)
9: emit (node:domain:488:12)
10: [_onLine] (node:internal/readline/interface:416:12)


zsh: abort      node

FWIW, this also reproduces on a linux/arm64 container image running in Docker for Mac.

root@f7a53c9fe57c:/# uname -a
Linux f7a53c9fe57c 6.6.16-linuxkit #1 SMP Fri Feb 16 11:54:02 UTC 2024 aarch64 aarch64 aarch64 GNU/Linux

root@f7a53c9fe57c:/# node -v       
v20.12.2

root@f7a53c9fe57c:/# dd if=/dev/random of=random.bin bs=1048576 count=3072
3072+0 records in
3072+0 records out
3221225472 bytes (3.2 GB, 3.0 GiB) copied, 8.45959 s, 381 MB/s

root@f7a53c9fe57c:/# node
Welcome to Node.js v20.12.2.
Type ".help" for more information.
> const b = await fs.openAsBlob("random.bin")
undefined
> b
Blob { size: 3221225472, type: '' }
> b.slice(2**31-1, 2**31)
Blob { size: 0, type: '' }
> b.slice(2**31, 2**31+1)

  #  node[15]: static void node::Blob::ToSlice(const v8::FunctionCallbackInfo<v8::Value>&) at ../src/node_blob.cc:247
  #  Assertion failed: args[0]->IsUint32()

----- Native stack trace -----

 1: 0xcab09c node::Assert(node::AssertionInfo const&) [node]
 2: 0xc65b38 node::Blob::ToSlice(v8::FunctionCallbackInfo<v8::Value> const&) [node]
 3: 0xf1f228 v8::internal::FunctionCallbackArguments::Call(v8::internal::CallHandlerInfo) [node]
 4: 0xf1f9e8  [node]
 5: 0xf1fe00 v8::internal::Builtin_HandleApiCall(int, unsigned long*, v8::internal::Isolate*) [node]
 6: 0x189c964  [node]

----- JavaScript stack trace -----

1: slice (node:internal/blob:266:21)
2: REPL4:1:3
3: runInThisContext (node:vm:136:12)
4: defaultEval (node:repl:598:22)
5: bound (node:domain:432:15)
6: runBound (node:domain:443:12)
7: onLine (node:repl:927:10)
8: emit (node:events:530:35)
9: emit (node:domain:488:12)
10: [_onLine] (node:internal/readline/interface:416:12)


Aborted

@joelrbrandt

Hi @joelrbrandt , In the implementation of Blob.slice, it's hinted that the start and end should be within the uint32 range,
Since it is never documented and actually there allows Blob have index out of range uint32, should it be documented to make it clear or validate the range of index instead of using such as start | 0 and end | 0? /cc @nodejs/buffer

@kylo5aby

@kylo5aby thanks for your quick response (and sorry for my slow follow-up).

it's hinted that the start and end should be within the uint32 range

I'm a bit confused by this part of your response. Where is this hinted? In the node implementation?

I don't see any mention of a 32-bit limitation in the File API spec. So, it doesn't seem like the Blob.prototype.slice() API should be limited to the uint32 range. Also, if it were limited to the uint32 range, it should go up to 2**32, not 2*31. But, the .slice() API supports negative values (distance from end), so maybe you meant int32?

Independent of the issue with .slice(), node reports the wrong size for Blobs that are larger than 4GB.

Interestingly, if I get a stream of the Blob (with Blob.prototype.stream()), I can read all the bytes in Blobs larger than 4GB.

See below where I do the following:

Generate 5GB of random data
Hash that data with openssl
Launch node v20.12.2
Open that random data in node with fs.openAsBlob
Observe the incorrectly reported size of 1GB (instead of 5GB)
Construct a stream, and send all the bytes to a hash using async iteration of the stream
Digest the hash to exactly the same value returned by openssl

$ dd if=/dev/random of=random.bin bs=1048576 count=5120
5120+0 records in
5120+0 records out
5368709120 bytes transferred in 6.613665 secs (811760063 bytes/sec)
$ openssl dgst random.bin 
SHA256(random.bin)= f80fac6d9dc913e33d2c6a69783fd8fc3b4ed18f9c844aca801358e437096e6c
$ node
Welcome to Node.js v20.12.2.
Type ".help" for more information.
> b = await fs.openAsBlob("random.bin")
Blob { size: 1073741824, type: '' }
> b.size
1073741824
> const { createHash } = require("node:crypto")
undefined
> const h = createHash("sha256")
undefined
> const s = b.stream()
undefined
> for await (const a of s) {
... h.update(a);
... }
undefined
> h.digest("hex")
'f80fac6d9dc913e33d2c6a69783fd8fc3b4ed18f9c844aca801358e437096e6c'

Finally, here's a codepen that can be used to check that files > 4GB work fine in all major modern browsers (open console to see output): https://codepen.io/joelrbrandt/pen/wvZRXvY

Finally, if fs.openAsBlob() is going to have a 4GB limit (and Blob.prototype.slice() is going to have a 2GB limit), it would certainly be more ideal if they threw RangeError exceptions rather than crashing node on out-of-range values.

Buffer.alloc() does this:

> b = Buffer.alloc(2**33)
Uncaught:
RangeError [ERR_OUT_OF_RANGE]: The value of "size" is out of range. It must be >= 0 && <= 4294967296. Received 8_589_934_592
    at Function.alloc (node:buffer:389:3) {
  code: 'ERR_OUT_OF_RANGE'
}

@joelrbrandt

Hi @joelrbrandt, Thank you for the information.

For fs.openAsBlob(), if it read a file with size larger than 4GB, the generated blob has a wrong size as you mentioned:

Observe the incorrectly reported size of 1GB (instead of 5GB)

There maybe a bug in fs.openAsBlob(), it wrongly set the size. do you think it works as expected? @jasnell . AFAIK， new Blob() can correctly set the size.
2. the uint32 range is for Blob.slice instead of Blob, it allows negative index as you mentioned, and will converted to positive under the hood and check whether it under the uin32 range. I have a PR #52588 to try to solve if there is a large start or end(larger than 2^31)

b.slice(231-1, 231)
Blob { size: 0, type: '' }
b.slice(231, 231+1)
node[15]: static void node::Blob::ToSlice(const v8::FunctionCallbackInfov8::Value&) at ../src/node_blob.cc:247
Assertion failed: args[0]->IsUint32()

@kylo5aby

@kylo5aby

AFAIK, new Blob() can correctly set the size

I'm not sure if this is accurate, either. Trying to construct a new Blob from a set of ArrayBuffers that total a size greater than 4GB also fails:

$ node
Welcome to Node.js v18.13.0.
Type ".help" for more information.
> const arrs = []
undefined
> for (let i = 0; i < 5 ; i++) {
...     arrs.push(new Uint8Array(2**30))
... }
5
> arrs.reduce((acc, cur) => acc + cur.byteLength, 0)
5368709120
> const b = new Blob(arrs)
Uncaught:
RangeError [ERR_BUFFER_TOO_LARGE]: Cannot create a Buffer larger than 4294967295 bytes
    at __node_internal_captureLargerStackTrace (node:internal/errors:491:5)
    at new NodeError (node:internal/errors:400:5)
    at new Blob (node:internal/blob:165:13) {
  code: 'ERR_BUFFER_TOO_LARGE'
}

However, this code also works fine in a browser, resulting in a 5GB Blob.

You are right. In versions up to v21, on 64-bit systems, the maximum size of a Buffer was 4GB. However, in the version I last used, compiled from the main branch, this limit has been exceeded, that's why I said new Blob is ok to create a blob larger than 4GB.
Conversely, fs.openAsBlob can successfully create blobs larger than 4GB, but there might be issues with the size.

buffer: remove lines setting indexes to integer value

It appears to persist this issue for files >4GB in latest Node.js v23.3.0 and v20.18.1:

$ dd if=/dev/random of=random.bin bs=1048576 count=5120
$ ls -l random.bin
-rw-r--r-- 1 user user 5368709120 Dec 10 23:39 random.bin
$ node
Welcome to Node.js v23.3.0.
Type ".help" for more information.
> b = await fs.openAsBlob("random.bin")
Blob { size: 1073741824, type: '' }
> b.slice(2**31-1, 2**31)
Blob { size: 0, type: '' }

Also:

$ node
Welcome to Node.js v23.3.0.
Type ".help" for more information.
> arrs = []; for (let i = 0; i < 5; i++) { arrs.push(new Uint8Array(2**30)); }
5
> b = new Blob(arrs)
Blob { size: 5368709120, type: '' }
> b.slice(2**32-2, 2**32-1) // 4GB-2, 4GB-1
Blob { size: 1, type: '' }
> b.slice(2**32-1, 2**32-0) // 4GB-1, 4GB-0

  #  node[91790]: static void node::Blob::ToSlice(const FunctionCallbackInfo<v8::Value> &) at ../src/node_blob.cc:264
  #  Assertion failed: args[1]->IsUint32()

----- Native stack trace -----

 1: 0x1026a2ff0 node::Assert(node::AssertionInfo const&) [/Users/qser/.local/share/mise/installs/node/23.3.0/bin/node]
 2: 0x10449da7c node::Blob::ToSlice(v8::FunctionCallbackInfo<v8::Value> const&) (.cold.1) [/Users/qser/.local/share/mise/installs/node/23.3.0/bin/node]
 3: 0x10266c86c node::Blob::HasInstance(node::Environment*, v8::Local<v8::Value>) [/Users/qser/.local/share/mise/installs/node/23.3.0/bin/node]
 4: 0x103452978 Builtins_CallApiCallbackGeneric [/Users/qser/.local/share/mise/installs/node/23.3.0/bin/node]
 5: 0x103450838 Builtins_InterpreterEntryTrampoline [/Users/qser/.local/share/mise/installs/node/23.3.0/bin/node]
 6: 0x103450838 Builtins_InterpreterEntryTrampoline [/Users/qser/.local/share/mise/installs/node/23.3.0/bin/node]
 7: 0x10344e50c Builtins_JSEntryTrampoline [/Users/qser/.local/share/mise/installs/node/23.3.0/bin/node]
 8: 0x10344e1b0 Builtins_JSEntry [/Users/qser/.local/share/mise/installs/node/23.3.0/bin/node]
 9: 0x102a28760 v8::internal::(anonymous namespace)::Invoke(v8::internal::Isolate*, v8::internal::(anonymous namespace)::InvokeParams const&) [/Users/qser/.local/share/mise/installs/node/23.3.0/bin/node]
10: 0x102a28d48 v8::internal::Execution::CallScript(v8::internal::Isolate*, v8::internal::Handle<v8::internal::JSFunction>, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>) [/Users/qser/.local/share/mise/installs/node/23.3.0/bin/node]
11: 0x1028d3268 v8::Script::Run(v8::Local<v8::Context>, v8::Local<v8::Data>) [/Users/qser/.local/share/mise/installs/node/23.3.0/bin/node]
12: 0x102693098 node::contextify::ContextifyScript::EvalMachine(v8::Local<v8::Context>, node::Environment*, long long, bool, bool, bool, v8::MicrotaskQueue*, v8::FunctionCallbackInfo<v8::Value> const&) [/Users/qser/.local/share/mise/installs/node/23.3.0/bin/node]
13: 0x1026927ac node::contextify::ContextifyScript::RunInContext(v8::FunctionCallbackInfo<v8::Value> const&) [/Users/qser/.local/share/mise/installs/node/23.3.0/bin/node]
14: 0x103452978 Builtins_CallApiCallbackGeneric [/Users/qser/.local/share/mise/installs/node/23.3.0/bin/node]
15: 0x103450838 Builtins_InterpreterEntryTrampoline [/Users/qser/.local/share/mise/installs/node/23.3.0/bin/node]
16: 0x103450838 Builtins_InterpreterEntryTrampoline [/Users/qser/.local/share/mise/installs/node/23.3.0/bin/node]
17: 0x103450838 Builtins_InterpreterEntryTrampoline [/Users/qser/.local/share/mise/installs/node/23.3.0/bin/node]
18: 0x103450838 Builtins_InterpreterEntryTrampoline [/Users/qser/.local/share/mise/installs/node/23.3.0/bin/node]
19: 0x103450838 Builtins_InterpreterEntryTrampoline [/Users/qser/.local/share/mise/installs/node/23.3.0/bin/node]
20: 0x10a006cf4
21: 0x10a04082c
22: 0x103450838 Builtins_InterpreterEntryTrampoline [/Users/qser/.local/share/mise/installs/node/23.3.0/bin/node]
23: 0x103450838 Builtins_InterpreterEntryTrampoline [/Users/qser/.local/share/mise/installs/node/23.3.0/bin/node]
24: 0x10a033c4c
25: 0x10a03db94
26: 0x10a038ffc
27: 0x10a006ba0
28: 0x10a04082c
29: 0x10a038310
30: 0x1034902c4 Builtins_GeneratorPrototypeNext [/Users/qser/.local/share/mise/installs/node/23.3.0/bin/node]
31: 0x10a01d0a0
32: 0x10a006ba0
33: 0x10a04082c
34: 0x10a03a574
35: 0x10a039df8
36: 0x10a03be0c
37: 0x10a0417e4
38: 0x10a00adf0
39: 0x10344e50c Builtins_JSEntryTrampoline [/Users/qser/.local/share/mise/installs/node/23.3.0/bin/node]
40: 0x10344e1b0 Builtins_JSEntry [/Users/qser/.local/share/mise/installs/node/23.3.0/bin/node]
41: 0x102a28760 v8::internal::(anonymous namespace)::Invoke(v8::internal::Isolate*, v8::internal::(anonymous namespace)::InvokeParams const&) [/Users/qser/.local/share/mise/installs/node/23.3.0/bin/node]
42: 0x102a280a8 v8::internal::Execution::Call(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, int, v8::internal::Handle<v8::internal::Object>*) [/Users/qser/.local/share/mise/installs/node/23.3.0/bin/node]
43: 0x1028e4d64 v8::Function::Call(v8::Isolate*, v8::Local<v8::Context>, v8::Local<v8::Value>, int, v8::Local<v8::Value>*) [/Users/qser/.local/share/mise/installs/node/23.3.0/bin/node]
44: 0x1025c1c88 node::InternalMakeCallback(node::Environment*, v8::Local<v8::Object>, v8::Local<v8::Object>, v8::Local<v8::Function>, int, v8::Local<v8::Value>*, node::async_context, v8::Local<v8::Value>) [/Users/qser/.local/share/mise/installs/node/23.3.0/bin/node]
45: 0x1025d6b34 node::AsyncWrap::MakeCallback(v8::Local<v8::Function>, int, v8::Local<v8::Value>*) [/Users/qser/.local/share/mise/installs/node/23.3.0/bin/node]
46: 0x1027aedbc node::StreamBase::CallJSOnreadMethod(long, v8::Local<v8::ArrayBuffer>, unsigned long, node::StreamBase::StreamBaseJSChecks) [/Users/qser/.local/share/mise/installs/node/23.3.0/bin/node]
47: 0x1027b0fc0 node::EmitToJSStreamListener::OnStreamRead(long, uv_buf_t const&) [/Users/qser/.local/share/mise/installs/node/23.3.0/bin/node]
48: 0x1027b55e0 node::LibuvStreamWrap::OnUvRead(long, uv_buf_t const*) [/Users/qser/.local/share/mise/installs/node/23.3.0/bin/node]
49: 0x1027b5ce8 node::LibuvStreamWrap::ReadStart()::$_1::__invoke(uv_stream_s*, long, uv_buf_t const*) [/Users/qser/.local/share/mise/installs/node/23.3.0/bin/node]
50: 0x10343a180 uv__stream_io [/Users/qser/.local/share/mise/installs/node/23.3.0/bin/node]
51: 0x103442384 uv__io_poll [/Users/qser/.local/share/mise/installs/node/23.3.0/bin/node]
52: 0x10342ef20 uv_run [/Users/qser/.local/share/mise/installs/node/23.3.0/bin/node]
53: 0x1025c24e4 node::SpinEventLoopInternal(node::Environment*) [/Users/qser/.local/share/mise/installs/node/23.3.0/bin/node]
54: 0x1026eb3b0 node::NodeMainInstance::Run(node::ExitCode*, node::Environment*) [/Users/qser/.local/share/mise/installs/node/23.3.0/bin/node]
55: 0x1026eb14c node::NodeMainInstance::Run() [/Users/qser/.local/share/mise/installs/node/23.3.0/bin/node]
56: 0x1026615ec node::Start(int, char**) [/Users/qser/.local/share/mise/installs/node/23.3.0/bin/node]
57: 0x18d294274 start [/usr/lib/dyld]

----- JavaScript stack trace -----

1: slice (node:internal/blob:271:21)
2: REPL8:1:3
3: runInThisContext (node:vm:137:12)
4: defaultEval (node:repl:597:22)
5: bound (node:domain:433:15)
6: runBound (node:domain:444:12)
7: onLine (node:repl:926:10)
8: emit (node:events:525:35)
9: emit (node:domain:489:12)
10: [_onLine] (node:internal/readline/interface:416:12)


fish: Job 1, 'node' terminated by signal SIGABRT (Abort)

Tested on linux/amd64 and darwin/arm64.

1.2.5

Why hasn't this issue been resolved for several years? Is it really that difficult? Both deno and bunjs are correct!

Bun handles this by using a custom u52 type (Number.MAX_SAFE_INTEGER). Deno implements Blobs in a WebIDL extension (deno_webidl converts between unsigned long long and Number).

The Blob.size issue is because of the usage of V8::Uint32 in BlobFromFilePath

node/src/node_blob.cc

Lines 123 to 126 in 9c4ca0a

    
           Local<Value> vals[2]{ 
        
               blob->object(), 
        
               Uint32::NewFromUnsigned(env->isolate(), blob->length()), 
        
           };

Blob::ToSlice is also restricted to V8::Uint32

node/src/node_blob.cc

Lines 272 to 283 in 9c4ca0a

    
           void Blob::ToSlice(const FunctionCallbackInfo<Value>& args) { 
        
             Environment* env = Environment::GetCurrent(args); 
        
             Blob* blob; 
        
             ASSIGN_OR_RETURN_UNWRAP(&blob, args.This()); 
        
             CHECK(args[0]->IsUint32()); 
        
             CHECK(args[1]->IsUint32()); 
        
             size_t start = args[0].As<Uint32>()->Value(); 
        
             size_t end = args[1].As<Uint32>()->Value(); 
        
             BaseObjectPtr<Blob> slice = blob->Slice(env, start, end); 
        
             if (slice) 
        
               args.GetReturnValue().Set(slice->object()); 
        
           }

The existing CheckNumberToSize should be able to convert a V8::Number to a size_t:

node/src/node_buffer.cc

Lines 1640 to 1654 in 6a3d80f

    
           // Converts a number parameter to size_t suitable for ArrayBuffer sizes 
        
           // Could be larger than uint32_t 
        
           // See v8::internal::TryNumberToSize and v8::internal::NumberToSize 
        
           inline size_t CheckNumberToSize(Local<Value> number) { 
        
             CHECK(number->IsNumber()); 
        
             double value = number.As<Number>()->Value(); 
        
             // See v8::internal::TryNumberToSize on this (and on < comparison) 
        
             double maxSize = static_cast<double>(std::numeric_limits<size_t>::max()); 
        
             CHECK(value >= 0 && value < maxSize); 
        
             size_t size = static_cast<size_t>(value); 
        
           #ifdef V8_ENABLE_SANDBOX 
        
             CHECK_LE(size, kMaxSafeBufferSizeForSandbox); 
        
           #endif 
        
             return size; 
        
           }

This is my best guess on how to convert a size_t to a V8::Number:

size_t val = 1;
Number v8num = Number::New(env->isolate(), static_cast<double>(val));

This issue has been marked as stale due to 90 days of inactivity.
It will be automatically closed in 30 days if no further activity occurs. If this is still relevant, please leave a comment or update it to keep it open.

Still persist in Node.js v26.5.0. Same result as #52585 (comment)

docs(node): note openAsBlob bugs with files over 2 GiB (#1879)

Navigation Menu

Repository navigation

`fs.openAsBlob()` does not work properly for files > 2GB #52585

Version

Platform

Subsystem

What steps will reproduce the bug?

How often does it reproduce? Is there a required condition?

What is the expected behavior? Why is that the expected behavior?

What do you see instead?

Additional information

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	Local<Value> vals[2]{
	blob->object(),
	Uint32::NewFromUnsigned(env->isolate(), blob->length()),
	};

	void Blob::ToSlice(const FunctionCallbackInfo<Value>& args) {
	Environment* env = Environment::GetCurrent(args);
	Blob* blob;
	ASSIGN_OR_RETURN_UNWRAP(&blob, args.This());
	CHECK(args[0]->IsUint32());
	CHECK(args[1]->IsUint32());
	size_t start = args[0].As<Uint32>()->Value();
	size_t end = args[1].As<Uint32>()->Value();
	BaseObjectPtr<Blob> slice = blob->Slice(env, start, end);
	if (slice)
	args.GetReturnValue().Set(slice->object());
	}

	// Converts a number parameter to size_t suitable for ArrayBuffer sizes
	// Could be larger than uint32_t
	// See v8::internal::TryNumberToSize and v8::internal::NumberToSize
	inline size_t CheckNumberToSize(Local<Value> number) {
	CHECK(number->IsNumber());
	double value = number.As<Number>()->Value();
	// See v8::internal::TryNumberToSize on this (and on < comparison)
	double maxSize = static_cast<double>(std::numeric_limits<size_t>::max());
	CHECK(value >= 0 && value < maxSize);
	size_t size = static_cast<size_t>(value);
	#ifdef V8_ENABLE_SANDBOX
	CHECK_LE(size, kMaxSafeBufferSizeForSandbox);
	#endif
	return size;
	}

fs.openAsBlob() does not work properly for files > 2GB #52585

Description

Version

Platform

Subsystem

What steps will reproduce the bug?

How often does it reproduce? Is there a required condition?

What is the expected behavior? Why is that the expected behavior?

What do you see instead?

Additional information

Activity

joelrbrandt commented on Apr 18, 2024

kylo5aby commented on Apr 19, 2024

joelrbrandt commented on Apr 23, 2024

joelrbrandt commented on Apr 23, 2024

kylo5aby commented on Apr 23, 2024

joelrbrandt commented on Apr 23, 2024

kylo5aby commented on Apr 24, 2024

heguro commented on Dec 10, 2024

yarlc commented on Mar 5, 2026

tcyrus commented on Apr 25, 2026

github-actions commented on Jul 25, 2026

heguro commented on Jul 25, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

`fs.openAsBlob()` does not work properly for files > 2GB #52585