Repository navigation
Corrupted file read/write #50717
Description
We noticed some corrupted files on our system and basically it seems that file read/write does not seem to increment the file handle offset. Hence writes will always write to offset 0 and reads will always read from offset 0.
From preliminary testing the issue seems to have been introduced in v20.3. I suspect #48078 and libuv/libuv#3952.
Host kernel: 5.15.104-0-lts
Docker Daemon: 20.10.9
Docker Image: node:21.1.0-bullseye and 20.9.0-bullseye
The issue only seems to occur when writing/reading to a specific SMB mount.
volumes:
backup:
driver: local
driver_opts:
type: cifs
o: 'username=secret,password=secret'
device: '//somednsname/nxt_backup'
import fs from 'fs'
import assert from 'assert'
const srcPath = '/backup/backup/0/0/0/0006d3befc1c41.mp4/0-TPtirLqtxVa6LP'
const dstPath = srcPath + '.tmp'
let position = 0
const src = await fs.promises.open(srcPath, 'r')
const dst = await fs.promises.open(dstPath, 'wx+')
try {
while (true) {
const { bytesRead, buffer } = await src.read() // { position })
console.log('chunk', bytesRead, position)
if (bytesRead === 0) {
break
}
const chunk = buffer.subarray(0, bytesRead)
await dst.write(chunk) // { position })
position += bytesRead
}
await dst.sync()
} finally {
await dst.close()
await src.close()
}
const stat = fs.statSync(dstPath)
console.log('copied', dstPath, stat.size, position)
assert(position === stat.size, `${dstPath}: ${position} !== ${stat.size}`)If we instead pass an explicit position to write/read then everything works as expected.
This also affects, fs.createWriteStream and fs.createReadStream.
Activity
ronag commentedon Nov 13, 2023
@deadbeef84
ronag commentedon Nov 13, 2023
I'm hopeful https://github.com/libuv/libuv/releases/tag/v1.47.0 might resolve it. But until then this is quite a serious issue.
ronag commentedon Nov 13, 2023
@nodejs/fs
ronag commentedon Nov 13, 2023
@bnoordhuis
ronag commentedon Nov 14, 2023
UV_USE_IO_URING=0resolves the issue. We have applied this to all our services in production as a safety measure.ronag commentedon Nov 14, 2023
If necessary I can demo the issue on the production system where it occurs over a Zoom call.
santigimeno commentedon Nov 14, 2023
Does calling
await dst.write(chunk)help somehow?ronag commentedon Nov 14, 2023
Sorry that was a typo
santigimeno commentedon Nov 15, 2023
One more thing. Are you checking for partial writes?
ronag commentedon Nov 15, 2023
Partial writes should not be possible here. But even if they were, not sure it matters? The file size is exactly the highWaterMark and stays that size regardless how much many times write is called.
Also the read never completes and reads infinite amount of data.
santigimeno commentedon Nov 15, 2023
fwiw, I've setup a shared samba folder in my computer which I'm accessing via the cifs driver and couldn't reproduce the issue.
These are the specs:
Host Kernel: Ubuntu 5.15.0-88.98-generic 5.15.126
Docker version: Docker version 24.0.7
Docker Image: node:21.1.0-bullseye
ronag commentedon Nov 15, 2023
Hm. Must be some combination of kernel ver + docker ver + smb + ioring.
ronag commentedon Nov 15, 2023
We are working on upgrading kernel + docker version. However, it will take a few weeks before I can confirm whether or not that fixes the issue.
santigimeno commentedon Nov 15, 2023
In the meantime, can you try running your script for a bit with perf like this and post the results somewhere? There might be some useful info there:
ronag commentedon Nov 16, 2023
Unfortunately that doesn't seem to work so well:
Inside node bullseye container:
santigimeno commentedon Nov 16, 2023
What's happenning is that the perf version installed in the container doesn't match the host kernel version. You should manually build and install the correct perf version in the container (here's a useful link with instructions you can adapt to your use case). Also, remember running the container with the
--privilegedflag so perf can run.deadbeef84 commentedon Nov 23, 2023
Here's the result from perf running in the
node:20.3.0-bullseye-slimdocker image.output.log
santigimeno commentedon Nov 26, 2023
@deadbeef84 Thanks for the sending the results.
From what I understand, it's the write operation the issue here as it's always reading from offset 0. The read operation seems to be working just fine. See from this portion of the logs in bold how the offset in the reads are changing correctly while not in the writes:
I'm not sure that upgrading kernel versions will fix this as I was briefly looking at the commits in io_uring and cifs between those versions and not sure there's anything related. Worth giving it a try though.
In the meantime, I saw you're using a much larger file than the one I used in my testing. Is this happening with files of any size?richardlau commentedon Nov 29, 2023
Are you able to test with #50650 to see if the further kernel restrictions in libuv around when io_uring is enabled helps?
Rush commentedon Dec 29, 2023
I believe we encountered a potentially related issue but with WebSockets that intermittently disconnect in a production environment with more users using our platform. Doesn't happen in testing with one/two connections. We're using cWS https://github.com/encharm/cWS
The issue started happening after we updated to Ubuntu 22.04 LTS from 20.04 LTS. The update triggered the io uring code path which wasn't available in an older kernel.
I confirmed UV_USE_IO_URING=0 to fix the issue.
Will libuv 1.47.0 be backported into Node 20 LTS?
deadbeef84 commentedon Jan 2, 2024
Sorry, I think I ran a modified version with explicit
{ position }specified in the read call - otherwise the code would get stuck in an infinite loop - so this does indeed effect both read/write.santigimeno commentedon Jan 9, 2024
@deadbeef84 thanks for the response. I would like to keep investigating this but I would need to be able to reproduce your environment. How difficult would it be?
santigimeno commentedon Jan 9, 2024
@Rush, that seems an unrelated issue (though also related to io_uring). Could you open a different issue describing the issue and, if possible, a reproducer? Thanks!
github-actions commentedon May 27, 2026
This issue has been marked as stale due to 210 days of inactivity.
It will be automatically closed in 30 days if no further activity occurs. If this is still relevant, please leave a comment or update it to keep it open.