Repository navigation
use stream instead of fs.readdir() #1471
Description
I have this question on SO:
https://stackoverflow.com/questions/52131509/read-large-directory-with-nodejs
say we have a directory of 10,000,000 items or whatever. Instead of reading all paths into memory, it would be nice to be able to read the directory about 15,000 items concurrently.
Does Node.js offer a way to do that? Even with Unix child processes, still not sure how to do this.
Activity
gireeshpunathil commentedon May 9, 2019
@ORESoftware -
the underlying capability that provides us the data is
readdir/getdents, and as long as those provide us the data in one shot, whether we stream it or not is immaterial, as the data is already in memory.In general terms, what this would mean is, where we don't have control over the volume of data that we receive from a source, making a streamified version do not make meaning?
gireeshpunathil commentedon May 9, 2019
on a different note, if child processes use shell commands and stream data to parent, it would be sub-optimal IMO, as we are just forcing the data to be streamed, but:
richardlau commentedon May 9, 2019
I think you're after nodejs/node#583 which AFAIK hasn't yet been implemented on the Node.js side. Possibly https://www.npmjs.com/package/fs-readdir-stream (from one of the comments in the issue)?
gireeshpunathil commentedon Jun 24, 2019
ping @ORESoftware
gireeshpunathil commentedon Jul 15, 2019
inactive, closing