http2 client stream processing is 20x slower than curl #50972

Activity

1. DavyLandman commented on Nov 30, 2023

   

   DavyLandman

   on Nov 30, 2023

   Last edited by DavyLandman

   ContributorAuthor

   More actions

   Ok, so even though I read all the issues about http2 in this repo, and specifically this one: #38426 , I didn't manage to have it in the right combination of settings.

   The fix was that the client needs to also do settings: { initialWindowSize: 8 * 1024 * 1024} in it's connect call. not just on the server side.

   But I think it's a dupe of #38426. I think it's time to at least tune that window a bit. 8MB was enough to get to close to curl speeds.

   However, this doesn't fix it in the other direction, so when sending a large stream to the server.
2. DavyLandman commented on Nov 30, 2023

   

   DavyLandman

   on Nov 30, 2023

   Last edited by DavyLandman

   ContributorAuthor

   More actions

   So I think there is still an issue, I would love to help debug whats going on. The initial issue I wrote it for downloading to the client, since thats a bit easier to write. but if we use the stream in "reverse" the fixes of correctly setting the initialWindowSize and the localWindowSize do not work. Interesting fact, curl also struggles with that direction.

   Here is a combination of client & server that do the same after each other:

   server:

   const fs = require("fs");
   const stream = require("stream");
   const http2 = require("http2");
   const crypto = require("crypto");
   
   const server = http2.createSecureServer({
     key: fs.readFileSync("./localhost-privkey.pem"),
     cert: fs.readFileSync("./localhost-cert.pem"),
     allowHTTP1: true, // http2 upgrade
     settings: { initialWindowSize: 8 * 1024 * 1024} 
   });
   
   server.on('error', (err) => console.log(err));
   
   const buffer = crypto.randomBytes(256*1024*1024);
   
   server.on("connect", (session) => {
     session.setLocalWindowSize(8*1024*1024);
   });
   
   server.on("stream", (st, headers) => {
     st.respond({
       'content-type': 'application/octet-stream',
       ":status": 200,
     });
     if (headers[':path'] === "/download") {
       stream.Readable.from(buffer).pipe(st);
     }
     else {
       st.on('data', d => st.resume());
       st.on('error', console.error);
       st.on('end', () => st.close());
     }
   })
   
   server.listen(8443, () => {
     console.log("HTTPS (HTTP/2) listening on 8443 port");
   });

   client:

   const fs = require("fs");
   const http2 = require("http2");
   const process = require("process");
   const crypto = require("crypto");
   const stream = require("stream");
   
   const buffer = crypto.randomBytes(256*1024*1024);
   
   function upload() {
     const client = http2.connect(process.env["TARGET_HOST"] ?? "https://localhost:8443", {
       ca: fs.readFileSync("./localhost-cert.pem"),
       checkServerIdentity: () => undefined,
       settings: { initialWindowSize: 8 * 1024 * 1024}
     });
   
     client.once('remoteSettings', settings => {
       client.setLocalWindowSize(settings.initialWindowSize ?? (8 * 1024 * 1024));
     });
   
     const recv = client.request({":path": "/upload"}, {endStream: false});
   
     recv.on("response", () => {
       const start = process.hrtime();
       stream.Readable.from(buffer)
         .pipe(recv)
         .on("end", () => {
           const [timeSpendS, timeSpendNS] = process.hrtime(start);
           const mbSize = buffer.length / (1024 * 1024);
           console.log(
             `Uploading to h2 speed (${Math.trunc(mbSize)} MB): ${
               mbSize / (timeSpendS + timeSpendNS * 1e-9)
             } MB/s`
           );
           client.close();
           round++;
           if (round < 3) {
             // to give node a fair chance of getting close to curl
             // we go throught the upload 3 times (sequentually).
             // such that node has had time to JIT etc.
             upload();
           }
         });
     });
   }
   
   let round = 0;
   function download() {
     const client = http2.connect(process.env["TARGET_HOST"] ?? "https://localhost:8443", {
       ca: fs.readFileSync("./localhost-cert.pem"),
       checkServerIdentity: () => undefined,
       settings: { initialWindowSize: 8 * 1024 * 1024}
     });
   
     client.once('remoteSettings', settings => {
       client.setLocalWindowSize(8 * 1024 * 1024);
     });
   
     const recv = client.request({":path": "/download"});
   
   
     recv.on("response", () => {
       const start = process.hrtime();
       let received = 0;
       recv.on('data', d => {
         received += d.length;
       } );
       recv.on('error', console.error);
       recv.on('end', () => {
         const [timeSpendS, timeSpendNS] = process.hrtime(start);
         const mbSize = received / (1024 * 1024);
         console.log(
           `Downloading from h2 speed (${Math.trunc(mbSize)} MB): ${
             mbSize / (timeSpendS + timeSpendNS * 1e-9)
           } MB/s`
         );
         client.close();
         round++;
         if (round < 3) {
             // to give node a fair chance of getting close to curl
             // we do the download 3 times (sequentually).
             // such that node has had time to JIT etc.
           download();
         }
         else {
           // download is done, now do the upload
           round = 0;
           upload();
         }
       });
     });
   }
   
   download()

   localhost:

   Downloading from h2 speed (256 MB): 543.5856301100234 MB/s
   Downloading from h2 speed (256 MB): 606.9770638304905 MB/s
   Downloading from h2 speed (256 MB): 649.8966868629931 MB/s
   Uploading to h2 speed (256 MB): 334.71601949601046 MB/s
   Uploading to h2 speed (256 MB): 376.4662688384861 MB/s
   Uploading to h2 speed (256 MB): 428.22049972124603 MB/s
   

   curl:

    curl --http2 -k -T test.bin https://localhost:8443/upload --output /dev/null
     % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                    Dload  Upload   Total   Spent    Left  Speed
   100  400M    0     0  100  400M      0   379M  0:00:01  0:00:01 --:--:--  380M
   

   10gbit local network:

   $ NODE_ENV=production TARGET_HOST="https://A:8443" node h2client.js
   Downloading from h2 speed (256 MB): 311.8253894461286 MB/s
   Downloading from h2 speed (256 MB): 310.88049402369046 MB/s
   Downloading from h2 speed (256 MB): 334.249506006649 MB/s
   Uploading to h2 speed (256 MB): 17.867146518683615 MB/s
   Uploading to h2 speed (256 MB): 17.351565167350397 MB/s
   Uploading to h2 speed (256 MB): 17.98656213881376 MB/s
   

   curl:

   curl --http2 -k -T test.bin https://A:8443/upload --output /dev/null
     % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                    Dload  Upload   Total   Spent    Left  Speed
   100  400M    0     0  100  400M      0  11.5M  0:00:34  0:00:34 --:--:-- 11.4M
   

   So is there maybe some extra window size parameter I have yet to discover to get streams to upload at the same speed as the other direction?
3. DavyLandman commented on Dec 2, 2023

   

   DavyLandman

   on Dec 2, 2023

   ContributorAuthor

   More actions

   Closing this issue in favor of a much simpler variant of the same problem: #51014
4. DavyLandman

   closed this as completedon Dec 2, 2023