RSS memory adjustment

[rambo-panda]

  const renderStrokesImage = async ([url, strokes, imageWH, color]) => {
    const image = await loadImages(url);

    const { imageWidth, imageHeight } = parseImgWh(imageWH),
      canvas = createCanvas(imageWidth, imageHeight),
      ctx = canvas.getContext("2d");

    ctx.drawImage(image, 0, 0, imageWidth, imageHeight);

    // ctx.lineto ......
    await drawStrokes(ctx, strokes, color);

    const dataUrl = await canvas.toDataURLAsync(
      'image/webp',
      90
    );

    return dataUrl;
  }

for(let i = 0; i < 2e2; i++) {
    renderStrokesImage(data);
});

// Log every 10s

child_process 74190 Memory usage by rss, 518.582272MB
child_process 74190 Memory usage by heapTotal, 9.084928MB
child_process 74190 Memory usage by heapUsed, 7.970952MB
child_process 74190 Memory usage by external, 2.716399MB
child_process 74190 Memory usage by arrayBuffers, 1.954131MB
================

child_process 74285 Memory usage by rss, 1453.232128MB
child_process 74285 Memory usage by heapTotal, 8.298496MB
child_process 74285 Memory usage by heapUsed, 7.252864MB
child_process 74285 Memory usage by external, 1.079374MB
child_process 74285 Memory usage by arrayBuffers, 0.317106MB
=================

it has been maintained at 1453.232128mb

---

• MacBook Pro (16-inch, 2019)
• 2.6 GHz 6C Intel Core i7
• 16 GB 2667 MHz DDR4

[Brooooooklyn]

I don't think the high usage of RSS memory is a memory leak 🤔

[rambo-panda]

I'm sorry;
my test cases are not accurate.
It seems that the oom caused by a large number of requests.
maybe l should be considered from the perspective of flow restriction

---

by the way:
looking forward to solving this problem (obsessive-compulsive disorder)

[rambo-panda]

relevant：
nodejs/node#12805

[rambo-panda]

@Brooooooklyn

Sorry to interrupt again; I solved it by exposing the global.gc, but I'm still a little confused;
I'm learning rust, so I don't quite understand your source code part；
In terms of performance, there is excessive memory usage in multi child process mode(fork), which is not reclaimed in time

---

Here is my code

In multi-child process mode, there will be memory contention and trigger OOM

---

2 child process; 400 tasks ; image ~2M webp

Very easy to reach 90%

Even at the end of the task, the memory usage is not less than 50%

Looking forward to your reply

[CypherHub]

@rambo-panda how did you solve it by exposing the global.gc? I call gc often but RSS grows to a point I get an out of memory error eventually...
{
"rss": "433.59 MB",
"heapTotal": "87.59 MB",
"heapUsed": "42.61 MB",
"external": "21.32 MB"
}

I've tried:

• --max-old-space-size=8192
• --expose-gc and calling gc

Any thoughts would be appreciated!

[rambo-panda]

@CypherHub
the --max-old-space-size option does not work for this. Because it is not strictly a memory leak. in other worlds “this matter has nothing to do with heap memory”. adjustment the old space size simply expands the size of the js heap, and instead increases the gc time.

---

the essence of the problem is just as @Brooooooklyn changed the name of the title;
👍🏻

1. rss grows too fast
2. toDataURLAsync process.send will temporarily “lock up” the memory
3. too many sub-processes can also take up memory
4. too many requests 😄😄😄

taken together, this gives us the illusion of a memory leak

---

Here is my code(hope it is useful for you); my task queue has no strict time requirements

const getCanvas = (() => {
  const getRss = isFunction(process.memoryUsage.rss)
      ? process.memoryUsage.rss
      : () => process.memoryUsage().rss,
    GC_LIMIT = 1024 * 1024 * 820, // 820M   we will start 8 child_process  -- 8G memory
    $gc = async () => {
      if (getRss() > GC_LIMIT) {
        global.gc?.({ type: "major", execution: "async" }); // Return an promise

        await sleep(10e3);
      }
    };

  setInterval($gc, 15e3).unref();

  return async (width, height) => {
    await $gc();

    const c = createCanvas(width, height);
    c.getContext("2d");

    return c;
  };
})();

  const renderStrokesImage = async ([w, h]) => {
    const canvas = await getCanvas(w, h),
        {ctx } = canvas;
    ....
  }