Repository navigation
Move debug testing over to ARM from X86 #41209
Description
The Red Hat team spent the day looking at how we could move the CI closer to green. One of the top issue causing failures reported in the reliability reports are the debug builds running out of memory (21 failures in last report). - See nodejs/reliability#146 as an example.
The container machine for x86 where this test runs already has 32G of memory but debug builds take a lot of memory and we have a number of containers that can all be running on that machine.
Our new arm machines form equinix metal have 512G memory which should be more than enough. We only have debug coverage on 1 platform/os so thinking an option might be to be to move our debug builds over to ARM. It looks like we have a similar number of container based agents running on the host, the main difference being that there should be a lot more memory.
I created a temporary job to see how the debug build runs on ARM. There are 2 consistent failures:
I'm still thinking it would be worth marking the 2 tests ask flaky (1 for main, 17.x,16,x and the other for 14.x) and move over the debug builds to see if that helps us get closer to green.
Any concerns with that approach ?
@nodejs/build, @Trott
Activity
Trott commentedon Dec 17, 2021
Not from me. Thanks for all the work on these CI issues!
sxa commentedon Dec 17, 2021
LGTM and while I'm ok with marking as flakey we should definitely look into why 41204 is occurring and whether it's related to core/dump settings of some sort on the ARM host.
test: mark test-worker-take-heapsnapshot flaky
mhdawson commentedon Dec 20, 2021
PR to mark 41204 as flaky -#41253. After that lands I'll see if I can configure the switch to arm just for the main branch so that we can validate we have better CI results before looking at backports
test: mark test-worker-take-heapsnapshot flaky
addaleax commentedon Dec 27, 2021
I don't remember the original concerns around why specifically it was that we added a debug build, but I don't think that it was about platform/compiler-specific issues. We now have two configure flags that build Node.js with the same code behavior as a full debug build, namely
--debug-node --v8-with-dchecks-- Consequently, for the debug builds in CI, using those flags is probably good enough. Those are fairly lightweight compared to a full debug build, so we could keep that configuration at least for x86?(And then either do nothing for arm, or add a
--debug-node --v8-with-dchecksbuilds for arm, or add a--debugbuild for arm, I don't care much, but if possible it would be nice not to lose debug coverage of the x86)test: mark test-worker-take-heapsnapshot flaky
test: mark test-worker-take-heapsnapshot flaky
test: mark test-worker-take-heapsnapshot flaky
github-actions commentedon Jun 25, 2026
This issue has been marked as stale due to 210 days of inactivity.
It will be automatically closed in 30 days if no further activity occurs. If this is still relevant, please leave a comment or update it to keep it open.
github-actions commentedon Jul 26, 2026
This issue has been automatically closed after 30 days of inactivity following its stale status (no activity for a total of 120 days).
If this is still relevant, feel free to reopen it or leave a comment with additional details so we can continue the discussion.