Repository navigation
Nightly rhel8-arm64-release build failing #4172
Description
Starting on 5 October, the rhel8-arm64-release nightly builds are failing,
e.g. https://ci-release.nodejs.org/job/iojs+release/11302/nodes=rhel8-arm64-release/
11:04:01 LD_LIBRARY_PATH=/home/iojs/build/ws/out/Release/lib.host:/home/iojs/build/ws/out/Release/lib.target:$LD_LIBRARY_PATH; export LD_LIBRARY_PATH; cd ../tools/v8_gypfiles; mkdir -p /home/iojs/build/ws/out/Release/obj.target/v8_snapshot/geni; "/home/iojs/build/ws/out/Release/mksnapshot" --turbo_instruction_scheduling --stress-turbo-late-spilling "--target_os=linux" "--target_arch=arm64" --startup_src "/home/iojs/build/ws/out/Release/obj.target/v8_snapshot/geni/snapshot.cc" --embedded_variant Default --embedded_src "/home/iojs/build/ws/out/Release/obj.target/v8_snapshot/geni/embedded.S" --concurrent-builtin-generation "--concurrent-turbofan-max-threads=0" --no-native-code-counters
11:04:01
11:04:01
11:04:01 #
11:04:01 # Fatal error in , line 0
11:04:01 # Check failed: 12 == (*__errno_location ()).
11:04:01 #
11:04:01 #
11:04:01 #
11:04:01 #FailureMessage Object: 0xffffcc1eecf0
11:04:01 ==== C stack trace ===============================
11:04:01
11:04:01 /home/iojs/build/ws/out/Release/mksnapshot(v8::base::debug::StackTrace::StackTrace()+0x18) [0x359e340]
11:04:01 /home/iojs/build/ws/out/Release/mksnapshot() [0x1e56874]
11:04:01 /home/iojs/build/ws/out/Release/mksnapshot(V8_Fatal(char const*, ...)+0x1d8) [0x1e48f88]
11:04:01 /home/iojs/build/ws/out/Release/mksnapshot(v8::base::OS::SetPermissions(void*, unsigned long, v8::base::OS::MemoryPermission)+0x94) [0x1e4fbc0]
11:04:01 /home/iojs/build/ws/out/Release/mksnapshot(v8::base::VirtualAddressSubspace::AllocatePages(unsigned long, unsigned long, unsigned long, v8::PagePermissions)+0x64) [0x1e4f184]
11:04:01 /home/iojs/build/ws/out/Release/mksnapshot(v8::internal::ExternalEntityTable<v8::internal::JSDispatchEntry, 268435456ul>::Initialize()+0x9c) [0x15654a8]
11:04:01 /home/iojs/build/ws/out/Release/mksnapshot(v8::internal::IsolateGroup::InitializeOncePerProcess()+0x90) [0x15655e4]
11:04:01 /home/iojs/build/ws/out/Release/mksnapshot(v8::internal::V8::Initialize()+0x1d4) [0x1566790]
11:04:01 /home/iojs/build/ws/out/Release/mksnapshot(v8::V8::Initialize(int)+0x20) [0x1321650]
11:04:01 /home/iojs/build/ws/out/Release/mksnapshot(main+0x8cc) [0x130549c]
11:04:01 /lib64/libc.so.6(__libc_start_main+0xdc) [0xffffa6384384]
11:04:01 /home/iojs/build/ws/out/Release/mksnapshot() [0x1302410]
11:04:01 /bin/sh: line 1: 2604538 Trace/breakpoint trap (core dumped) "/home/iojs/build/ws/out/Release/mksnapshot" --turbo_instruction_scheduling --stress-turbo-late-spilling "--target_os=linux" "--target_arch=arm64" --startup_src "/home/iojs/build/ws/out/Release/obj.target/v8_snapshot/geni/snapshot.cc" --embedded_variant Default --embedded_src "/home/iojs/build/ws/out/Release/obj.target/v8_snapshot/geni/embedded.S" --concurrent-builtin-generation "--concurrent-turbofan-max-threads=0" --no-native-code-counters
11:04:01 make[2]: *** [tools/v8_gypfiles/v8_snapshot.target.mk:17: 9ea847fdb69e2e49207cd1a7d9992a6b592d15fd.intermediate] Error 133
11:04:01 rm 97b3b35c0261bedd03bbbab54f6b138f20a11b4f.intermediate 0d2ce10ea44d84275198a857d5f920d0af8bab0c.intermediate 9ea847fdb69e2e49207cd1a7d9992a6b592d15fd.intermediate d8e3a47e1673322d324ac597a559f10bdc575bdf.intermediate
11:04:01 make[1]: *** [Makefile:135: node] Error 2
11:04:01 make: *** [Makefile:1316: node-v25.0.0-nightly2025100532851f31c2-linux-arm64.tar] Error 2
This only appears to be affecting the nightly build from main -- the V8 nightly builds are still passing for rhel8-arm64-release.
Activity
richardlau commentedon Oct 6, 2025
https://ci-release.nodejs.org/job/iojs+release/nodes=rhel8-arm64-release/11300/ passed on nodejs/node@6f941fc.
https://ci-release.nodejs.org/job/iojs+release/nodes=rhel8-arm64-release/11302/ failed on nodejs/node@32851f3.
nodejs/node@6f941fc...32851f3 includes the V8 14.1 update.
richardlau commentedon Oct 6, 2025
AFAIK we're not seen anything like this in the test CI.
richardlau commentedon Oct 8, 2025
I updated and rebooted release-osuosl-rhel8-arm64-1 but problem persists: https://ci-release.nodejs.org/job/iojs+release/nodes=rhel8-arm64-release/11310/console
I might try wiping out the ccache in case it is a cache issue, but I might wait for the release build for 24.10.0 as wiping out the cache will make builds take longer.
richardlau commentedon Oct 9, 2025
I've wiped out the ccache cache and rerunning today's (still failing) nightly:
https://ci-release.nodejs.org/job/iojs+release/11314/nodes=rhel8-arm64-release/
richardlau commentedon Oct 9, 2025
Clearing the ccache hasn't made a difference and
mksnapshotis still failing a check. This doesn't fail in the test CI and AFAICT we're on the same compiler (clang 19). I'm going to need help to progress this further.richardlau commentedon Oct 10, 2025
Questions:
mainin the release CI but not forv8-canaryfrom https://github.com/nodejs/node-v8.git?I've just found nodejs/node#55509, which is a similar check failure from
v8::base::OS::SetPermissionsalbeit that was reported on x64 and this is arm64.richardlau commentedon Oct 10, 2025
If we're in
v8::base::OS::SetPermissionsI wonder if this is failing the CHECK_EQ at https://github.com/nodejs/node/blob/75a6fff6befa00dd4338bca5dd260a7478989965/deps/v8/src/base/platform/platform-posix.cc#L557?at least
ENOMEMis 12 on this platform (not sure howerrnowould map to(*__errno_location ())). Maybe this is memory related (I think the release machine has less available memory than the test machines)?richardlau commentedon Oct 10, 2025
FWIW
Machine has 4GB of swap:
richardlau commentedon Oct 10, 2025
There's also nodejs/help#4465, which includes
stracecommands to run for further debugging. There's references there to aMemoryDenyWriteExecutesetting in the systemd file in that issue but I checked the file for jenkins on the machine and didn't find any equivalent set.richardlau commentedon Oct 10, 2025
It might be https://issues.chromium.org/issues/425634685.
richardlau commentedon Oct 10, 2025
So this looks plausible and has supporting circumstantial evidence.
Going back further in history:
canarybranch build on 22 September failed in the same way .canarybuild on 24 September and onwards passed.canaryback to 13 September (earliest I can see history for) failed in the same way.v8/v8@1e190bb landed upstream V8 on 23 September. It's not in the version of V8 we currently have on
main.I'm currently rebuilding on the release machine and planning to see if applying v8/v8@1e190bb fixes the failure. I'm not going to be around much over the weekend, so if the rebuild doesn't complete in a timely manner today I might just open the backport PR.
targos commentedon Oct 10, 2025
Amazing investigation!
richardlau commentedon Oct 10, 2025
More supporting evidence -- the release machine is running a 64K page size, while the host machine for the rhel8-arm64 containers are running with 4K page size:
Rebuild on the release machine is... still building (without the patch applied). Rather than wait, I've opened nodejs/node#60206.
I honestly started today thinking that I had no idea how to continue investigating this (see my comment from yesterday about needing help). Just shows what a little persistence can do 🙂.
richardlau commentedon Oct 10, 2025
Took a while to rebuild, but applying the changes from v8/v8@1e190bb does appear to fix the build:
deps: V8: cherry-pick 1e190bbb0396
deps: V8: cherry-pick 1e190bbb0396