Equinix Move: Rebuild SmartOS Hosts

@bahamat

This is a sub step of
#3597

The two release and four testing smartOS hosts need to be replaced.
The current ones are running SmartOS 18 and SmartOS 20, which are both EOL.

In discussions with @bahamat It was determined that the likely targets for smartos should change to be SmartOS 21 and SmartOS 23

I plan to Provision the following hosts:

Once those are up I'll connect them to jenkins and we can run some preliminary tests to make sure they work and are accepting jobs.

Theres a non-zero chance that something has broken from smartos20 all the way to smartos23, so somebody with SmartOS/node knowledge is likely going to have to manage anticipated failures.

Are release hosts needed? We stopped building SmartOS a while ago.

@targos

I've provisioned the smartos test hosts and got ansible to run through end to end. #3737
That being said I don't yet have jenkins admin access to fully add these machines, nor do I really know the next steps to verify that tests can actually run on them.

I've held off on provisioning the release hosts because @targos makes a good point that we probably do not need to provision machines that are not used.

Discovered the supporting info that we do not need release nodes for smartos: #2168

I have added the four smartos nodes to jenkins, and re-ran ansible with their jenkins secret set, however I do not yet have infra level access so I cannot change the firewall settings to allow for these nodes to connect to jenkins to test and verify that they are functioning properly.

I added the IPs to the firewall rules. The 4 machines are connected to Jenkins now.

Great. I added them with the smartos21-64 and smartos23-64 labels, and altered the node-test-commit-smartos job to attempt to run on them. https://ci.nodejs.org/job/node-test-commit-smartos/

Both smartos21 and smartos23 are failing during the make step, but this is where I dont think I can be of much assistance in getting these jobs to successfully build on smartos -> Im unfamiliar with the build/compilation process, and Im unfamiliar with smartos in general.

Example failures:

https://ci.nodejs.org/job/node-test-commit-smartos/nodes=smartos23-64/54688/console
https://ci.nodejs.org/job/node-test-commit-smartos/nodes=smartos21-64/54688/console

Is there a point of contact we can ask in the smartos community to attempt to get these jobs to build?

@nodejs/platform-smartos PTAL.

Example failures:

https://ci.nodejs.org/job/node-test-commit-smartos/nodes=smartos23-64/54688/console

21:55:02 ../deps/v8/src/base/platform/platform-posix.cc:81:16: error: conflicting declaration of C function 'int madvise(caddr_t, std::size_t, int)'
21:55:02    81 | extern "C" int madvise(caddr_t, size_t, int);
21:55:02       |                ^~~~~~~
21:55:02 In file included from ../deps/v8/src/base/platform/platform-posix.cc:20:
21:55:02 /usr/include/sys/mman.h:268:12: note: previous declaration 'int madvise(void*, std::size_t, int)'
21:55:02   268 | extern int madvise(void *, size_t, int);
21:55:02       |            ^~~~~~~
21:55:02 make[2]: *** [tools/v8_gypfiles/v8_libbase.target.mk:209: /home/iojs/build/workspace/node-test-commit-smartos/nodes/smartos23-64/out/Release/obj.target/v8_libbase/deps/v8/src/base/platform/platform-posix.o] Error 1
21:55:02 rm 7a898a1853eba0e19e78f94553f35c8ae3ef2670.intermediate

looks like #3108 (comment) which was sort of machine configuration issue with the system header files.

It appears that smartos updated those header files

illumos/illumos-gate@df5cd01#diff-e7ee1077e2440c045ad16995cdc8ecba3b955d39cab24eaa5f022fe737a234eaL38

So If I understand correctly, v8 is checking whether or not it is compiling on solaris and accounting for atypical header file structure and in the meantime, smartos/illumos has updated the atypical header structure, causing v8 to no longer compile.

This also seems like only the first issue we'll encounter trying to get v8 to compile on modern SmartOS, and this will likely take somebody some dedicated time to get smartOS building again, and that also, this is upstream v8, and not nodejs itself that needs patching.

I'm unclear how to proceed here. I don't imagine that we'll be able to get this resolved before we need to remove the smartos18/smartos20 hosts, so it appears as though smartos tests are going to be broken for the forseeable future.

@ryanaslett

@ryanaslett I think we have patches for this. We can get @jperkin to take a look, but he’s currently out on vacation so it won’t be right away.

Yeh, we've been running with this patch for the nodejs pkgsrc builds for the last few years:

NetBSD/pkgsrc@527b29c

The problem is that you can't just update the prototypes, as that would break builds on older platforms, and there's no way to test which is available using the preprocessor.

I'd recommend applying a similar patch to just remove the prototypes (it's unclear why they were added in the first place but they certainly aren't required, at least on modern illumos or Solaris platforms) and probably the posix_madvise() chunk too.

@bahamat

@nodejs/build to facilitate this we have:

created this copy of the smartos job - https://ci.nodejs.org/job/node-test-commit-smartos-test-ryan/. It currently pulls additional patches from this PR - Jenkins: Add smartos stub patches and patch script #3753 and runs on smartos 21 and 23
I'm going to give @bahamat the ability to run that job unless there are any objections by Tomorrow. EDIT: also added @jperkin as mentioned below

The path forward would be:

Somebody from the smartos community (@bahamat to identify), works to get the patch to apply. Once the patch applies and the job passes we can update the PR and land.
One the build PR is landed we can move the changes from the test job into the regular job for smartos. They include the lines to pull/apply the patch script, and change to run on smartos 21 and 23 instead of the older versions.
If 2) cannot be completed by June 14 we will need to remove smartos testing from the PR testing job until it is.

@bahamat we'd also need you to test with the current Node.js versions which include 18, 20 and 22. The patch will either need to apply to all of them, or different patches will need to be applied for each version by the script. We can also configure the job to only run on a subset of those versions (for example latest) if that is what makes sense for the smartos community.

@mhdawson

@mhdawson The best person to handle that from my side is @jperkin.

@bahamat

@bahamat, you and @jperkin should now be able to run that job.

So it looks like we've entirely replicated Dan's previous debugging of this issue in https://sourceware.org/bugzilla/show_bug.cgi?id=32211

I can confirm that adding CXXFLAGS+= -fno-zero-initialized-in-bss in my test build fixes this for me. Is it easy enough for you to add this to the build environment or does this require a node patch?

It depends on the scope/impact.

The configuration where we run into this issue is Nodejs 18, on smartos23.

Will that flag impact any other Node versions? i.e. can it be set at the smartos23 host level, or does it need to be set only when Node18 is being tested? (also, I may be wrong, but Im pretty sure theres almost no chance that node would patch node 18 for this).

also, would it be okay to set that flag on the smartos 22 hosts as well? (the ones with binutils-2.39)

also, would it be okay to set that flag on the smartos 22 hosts as well? (the ones with binutils-2.39)

Yes, this can be applied to all builds.

I'll set it at the jenkins level as an injected env variable for smartos builds.

Ah, re-reading this I realize that its merely adding that to the end of the CXXFLAGS variable. I may have to re-evaluate how we do this.

Adjusted
I would kick off a daily build to verify that this works, but the build queue is currently tremendously clogged because we've had smartos testing machines up for one whole day and the ccaches got invalidated immediately by the work in this PR. (nodejs/node#56275). That kind of thing is expected, I just wish it didnt happen less than 24 hours after getting these machines up.

It does underscore an issue with these machines : the tremendous discrepancy between the ccache build times (reasonable < 1 hour builds) and fresh builds (8-11 hours depending).

All SmartOS builds are failing with g++: error: unrecognized command-line option '-fno-zero-initialized-in-bss -fno-zero-initialized-in-bss' now

@ryanaslett

@ryanaslett any chance you had time to look at the compilation issue mentioned in my previous comment? SmartOS builds are failing on all PRs rn, which is frustrating

Somehow that option got in there twice, and I think that’s what’s causing this.

@aduh95

@aduh95 That was an error in how jenkins was injecting those env variables. I had remedied that, and builds were running with the adjusted fix.

however the builds were in the midst of a long running "rebuild their ccaches" as a result of nodejs/node#56275 and the related work, and you cancelled those running builds before they finished.

builds are passing now, (https://ci.nodejs.org/job/node-test-commit-smartos/58244/) but tonight the daily build that we were trying to fix (see #3731 (comment)) will, again, take a very long time to get its ccaches warmed up and back to performing normally.

Please do not abort these long running smartos builds. They need to complete to get to a state where we have performant CI runners.

I will keep aborting those as long as I need to get the release proposal ready on time, sorry not sorry. Getting those long running jobs through is something that needs to happen eventually I agree, but it seems like a low priority in comparison to not slowing down the day-to-day activities of the project (i.e. landing PRs) and the time sensitive ones (i.e. releasing Node.js). I'm hopeful the caches will populate over the weekend when the CI is less busy.

Anyway, thanks for fixing the builds!

The replacement smartos nodes have been successfully running builds for a good while now. I plan on decomissioning the old smartos nodes and removing them from Jenkins.

This is very much done and documented.

Navigation Menu

Repository navigation

Equinix Move: Rebuild SmartOS Hosts #3731

21 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Equinix Move: Rebuild SmartOS Hosts #3731

Description

Activity

targos commented on May 18, 2024

ryanaslett commented on May 23, 2024

ryanaslett commented on May 28, 2024

ryanaslett commented on May 29, 2024

targos commented on May 30, 2024

ryanaslett commented on May 30, 2024

targos commented on May 31, 2024

richardlau commented on May 31, 2024

ryanaslett commented on May 31, 2024

bahamat commented on May 31, 2024

jperkin commented on Jun 3, 2024

mhdawson commented on Jun 5, 2024

bahamat commented on Jun 6, 2024

mhdawson commented on Jun 6, 2024

21 remaining items

jperkin commented on Dec 16, 2024

ryanaslett commented on Dec 16, 2024

ryanaslett commented on Dec 16, 2024

bahamat commented on Dec 17, 2024

ryanaslett commented on Dec 17, 2024

ryanaslett commented on Dec 17, 2024

ryanaslett commented on Dec 17, 2024

aduh95 commented on Dec 18, 2024

aduh95 commented on Dec 18, 2024

bahamat commented on Dec 18, 2024

ryanaslett commented on Dec 19, 2024

aduh95 commented on Dec 19, 2024

ryanaslett commented on Jan 7, 2025

ryanaslett commented on May 23, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions