AIX timing out when building V8 updates

@richardlau

At first it timed out when the timeout is 60 minutes https://ci.nodejs.org/job/node-test-commit-aix/65790/nodes=aix73-power9/console

Then @richardlau bumped it to 120minutes but it is still timing out https://ci.nodejs.org/job/node-test-commit-aix/65790/nodes=aix73-power9/console

From @sxa on slack:

OK that's while building the tq-csa files so it's the point at which the system is pushed to the limit in terms of system use. It's possible we're over the limit now when JOBS=10 on a 16GB system (which is how that machine is configured). I've dropped it to JOBS=9 so hopefully that will resolve it.

Opening an issue to track this since it appears more complicated than we thought

From the logs I expect it's hitting an out of memory condition on that system which has 16GB RAM and is set to run with JOBS=10. I ran some tests in #4230 (comment) which suggested that JOBS=10 was ok but it's possible that it was close to the limit, particularly when building the tq-csa files which are very resource intensive to compile. It's also possible that the V8 update is resulting in it using more memory than it did during those compiles with 14.6.

What I'll try and do is run a couple of tests clearing the ccache and building the 14.6 and 15.2 builds

In the interim I've dropped that machine to JOBS=9 so that if we are just close to the limit that will hopefully help.

https://ci.nodejs.org/job/node-test-commit-aix/nodes=aix73-power9/65841 is running on the same machine and has just been chewing through the tq-csa files on the V8 15.2 branch and looks to have got through that phase without problems.

Hmmm seen again today with JOBS=9 durignt he test build of V5 15.6 on ibm-aix73-3. I don't see it always happening though (I've done some builds with an unpopulated ccache in the last week and haven't hit it).

Ran through with JOBS=8 at https://ci.nodejs.org/job/node-test-commit-aix/nodes=aix73-power9/65966/console

Hard to say if this is purely down to increased RAM requirements of the later V8s or if there's another thing we need to address, but I never hit an issue on these machines when building from scratch without a populated ccache in the past when it would have needed to rebuild everything

This has also been seen recently even with JOBS=8 for example in https://ci.nodejs.org/job/node-test-commit-aix/nodes=aix73-power9/66336/console (V8 15.8 update PR) on aix73-5 and at https://ci.nodejs.org/job/node-test-commit-aix/nodes=aix73-power9/66256/consoleFull the V8 15.6 update PR on aix73-3

The first of these got stuck while going through the v8_base_without_compiler/deps/v8/src/compiler directory, the second on v8_base_without_compiler/deps/v8/src/heap

I'm going to try re-running the 66336 job - if that passes I'll try doing a few builds of it with ccache cleared and see if it shows up again.

Hit the failure in https://ci.nodejs.org/job/node-test-commit-aix/nodes=aix73-power9/66360/ which throw resource exhaustion errors while building the tq-csa files in src/objects

13:48:27   CXX(target) /home/iojs/build/workspace/node-test-commit-aix/nodes/aix73-power9/out/Release/obj.target/v8_initializers/gen/torque-generated/src/objects/js-relative-time-format-tq-csa.o
13:50:02 clang++: error: unable to execute command: Killed
13:50:02 clang++: error: clang frontend command failed due to signal (use -v to see invocation)
13:50:02 clang version 20.1.7 (https://github.com/llvm/llvm-project.git 6146a88f60492b520a36f8f8f3231e15f3cc6082)
13:50:02 Target: powerpc64-ibm-aix7.3.0.0
13:50:02 Thread model: posix
13:50:02 InstalledDir: /opt/clang+llvm-20.1.7-powerpc64-ibm-aix-7.2/bin
13:50:02 clang++: error: unable to execute command: posix_spawn failed: Not enough space
13:50:02 clang++: note: diagnostic msg: Error generating preprocessed source(s).
13:50:02 clang++: error: unable to execute command: posix_spawn failed: Not enough space
13:50:02 clang++: error: assembler command failed with exit code 1 (use -v to see invocation)
13:50:03 clang++: error: unable to execute command: posix_spawn failed: Not enough space
13:50:03 clang++: error: assembler command failed with exit code 1 (use -v to see invocation)
13:50:05 clang++: error: unable to execute command: posix_spawn failed: Not enough space

and then a few mintes later it sopped producing output (make was still running but no compile jobs)

Running monitoring on the machine showed that the build was using all of the 16GiB of RAM

$ vmstat 60
thr    memory              page              faults              cpu          
----- ----------- ------------------------ ------------ -----------------------
 r  b   avm   fre  re  pi  po  fr   sr  cy  in   sy  cs us sy id wa    pc    ec
 8  0 3699522 374353   0   0   0   0    0   0  38 12763 508 52  2 46  0  3.74  93.6
 8  1 3227598 879802   0   0  91 615 1762   0  54 12216 578 50  2 48  0  3.73  93.4
 7  3 2725855 1377466   0   0  33   0    0   0  67 15610 703 45  2 53  0  3.62  90.6
 6 15 1987060 2119966   0   0 156 231 3838   0  56 7956 643 41  1 55  2  3.25  81.2
 6  8 4109171 13518   0  15  65 367 6996   0  66 11610 658 34  1 63  2  2.61  65.3
 3 17 2181772 1946333   0  24 166  64   81   0 107 10708 776 18  1 76  5  1.56  39.0
 5 20 1977556 2220817   0  18 161 1216 13825   0  59 1740 939 35  1 61  3  2.98  74.6
 4 12 4204572 14443   0  27 124 311 7935   0  93 6807 688 22  1 71  5  1.62  40.5

The values in the third column are a problem when they get over 4000000 as that indicates that memory usage is at around 16GB (the figure is in 4k pages). Similarly the non-zer numbers in pi and po indicate that paging is occurring. At the time it was also observable in svmon -G -O unit=GB that virtual had reached size which is another indication of memory exhaustion.

Options:

Reduce to JOBS=7 (I'd rather not do this as we don't want to lose performance)
Add some extra swap space to stop it filling up (although that will have a performance impact)
Increase the memory slightly - perhaps to 20GiB - and then increase JOBS again to at least 8.

I'm going to try approach 3 on aix73-3 and also try the same experiment with monitoring on aix73-4 which has 8GiB swap so will be a valid check of approach 2 (will increase JOBS from 6 to 8 for that test - it historically runs with fewer as it has 2 CPUs instead of 4)

Both looked ok. I'll keep a vmstat process running in the background logging the RAM usage and will take a look if it occurs again on one of the machines (in which case I'll reopen this. A test of the V8 15.7 PR has just run through on the aix73-5 machine without issues too with -j9.

I will note that on the aix73-4 machine with 16GiB of RAM + 8GiB of swap a build paged a little in the src/compiler/turboshaft directory but it didn't seem to affect the run too badly - it's possibly the larger amount of swap on that machine helps.

Navigation Menu

Repository navigation

AIX timing out when building V8 updates #4496

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

AIX timing out when building V8 updates #4496

Description

Activity

sxa commented on Oct 1, 2026

sxa commented on Oct 1, 2026

sxa commented on Oct 8, 2026

sxa commented on Oct 8, 2026

sxa commented on Oct 29, 2026

sxa commented on Oct 29, 2026

sxa commented on Oct 29, 2026

sxa commented on Oct 29, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions