contrib/aws: add non-blocking persistent manual cluster (PMC) test stages by nmazzilli3 · Pull Request #12743 · ofiwg/libfabric

nmazzilli3 · 2026-08-26T04:54:04Z

contrib/aws: add non-blocking persistent manual cluster (PMC) test stages

Adds PMC test stages that run in parallel with the existing ephemeral PR CI stages. The existing stages are unchanged and still gate the PR; the PMC stages are non-blocking (wrapped in catchError → UNSTABLE, never touch build_ok) so they can A/B the persistent-cluster path against the current create-then-destroy path on every run.

New helpers + build_pmc_test_stages(), merged into the existing parallel block.
PMC stages attach to pre-created persistent clusters by public IP (from the Jenkins lock→IP SSM mapping) and run tests in-container instead of creating/destroying a cluster per run.
Tests the same PR/commit as the ephemeral stages (--test-libfabric-pr $CHANGE_ID, or $GIT_COMMIT post-merge).
Uses separate lock labels so the two paths don't contend for capacity.

Validated on the internal persistent-cluster POC across all 9 clusters (c5n, c7g-ub22, c7g-ub24, hpc7g, g4dn, hpc8a, hpc6a, c8gn, c7i).

nmazzilli3 · 2026-09-03T17:17:24Z

bot:aws:retest

nmazzilli3 · 2026-09-04T18:45:48Z

bot:aws:retest

a-szegel · 2026-09-09T14:45:58Z

+    stages["pmc_2_hpc6a_rhel8_efa"]      = get_test_stage_with_lock_persistent("pmc_2_hpc6a_rhel8_efa",      "hpc6a",    2, efa_addl_args)
+    stages["pmc_2_c8gn_alinux2023_efa"]  = get_test_stage_with_lock_persistent("pmc_2_c8gn_alinux2023_efa",  "c8gn",     2, efa_addl_args)
+    stages["pmc_2_c7i_alinux2023_tcp"]   = get_test_stage_with_lock_persistent("pmc_2_c7i_alinux2023_tcp",   "c7i",      2, tcp_addl_args)
+    stages["pmc_1_g4dn_alinux2023_shm"]  = get_test_stage_with_lock_persistent("pmc_1_g4dn_alinux2023_shm",  "g4dn",     1, shm_addl_args)


pmc_1_g4dn_alinux2023_shm switched OS test coverage from ub24 to al2023 silently

Yes with an active backlog ticket and decision needed.

Can replace with a large TODO dedicated to this.

a-szegel · 2026-09-09T14:47:36Z

+    }
+}
+
+def build_pmc_test_stages() {


I don't want this second function, I want to do everything in 1 function and use the same vars as before (to the extent that we can... for test matrix at least) ... this way, when we make changes to our test matrix, we don't have to do it in two places and they don't drift/get dropped.

im not following, could you be more specific with what you are looking to gain from this and why having a function is bad here.

There is a chance both build_pmc_test_stages() and build_pr_test_stages() might exist in parallel for a long time. In that time frame, there is a chance someone wants to update the Jenkinsfile to control what tests run. Rather than copy/paste the same test logic in 2 places, the request was to either use common variables to define the test matrix (make updates in 1 place, potentially by moving both to one function).

All of these are copied/pasted twice (not 100% due to ${generic_pf} being different... but you get the point)

def efa_addl_args = "${timeout} ${generic_pf} --test-libfabric-provider efa --test-list test_pr_ci_fabtests test_run_efa_unit_tests ${job_args}${persistent_manual_cluster_addl_args}${container_addl_args}" def efa_hpc_addl_args = "${timeout} ${generic_pf} --test-libfabric-provider efa --test-list test_pr_ci_fabtests test_pr_ci_imb test_run_efa_unit_tests ${job_args}${persistent_manual_cluster_addl_args}${container_addl_args}" def efa_hpc_dso_addl_args = "${timeout} ${generic_pf} --test-libfabric-provider efa --test-list test_pr_ci_fabtests test_pr_ci_imb test_run_efa_unit_tests test_pr_ci_dso ${job_args}${persistent_manual_cluster_addl_args}${container_addl_args}" def tcp_addl_args = "${timeout} ${generic_pf} --test-libfabric-provider tcp --enable-efa false --test-list test_pr_ci_fabtests test_run_efa_unit_tests ${job_args}${persistent_manual_cluster_addl_args}${container_addl_args}" def shm_addl_args = "${timeout} ${generic_pf} --test-libfabric-provider shm --enable-efa false --test-list test_pr_ci_fabtests test_run_efa_unit_tests ${job_args}${persistent_manual_cluster_addl_args}${container_addl_args}"

a-szegel · 2026-09-09T14:56:50Z

+}
+
+def build_pmc_test_stages() {
+    def stages = [:]


Missing/Added Test Stages:

Missing 1_c5n_alinux2023_neuron -> I would like you to add a trn2 stage b/c we have already have capacity and one of the major goals of this effort was to restore trn2 testing.

Missing 2_c7gn_ubuntu2204_efa

Missing 2_c7i_ubuntu2204_sockets

Missing 1_g4dn_alinux2023_sm2

Missing 1_g4dn_ubuntu2404_shm_disable-cma

New pmc_2_c7g_ubuntu2204_efa (not in old matrix)

trn2 was not part of the initial migration plan. I will send the doc offline

c7gn for ub 22 and 24 seems to be there

We will need to add capacity to fit this test in as the original count was only 6

yes there is open bug for this, we will need to add the single gpu g4dn or set visible cuda devices to 0.

All single node instance tests need further capacity / requirements finalization

isnt this what you said was missing in 2

The entire point of this project is to add trn2. It is ok if it doesn't come in with the first round, but the project cannot be considered complete without it.

pmc_2_c7g_ubuntu2204_efa is not equal to c7gn. It doesn't look like there is c7gn. We had both c7g and c7gn and that is important for each generation of NIC

3-5. The goal is for the test matrix not to change. Any change needs a large TODO comment before this can get merged.

c7g is not equal to c7gn, they have different hardware.

a-szegel · 2026-09-09T15:01:32Z

+    def shm_addl_args = "${timeout} ${generic_pf} --test-libfabric-provider shm --enable-efa false --test-list test_pr_ci_fabtests test_run_efa_unit_tests ${job_args}${persistent_manual_cluster_addl_args}${container_addl_args}"
+
+    stages["pmc_2_c5n_ubuntu2604_efa"]   = get_test_stage_with_lock_persistent("pmc_2_c5n_ubuntu2604_efa",   "c5n",      2, efa_addl_args)
+    stages["pmc_2_c7g_ubuntu2204_efa"]   = get_test_stage_with_lock_persistent("pmc_2_c7g_ubuntu2204_efa",   "c7g-ub22", 2, efa_addl_args)


This didn't exist in previous test matrix.

c7g is not c7gn

a-szegel · 2026-09-09T15:05:40Z

+    def persistent_manual_cluster_addl_args = " --keep-cluster --skip-fixture-setup --skip-health-checks --use-existing-installer --enable-placement-group false --lean-cluster-setup --use-prebuilt-ami-with-efa-installer true --cleanup-pf-directory true --libfabric-prci true --enable-live-log true"
+    def container_addl_args = " --test-in-containers-on-ec2"
+
+    def pr_selector = is_post_merge() ? "--test-type commit --test-libfabric-commit ${env.GIT_COMMIT}"


Post Merge is only supposed to be supported on EFA, this silently changes it to be supported everywhere.

ack, we can remove this.

I don't think this is fixed yet.

…ages Signed-off-by: Nick Mazzilli <nmazzill@amazon.com>

nmazzilli3 force-pushed the pmc_jenkins branch 4 times, most recently from e4ce152 to f3fa01e Compare August 31, 2026 17:35

a-szegel reviewed Aug 31, 2026

View reviewed changes

Comment thread contrib/aws/Jenkinsfile Outdated

nmazzilli3 force-pushed the pmc_jenkins branch 4 times, most recently from 3f73c1e to 85c7497 Compare September 2, 2026 04:12

nmazzilli3 force-pushed the pmc_jenkins branch 2 times, most recently from 14c137b to 42aa1f7 Compare September 8, 2026 22:28

a-szegel requested changes Sep 9, 2026

View reviewed changes

nmazzilli3 force-pushed the pmc_jenkins branch from 42aa1f7 to e8f8912 Compare September 13, 2026 19:10

contrib/aws: add non-blocking persistent manual cluster (PMC) test st…

707880a

…ages Signed-off-by: Nick Mazzilli <nmazzill@amazon.com>

nmazzilli3 force-pushed the pmc_jenkins branch from e8f8912 to 707880a Compare September 13, 2026 22:08

Conversation

nmazzilli3 commented Aug 26, 2026

Uh oh!

Uh oh!

nmazzilli3 commented Sep 3, 2026

Uh oh!

nmazzilli3 commented Sep 4, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

a-szegel Sep 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

a-szegel Sep 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

a-szegel Sep 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

a-szegel Sep 9, 2026 •

edited

Loading

a-szegel Sep 15, 2026 •

edited

Loading

a-szegel Sep 9, 2026 •

edited

Loading