Skip to content

contrib/aws: add non-blocking persistent manual cluster (PMC) test stages #12743

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
nmazzilli3 wants to merge 1 commit into
base: main
Choose a base branch
from
+121 −0
Open

contrib/aws: add non-blocking persistent manual cluster (PMC) test stages #12743

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
121 changes: 121 additions & 0 deletions contrib/aws/Jenkinsfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -317,6 +317,124 @@ def post_build_actions() {
}
}

/*
* Persistent Manual Cluster (PMC) configuration tables and helpers.
* These stages run as non-blocking (catchError) so failures only mark
* the stage UNSTABLE without failing the overall build.
*/
def prci_instance_configuration(cluster_key) {
def cluster_configs = [
"c5n": [
"instance_type": "c5n.18xlarge", "os": "ubuntu2604", "region": "us-east-1",
"cluster_name": "LibfabricPRCI-PersistentManualCluster-c5n", "lock_label": "c5n-ub26-iad"
],
"c7g-ub22": [
"instance_type": "c7g.16xlarge", "os": "ubuntu2204", "region": "us-east-1",
"cluster_name": "LibfabricPRCI-PersistentManualCluster-c7g-ub22", "lock_label": "c7g-ub22-iad"
],
"c7g-ub24": [
"instance_type": "c7g.16xlarge", "os": "ubuntu2404", "region": "us-east-1",
"cluster_name": "LibfabricPRCI-PersistentManualCluster-c7g-ub24", "lock_label": "c7g-ub24-iad"
],
"hpc7g": [
"instance_type": "hpc7g.16xlarge", "os": "rhel9", "region": "us-east-1",
"cluster_name": "LibfabricPRCI-PersistentManualCluster-hpc7g", "lock_label": "hpc7g-rhel9-iad"
],
"g4dn": [
"instance_type": "g4dn.12xlarge", "os": "alinux2023", "region": "us-west-2",
"cluster_name": "LibfabricPRCI-PersistentManualCluster-g4dn", "lock_label": "g4dn-al2023-pdx"
],
"hpc8a": [
"instance_type": "hpc8a.96xlarge", "os": "alinux2023", "region": "eu-north-1",
"cluster_name": "LibfabricPRCI-PersistentManualCluster-hpc8a", "lock_label": "hpc8a-al2023-arn"
],
"hpc6a": [
"instance_type": "hpc6a.48xlarge", "os": "rhel8", "region": "eu-north-1",
"cluster_name": "LibfabricPRCI-PersistentManualCluster-hpc6a", "lock_label": "hpc6a-rhel8-arn"
],
"c8gn": [
"instance_type": "c8gn.16xlarge", "os": "alinux2023", "region": "us-west-2",
"cluster_name": "LibfabricPRCI-PersistentManualCluster-c8gn", "lock_label": "c8gn-al2023-pdx"
],
"c7i": [
"instance_type": "c7i.16xlarge", "os": "alinux2023", "region": "us-west-2",
"cluster_name": "LibfabricPRCI-PersistentManualCluster-c7i", "lock_label": "c7i-al2023-pdx"
]
]
return cluster_configs.get(cluster_key, null)
}

def manage_jenkins_lock_ip_mapping_read(region, cluster_name, locked_resource_prefix) {
return sh(
script: ". ${venv_path}/bin/activate; ${portafiducia_path}/scripts/manage_jenkins_lock_ip_mapping.py --region ${region} --cluster-name ${cluster_name} --operation read --locked-resource-prefix ${locked_resource_prefix}",
returnStdout: true
).trim()
}

def run_test_orchestrator_once_persistent(run_name, cluster_name, os, instance_type, instance_count, region, public_ip_addresses, addl_args) {
def args = "--os ${os} --container-os ${os} --instance-type ${instance_type} --instance-count ${instance_count} --region ${region}"
Comment thread
nmazzilli3 marked this conversation as resolved.
args += " --cluster-name ${cluster_name} --cluster-type manual_cluster"
args += " --public-ip-addresses ${public_ip_addresses}"
args += " ${addl_args}"
args += " --perf-metrics-path outputs"
args += " --junit-xml outputs/${run_name}.xml"
sh ". ${venv_path}/bin/activate; cd ${portafiducia_path}/tests && STAGE_NAME='" + run_name + "' ./test_orchestrator.py ${args}"
}

def get_test_stage_with_lock_persistent(stage_name, cluster_key, lock_count, addl_args) {
def instance_info = prci_instance_configuration(cluster_key)
if (instance_info == null) {
error("No prci_instance_configuration entry for cluster key: ${cluster_key}")
}
return {
stage("${stage_name}") {
catchError(buildResult: 'SUCCESS', stageResult: 'UNSTABLE', catchInterruptions: false) {
lock(label: instance_info.lock_label, quantity: lock_count, variable: 'LOCKED_RESOURCE') {
echo "Acquired lock resource(s): ${env.LOCKED_RESOURCE}"
def public_ip_addresses = env.LOCKED_RESOURCE.tokenize(',').collect { resource ->
manage_jenkins_lock_ip_mapping_read(instance_info.region, instance_info.cluster_name, resource)
}.join(' ')
run_test_orchestrator_once_persistent(stage_name, instance_info.cluster_name, instance_info.os, instance_info.instance_type, lock_count, instance_info.region, public_ip_addresses, addl_args)
}
}
}
}
}

def build_pmc_test_stages() {

@a-szegel a-szegel Sep 9, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't want this second function, I want to do everything in 1 function and use the same vars as before (to the extent that we can... for test matrix at least) ... this way, when we make changes to our test matrix, we don't have to do it in two places and they don't drift/get dropped.

@nmazzilli3 nmazzilli3 Sep 15, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

im not following, could you be more specific with what you are looking to gain from this and why having a function is bad here.

@a-szegel a-szegel Sep 15, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is a chance both build_pmc_test_stages() and build_pr_test_stages() might exist in parallel for a long time. In that time frame, there is a chance someone wants to update the Jenkinsfile to control what tests run. Rather than copy/paste the same test logic in 2 places, the request was to either use common variables to define the test matrix (make updates in 1 place, potentially by moving both to one function).

All of these are copied/pasted twice (not 100% due to ${generic_pf} being different... but you get the point)

    def efa_addl_args = "${timeout} ${generic_pf} --test-libfabric-provider efa --test-list test_pr_ci_fabtests test_run_efa_unit_tests ${job_args}${persistent_manual_cluster_addl_args}${container_addl_args}"
    def efa_hpc_addl_args = "${timeout} ${generic_pf} --test-libfabric-provider efa --test-list test_pr_ci_fabtests test_pr_ci_imb test_run_efa_unit_tests ${job_args}${persistent_manual_cluster_addl_args}${container_addl_args}"
    def efa_hpc_dso_addl_args = "${timeout} ${generic_pf} --test-libfabric-provider efa --test-list test_pr_ci_fabtests test_pr_ci_imb test_run_efa_unit_tests test_pr_ci_dso ${job_args}${persistent_manual_cluster_addl_args}${container_addl_args}"
    def tcp_addl_args = "${timeout} ${generic_pf} --test-libfabric-provider tcp --enable-efa false --test-list test_pr_ci_fabtests test_run_efa_unit_tests ${job_args}${persistent_manual_cluster_addl_args}${container_addl_args}"
    def shm_addl_args = "${timeout} ${generic_pf} --test-libfabric-provider shm --enable-efa false --test-list test_pr_ci_fabtests test_run_efa_unit_tests ${job_args}${persistent_manual_cluster_addl_args}${container_addl_args}"

def stages = [:]

@a-szegel a-szegel Sep 9, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missing/Added Test Stages:

  1. Missing 1_c5n_alinux2023_neuron -> I would like you to add a trn2 stage b/c we have already have capacity and one of the major goals of this effort was to restore trn2 testing.
  2. Missing 2_c7gn_ubuntu2204_efa
  3. Missing 2_c7i_ubuntu2204_sockets
  4. Missing 1_g4dn_alinux2023_sm2
  5. Missing 1_g4dn_ubuntu2404_shm_disable-cma
  6. New pmc_2_c7g_ubuntu2204_efa (not in old matrix)

@nmazzilli3 nmazzilli3 Sep 15, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. trn2 was not part of the initial migration plan. I will send the doc offline
  2. c7gn for ub 22 and 24 seems to be there
  3. We will need to add capacity to fit this test in as the original count was only 6
  4. yes there is open bug for this, we will need to add the single gpu g4dn or set visible cuda devices to 0.
  5. All single node instance tests need further capacity / requirements finalization
  6. isnt this what you said was missing in 2

@a-szegel a-szegel Sep 15, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. The entire point of this project is to add trn2. It is ok if it doesn't come in with the first round, but the project cannot be considered complete without it.
  2. pmc_2_c7g_ubuntu2204_efa is not equal to c7gn. It doesn't look like there is c7gn. We had both c7g and c7gn and that is important for each generation of NIC

3-5. The goal is for the test matrix not to change. Any change needs a large TODO comment before this can get merged.

  1. c7g is not equal to c7gn, they have different hardware.

def timeout = "--timeout 90"
def test_suite_pkg = "--test-suite-package subspace_nightly_tests --owner subspace"
def job_args = "--job-name '${env.JOB_NAME}' --job-url ${env.BUILD_URL}"
def persistent_manual_cluster_addl_args = " --keep-cluster --skip-fixture-setup --skip-health-checks --use-existing-installer --enable-placement-group false --lean-cluster-setup --use-prebuilt-ami-with-efa-installer true --cleanup-pf-directory true --libfabric-prci true --enable-live-log true"
def container_addl_args = " --test-in-containers-on-ec2"

def pr_selector = is_post_merge() ? "--test-type commit --test-libfabric-commit ${env.GIT_COMMIT}"

@a-szegel a-szegel Sep 9, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Post Merge is only supposed to be supported on EFA, this silently changes it to be supported everywhere.

@nmazzilli3 nmazzilli3 Sep 15, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ack, we can remove this.

@a-szegel a-szegel Sep 15, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this is fixed yet.

: "--test-type pr --test-libfabric-pr $env.CHANGE_ID"
def generic_pf = "--test-target libfabric ${pr_selector} ${test_suite_pkg}"

def efa_addl_args = "${timeout} ${generic_pf} --test-libfabric-provider efa --test-list test_pr_ci_fabtests test_run_efa_unit_tests ${job_args}${persistent_manual_cluster_addl_args}${container_addl_args}"
def efa_hpc_addl_args = "${timeout} ${generic_pf} --test-libfabric-provider efa --test-list test_pr_ci_fabtests test_pr_ci_imb test_run_efa_unit_tests ${job_args}${persistent_manual_cluster_addl_args}${container_addl_args}"
def efa_hpc_dso_addl_args = "${timeout} ${generic_pf} --test-libfabric-provider efa --test-list test_pr_ci_fabtests test_pr_ci_imb test_run_efa_unit_tests test_pr_ci_dso ${job_args}${persistent_manual_cluster_addl_args}${container_addl_args}"
def tcp_addl_args = "${timeout} ${generic_pf} --test-libfabric-provider tcp --enable-efa false --test-list test_pr_ci_fabtests test_run_efa_unit_tests ${job_args}${persistent_manual_cluster_addl_args}${container_addl_args}"
def shm_addl_args = "${timeout} ${generic_pf} --test-libfabric-provider shm --enable-efa false --test-list test_pr_ci_fabtests test_run_efa_unit_tests ${job_args}${persistent_manual_cluster_addl_args}${container_addl_args}"

stages["pmc_2_c5n_ubuntu2604_efa"] = get_test_stage_with_lock_persistent("pmc_2_c5n_ubuntu2604_efa", "c5n", 2, efa_addl_args)
stages["pmc_2_c7g_ubuntu2204_efa"] = get_test_stage_with_lock_persistent("pmc_2_c7g_ubuntu2204_efa", "c7g-ub22", 2, efa_addl_args)

@a-szegel a-szegel Sep 9, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This didn't exist in previous test matrix.

@a-szegel a-szegel Sep 15, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

c7g is not c7gn

stages["pmc_2_c7g_ubuntu2404_efa"] = get_test_stage_with_lock_persistent("pmc_2_c7g_ubuntu2404_efa", "c7g-ub24", 2, efa_addl_args)
stages["pmc_2_hpc7g_rhel9_efa"] = get_test_stage_with_lock_persistent("pmc_2_hpc7g_rhel9_efa", "hpc7g", 2, efa_hpc_addl_args)
stages["pmc_2_g4dn_alinux2023_efa"] = get_test_stage_with_lock_persistent("pmc_2_g4dn_alinux2023_efa", "g4dn", 2, efa_addl_args)
stages["pmc_2_hpc8a_alinux2023_efa"] = get_test_stage_with_lock_persistent("pmc_2_hpc8a_alinux2023_efa", "hpc8a", 2, efa_hpc_addl_args)
stages["pmc_2_hpc6a_rhel8_efa"] = get_test_stage_with_lock_persistent("pmc_2_hpc6a_rhel8_efa", "hpc6a", 2, efa_hpc_dso_addl_args)
stages["pmc_2_c8gn_alinux2023_efa"] = get_test_stage_with_lock_persistent("pmc_2_c8gn_alinux2023_efa", "c8gn", 2, efa_addl_args)
stages["pmc_2_c7i_alinux2023_tcp"] = get_test_stage_with_lock_persistent("pmc_2_c7i_alinux2023_tcp", "c7i", 2, tcp_addl_args)
stages["pmc_1_g4dn_alinux2023_shm"] = get_test_stage_with_lock_persistent("pmc_1_g4dn_alinux2023_shm", "g4dn", 1, shm_addl_args)

@a-szegel a-szegel Sep 9, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pmc_1_g4dn_alinux2023_shm switched OS test coverage from ub24 to al2023 silently

@nmazzilli3 nmazzilli3 Sep 15, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes with an active backlog ticket and decision needed.

@a-szegel a-szegel Sep 15, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can replace with a large TODO dedicated to this.


return stages
}



def get_test_stage_with_lock(stage_name, build_tag, os, instance_type, instance_count, region, lock_label, addl_args) {
/*
* Generate a single test stage that run test_orchestrator.py with the given parameters.
Expand Down Expand Up @@ -387,6 +505,7 @@ def build_pr_test_stages() {
// single-GPU worker oversubscription that hangs the EFA progress engine.
stages["2_g4dn_alinux2023_efa"] = get_test_stage_with_lock("2_g4dn_alinux2023_efa", env.BUILD_TAG, "alinux2023", "g4dn.12xlarge", 2, "us-east-1", g4dn12x_lock_label, "--odcr cr-06968bbb5916ec4d3 ${addl_args_efa}")


// Multi Node Tests - Other Providers
stages["2_c7i_alinux2023_tcp"] = get_test_stage_with_lock("2_c7i_alinux2023_tcp", env.BUILD_TAG, "alinux2023", "c7i.16xlarge", 2, "us-west-2", c7i16x_lock_label, "--odcr cr-02f365311e1143b20 ${addl_args_tcp}")
stages["2_c7i_ubuntu2204_sockets"] = get_test_stage_with_lock("2_c7i_ubuntu2204_sockets", env.BUILD_TAG, "ubuntu2204", "c7i.16xlarge", 2, "us-west-2", c7i16x_lock_label, "--odcr cr-02f365311e1143b20 ${addl_args_sockets}")
Expand Down Expand Up @@ -496,6 +615,8 @@ pipeline {
script {
def stages = is_post_merge() ? build_postmerge_test_stages()
: build_pr_test_stages()
// Add PMC stages (non-blocking) to run in parallel with existing stages
stages = stages + build_pmc_test_stages()
parallel stages
}
}
Expand Down
Loading