Add a comprehensive E2E test stressing Kueue + MultiKueue on CAPZ-provisioned AKS

[jackfrancis]

Summary

Introduce a new end-to-end (E2E) test suite in CAPZ (test/e2e/) that provisions a multi-cluster MultiKueue topology on Azure (one manager + N worker AKS clusters) and stress-tests the canonical behaviors of Kueue MultiKueue: cross-cluster job dispatching, single-cluster admission, status synchronization, loser-clone cleanup, quota handling, and worker failure/recovery.

Goals

• Provision a MultiKueue topology of CAPZ-managed AKS clusters: 1 manager + ≥2 workers (≥3 workers preferred to exercise nomination/cleanup).
• Install and configure Kueue + MultiKueue on all clusters and validate the canonical job flow under load.
• Assert the canonical behaviors enumerated below, with stress dimensions (many concurrent workloads, multiple workers, quota exhaustion, worker disconnect/recovery, mixed job types).
• Structure the connection layer so the worker-attachment mechanism is pluggable: kubeconfig secret (baseline) → ClusterProfile / Cluster Inventory (AKS Fleet target).

Non-goals

• Performance benchmarking with published SLAs (we assert correctness + liveness under load, not throughput numbers).

Canonical MultiKueue behaviors to validate

Sourced from the Kueue docs (see References). The suite should assert:

1. Cluster roles & setup activation — After applying MultiKueueConfig, MultiKueueCluster, MultiKueue AdmissionCheck, ClusterQueue, ResourceFlavor, LocalQueue, the ClusterQueue/AdmissionCheck/MultiKueueCluster all report Active=True (Reason: Ready/Active/Connected). Workers mirror the manager's namespaces + LocalQueues.
2. Job flow / dispatching — On QuotaReservation in the manager, the workload is created on worker clusters per the dispatcher mode; exactly one worker admits it; the manager deletes the workload from the other workers and creates the Job copy on the winner labeled kueue.x-k8s.io/prebuilt-workload-name.
3. Status sync — status.nominatedClusterNames is populated while pending and reset once status.clusterName is set (immutable). Manager-side Job/Workload status reflects remote execution; on remote Finished, the manager does a final sync then garbage-collects remote objects (no orphaned Workloads/Jobs on losers or winner).
4. Dispatcher modes — Validate AllAtOnce (default; fan-out to all workers) and Incremental (rounds of up to 3 workers in dictionary order, 5-min escalation). Optionally smoke-test External (custom controller sets nominatedClusterNames).
5. Quota behavior — Manager quota ≈ sum of worker quotas; verify under/over-provisioned quota effects (underutilization vs. dispatch of un-admittable workloads), queueing under exhaustion.
6. Resilience — Disconnect/cordon a worker (or drop its kubeconfig/ClusterProfile): MultiKueueCluster flips Active=False, in-flight work re-dispatches/holds correctly, and recovers when the worker returns.
7. Mixed job types — At minimum batch/Job and JobSet; ideally one of RayJob (reuse KubeRay install), MPIJob, AppWrapper, Deployment, StatefulSet, LeaderWorkerSet (note documented status-visibility limitation for the latter two).
8. Feature-gate matrix — Exercise MultiKueueWaitForWorkloadAdmitted (deletion timing of loser clones) and, for the Fleet path, MultiKueueClusterProfile.

Stress dimensions

Concurrent submission of many workloads (fan-out + contention), ≥3 workers (nomination + cleanup), quota exhaustion + queueing, AKS worker autoscaling/scale-from-zero interplay, worker disconnect/recovery, and kubeconfig/ClusterProfile rotation — all asserting "exactly-once admission" and "no orphaned remote objects."

Where to source canonical MultiKueue behavior

• Concepts: https://kueue.sigs.k8s.io/docs/concepts/multikueue/ (roles, job flow, dispatcher modes, status.clusterName/nominatedClusterNames, limitations).
• Setup task: https://kueue.sigs.k8s.io/docs/tasks/manage/setup_multikueue/ (objects, worker kubeconfig secret, create-multikueue-kubeconfig.sh, ClusterProfile section, feature gates, TAS).
• Supported job types: https://kueue.sigs.k8s.io/docs/tasks/run/multikueue/ (job, jobsets, kubeflow, mpijob, kuberay, appwrapper, deployment, statefulset, leaderworkerset, plain_pods, trainjob, external-frameworks).
• Upstream e2e to mirror: kubernetes-sigs/kueue → test/e2e/multikueue/{baseline,extended,sequential,dra}/ (e.g. baseline/e2e_test.go, tas_test.go) — canonical assertions and helper patterns.
• Cluster Inventory / ClusterProfile API: https://multicluster.sigs.k8s.io/ and the GKE Fleet precedent: https://github.com/GoogleCloudPlatform/gke-fleet-management/tree/main/multikueue-clusterprofile.
• AKS Fleet Manager: https://learn.microsoft.com/azure/kubernetes-fleet/overview.

Proposed approach — leveraging Cluster API + CAPZ on Azure

Add a new spec file test/e2e/azure_multikueue.go (build tag //go:build e2e) wired into the AKS test entry point, following existing conventions.

Reuse existing CAPZ building blocks:

• AKS provisioning via the managed-cluster flavors/ClusterClass already used by the AKS specs (AzureManagedControlPlane / AzureManagedMachinePool).
• Workload-cluster access: bootstrapClusterProxy.GetWorkloadCluster(ctx, ns, name) → GetClient()/GetClientSet(), and e2eConfig.GetIntervals(specName, "wait-...") for timeouts (pattern from azure_kuberay.go).
• Operator/Helm install + CRD handling: mirror azure_kuberay.go (Helm install of Kueue, dynamic.Interface for kueue.x-k8s.io and job CRDs).
• AKS Fleet wiring: reuse test/e2e/aks_fleets_member.go (armcontainerservicefleet + AzureManagedControlPlane.Spec.FleetsMember) to enroll workers into a fleet for the ClusterProfile-based path.

Spec outline:

1. Provision/adopt 1 manager + N worker AKS clusters (optionally enroll all in an AKS Fleet).
2. helm install Kueue on every cluster (pin version; enable required feature gates).
3. On each worker: apply mirrored Namespace + ResourceFlavor + ClusterQueue + LocalQueue; generate the MultiKueue SA kubeconfig.
4. On the manager: create worker-N-secret kubeconfig secrets (baseline) or consume Fleet-provided ClusterProfiles (target), then apply MultiKueueConfig + MultiKueueCluster + MultiKueue AdmissionCheck + manager ClusterQueue/LocalQueue.
5. Wait for Active=True across ClusterQueue/AdmissionCheck/MultiKueueCluster.
6. Run the behavior + stress scenarios above, asserting single-admission, status sync, and cleanup.
7. DeferCleanup all clusters/resources; collect Kueue controller + workload logs via the existing log-collector on failure.

Config/CI wiring: add intervals + any flavor vars to test/e2e/config/*.yaml, gate behind a Ginkgo label (e.g. MultiKueue) so it can run as an opt-in/periodic job rather than in the default PR gate (cost/time).

Acceptance criteria

• New e2e-tagged spec provisions a manager + ≥2 worker AKS clusters via CAPZ and reaches Active MultiKueue setup.
• Asserts behaviors 1–7 above for at least batch/Job + JobSet, including "exactly-once admission" and "no orphaned remote objects" after completion.
• Includes ≥1 stress scenario (many concurrent workloads across ≥3 workers) and ≥1 resilience scenario (worker disconnect/recovery).
• Connection layer is abstracted so kubeconfig-secret and ClusterProfile paths are interchangeable.
• Runs green in a dedicated/periodic e2e lane with log collection on failure; documented in test/e2e docs.

Open questions

• Reuse one AKS Fleet across workers, or independent clusters for the baseline lane?
• Minimum worker count for CI cost vs. coverage (2 vs. 3+)?
• Pin to a specific Kueue release or track main/beta feature gates?

References

See "Where to source canonical MultiKueue behavior" above, plus CAPZ e2e: test/e2e/azure_kuberay.go, test/e2e/aks_fleets_member.go, test/e2e/aks_machinepools.go.

/kind feature
/area testing

[kubernetes-prow]

@jackfrancis: The label(s) area/testing cannot be applied, because the repository doesn't have them.

Details

In response to this:

Summary

Introduce a new end-to-end (E2E) test suite in CAPZ (test/e2e/) that provisions a multi-cluster MultiKueue topology on Azure (one manager + N worker AKS clusters) and stress-tests the canonical behaviors of Kueue MultiKueue: cross-cluster job dispatching, single-cluster admission, status synchronization, loser-clone cleanup, quota handling, and worker failure/recovery.

This is groundwork for an upcoming collaboration on integrating AKS Fleet Manager with MultiKueue (via the Cluster Inventory / ClusterProfile API), so the suite must be structured to swap the cluster-connection mechanism from raw kubeconfig secrets to Fleet-generated ClusterProfiles.

Motivation

• We are about to collaborate as SMEs on AKS Fleet → Kueue MultiKueue integration. We need a reproducible, CI-able harness that exercises MultiKueue on Azure before/while that integration lands.
• GKE Fleet already has a MultiKueue + ClusterProfile walkthrough; Azure has no equivalent. CAPZ is the natural home because it already provisions AKS and already has AKS Fleet membership wiring in e2e (test/e2e/aks_fleets_member.go) and an operator-driven batch-workload test to model after (test/e2e/azure_kuberay.go).

Goals

• Provision a MultiKueue topology of CAPZ-managed AKS clusters: 1 manager + ≥2 workers (≥3 workers preferred to exercise nomination/cleanup).
• Install and configure Kueue + MultiKueue on all clusters and validate the canonical job flow under load.
• Assert the canonical behaviors enumerated below, with stress dimensions (many concurrent workloads, multiple workers, quota exhaustion, worker disconnect/recovery, mixed job types).
• Structure the connection layer so the worker-attachment mechanism is pluggable: kubeconfig secret (baseline) → ClusterProfile / Cluster Inventory (AKS Fleet target).

Non-goals

• Shipping the AKS Fleet ↔ MultiKueue ClusterProfile controller itself (separate effort; this issue tests/validates it once available).
• Upstreaming changes to Kueue.
• Performance benchmarking with published SLAs (we assert correctness + liveness under load, not throughput numbers).

Canonical MultiKueue behaviors to validate

Sourced from the Kueue docs (see References). The suite should assert:

1. Cluster roles & setup activation — After applying MultiKueueConfig, MultiKueueCluster, MultiKueue AdmissionCheck, ClusterQueue, ResourceFlavor, LocalQueue, the ClusterQueue/AdmissionCheck/MultiKueueCluster all report Active=True (Reason: Ready/Active/Connected). Workers mirror the manager's namespaces + LocalQueues.
2. Job flow / dispatching — On QuotaReservation in the manager, the workload is created on worker clusters per the dispatcher mode; exactly one worker admits it; the manager deletes the workload from the other workers and creates the Job copy on the winner labeled kueue.x-k8s.io/prebuilt-workload-name.
3. Status sync — status.nominatedClusterNames is populated while pending and reset once status.clusterName is set (immutable). Manager-side Job/Workload status reflects remote execution; on remote Finished, the manager does a final sync then garbage-collects remote objects (no orphaned Workloads/Jobs on losers or winner).
4. Dispatcher modes — Validate AllAtOnce (default; fan-out to all workers) and Incremental (rounds of up to 3 workers in dictionary order, 5-min escalation). Optionally smoke-test External (custom controller sets nominatedClusterNames).
5. Quota behavior — Manager quota ≈ sum of worker quotas; verify under/over-provisioned quota effects (underutilization vs. dispatch of un-admittable workloads), queueing under exhaustion.
6. Resilience — Disconnect/cordon a worker (or drop its kubeconfig/ClusterProfile): MultiKueueCluster flips Active=False, in-flight work re-dispatches/holds correctly, and recovers when the worker returns.
7. Mixed job types — At minimum batch/Job and JobSet; ideally one of RayJob (reuse KubeRay install), MPIJob, AppWrapper, Deployment, StatefulSet, LeaderWorkerSet (note documented status-visibility limitation for the latter two).
8. Feature-gate matrix — Exercise MultiKueueWaitForWorkloadAdmitted (deletion timing of loser clones) and, for the Fleet path, MultiKueueClusterProfile.

Stress dimensions

Concurrent submission of many workloads (fan-out + contention), ≥3 workers (nomination + cleanup), quota exhaustion + queueing, AKS worker autoscaling/scale-from-zero interplay, worker disconnect/recovery, and kubeconfig/ClusterProfile rotation — all asserting "exactly-once admission" and "no orphaned remote objects."

Where to source canonical MultiKueue behavior

• Concepts: https://kueue.sigs.k8s.io/docs/concepts/multikueue/ (roles, job flow, dispatcher modes, status.clusterName/nominatedClusterNames, limitations).
• Setup task: https://kueue.sigs.k8s.io/docs/tasks/manage/setup_multikueue/ (objects, worker kubeconfig secret, create-multikueue-kubeconfig.sh, ClusterProfile section, feature gates, TAS).
• Supported job types: https://kueue.sigs.k8s.io/docs/tasks/run/multikueue/ (job, jobsets, kubeflow, mpijob, kuberay, appwrapper, deployment, statefulset, leaderworkerset, plain_pods, trainjob, external-frameworks).
• Upstream e2e to mirror: kubernetes-sigs/kueue → test/e2e/multikueue/{baseline,extended,sequential,dra}/ (e.g. baseline/e2e_test.go, tas_test.go) — canonical assertions and helper patterns.
• Cluster Inventory / ClusterProfile API: https://multicluster.sigs.k8s.io/ and the GKE Fleet precedent: https://github.com/GoogleCloudPlatform/gke-fleet-management/tree/main/multikueue-clusterprofile.
• AKS Fleet Manager: https://learn.microsoft.com/azure/kubernetes-fleet/overview.

Proposed approach — leveraging Cluster API + CAPZ on Azure

Add a new spec file test/e2e/azure_multikueue.go (build tag //go:build e2e) wired into the AKS test entry point, following existing conventions.

Reuse existing CAPZ building blocks:

• AKS provisioning via the managed-cluster flavors/ClusterClass already used by the AKS specs (AzureManagedControlPlane / AzureManagedMachinePool).
• Workload-cluster access: bootstrapClusterProxy.GetWorkloadCluster(ctx, ns, name) → GetClient()/GetClientSet(), and e2eConfig.GetIntervals(specName, "wait-...") for timeouts (pattern from azure_kuberay.go).
• Operator/Helm install + CRD handling: mirror azure_kuberay.go (Helm install of Kueue, dynamic.Interface for kueue.x-k8s.io and job CRDs).
• AKS Fleet wiring: reuse test/e2e/aks_fleets_member.go (armcontainerservicefleet + AzureManagedControlPlane.Spec.FleetsMember) to enroll workers into a fleet for the ClusterProfile-based path.

Spec outline:

1. Provision/adopt 1 manager + N worker AKS clusters (optionally enroll all in an AKS Fleet).
2. helm install Kueue on every cluster (pin version; enable required feature gates).
3. On each worker: apply mirrored Namespace + ResourceFlavor + ClusterQueue + LocalQueue; generate the MultiKueue SA kubeconfig.
4. On the manager: create worker-N-secret kubeconfig secrets (baseline) or consume Fleet-provided ClusterProfiles (target), then apply MultiKueueConfig + MultiKueueCluster + MultiKueue AdmissionCheck + manager ClusterQueue/LocalQueue.
5. Wait for Active=True across ClusterQueue/AdmissionCheck/MultiKueueCluster.
6. Run the behavior + stress scenarios above, asserting single-admission, status sync, and cleanup.
7. DeferCleanup all clusters/resources; collect Kueue controller + workload logs via the existing log-collector on failure.

Config/CI wiring: add intervals + any flavor vars to test/e2e/config/*.yaml, gate behind a Ginkgo label (e.g. MultiKueue) so it can run as an opt-in/periodic job rather than in the default PR gate (cost/time).

Acceptance criteria

• New e2e-tagged spec provisions a manager + ≥2 worker AKS clusters via CAPZ and reaches Active MultiKueue setup.
• Asserts behaviors 1–7 above for at least batch/Job + JobSet, including "exactly-once admission" and "no orphaned remote objects" after completion.
• Includes ≥1 stress scenario (many concurrent workloads across ≥3 workers) and ≥1 resilience scenario (worker disconnect/recovery).
• Connection layer is abstracted so kubeconfig-secret and ClusterProfile paths are interchangeable.
• Runs green in a dedicated/periodic e2e lane with log collection on failure; documented in test/e2e docs.

Open questions

• Reuse one AKS Fleet across workers, or independent clusters for the baseline lane?
• Minimum worker count for CI cost vs. coverage (2 vs. 3+)?
• Pin to a specific Kueue release or track main/beta feature gates?
• Should the ClusterProfile path block this issue, or land as a follow-up once the AKS Fleet inventory controller exists?

References

See "Where to source canonical MultiKueue behavior" above, plus CAPZ e2e: test/e2e/azure_kuberay.go, test/e2e/aks_fleets_member.go, test/e2e/aks_machinepools.go.

/kind feature
/area testing

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[jackfrancis]

/assign @nojnhuh @mboersma

[nojnhuh]

@mboersma Do you want to start scaffolding the clusters in the CAPZ test framework? I can start some functional MultiKueue tests with kind/CAPD in the meantime which hopefully are easy to drop on top of CAPZ clusters.

[nojnhuh]

I got a start on this in #6429. I have yet to see if that actually does what we want and passes, but I'll try running it tomorrow.