Orphaned NICs Cause Cluster-Wide LoadBalancer Failure During Quota Exhaustion

[cgiradkar]

/kind bug

[Before submitting an issue, have you checked the Troubleshooting Guide self-managed & managed?]

What steps did you take and what happened:
[A clear and concise description of what the bug is.]
Orphaned NICs Cause Cluster-Wide LoadBalancer Failure During Quota Exhaustion
Problem
When Azure quota is exhausted during VM provisioning, orphaned NICs cause cluster-wide LoadBalancer failure:

1. Azure creates NIC before allocating compute resources
2. VM creation fails due to quota exhaustion
3. NIC remains orphaned (no VM attached)
4. Orphaned NIC stays in LoadBalancer backend pool
5. LB marks backend as unhealthy → entire cluster loses connectivity
6. NIC is never cleaned up → LB remains broken
   Impact

• Cluster-wide connectivity failure - all services become unreachable
• Affects production workloads when Azure quota exhausted
• Persists until manual NIC cleanup by cluster admin
• No automatic recovery mechanism

Steps to Reproduce

1. Exhaust Azure VM quota in subscription
2. Scale up MachineSet to trigger new Machine creation
3. Observe VM creation failure with quota error
4. Check LoadBalancer backend pool - orphaned NIC present

What did you expect to happen:

• Orphaned NIC should be cleaned up automatically
• Machine should retry with exponential backoff
• No MHC remediation triggered (quota exhaustion is cluster-level, not machine-level failure)
• Cluster connectivity restored within seconds
• Auto-recovery when quota becomes available

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]
Upstream PR:

• Fix LoadBalancer hang by cleaning up orphaned NIC on quota failures #6386
• Review comment: "If a Machine fails for any reason (if the FailureReason is set), the Machine will be remediated immediately."

Environment:

• cluster-api-provider-azure version: 1.22
• Kubernetes version: (use kubectl version):
• OS (e.g. from /etc/os-release):