You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Signal node-out-of-service state in ControllerUnpublishVolume #610
Is your feature request related to a problem?/Why is this needed
When a node is shut down or otherwise unrecoverable, NodeUnpublishVolume and NodeUnstageVolume cannot run, but a CO may still need to call ControllerUnpublishVolume. For example, today Kubernetes reaches this state in two ways:
The attach/detach controller's maximum wait for the volume to be safely unmounted has expired.
The first path says nothing about whether the node can still write to the volume, while the second follows the CO's explicit decision to take the node out of service. These are meaningfully different but that distinction is lost today because both produce the same ControllerUnpublishVolumeRequest, so the SP cannot handle them differently.
Describe the solution you'd like in detail
Add one field to ControllerUnpublishVolumeRequest:
message ControllerUnpublishVolumeRequest {
string volume_id = 1;
string node_id = 2;
map<string, string> secrets = 3 [(csi_secret) = true];
// Indicates that the CO has explicitly taken the node out of service
// and is calling `ControllerUnpublishVolume` without completing
// node-side volume cleanup.
// The CO MUST NOT set this field to true solely because the node is
// unreachable or a timeout waiting for node-side cleanup has expired.
// This field is OPTIONAL.
// The default value is false.
bool node_out_of_service = 4;
}
An older SP will ignore the field and continue handling ControllerUnpublishVolume as it does today, so I don't think a new capability is needed.
Describe alternatives you've considered
In practice, an SP may use node_out_of_service=true when deciding whether to issue a force detach after the CO has taken the node out of service. PR #477 covers the case where the node may still be running, so the SP must fence the volume before it can be reused. This field does not provide that guarantee, but an SP could support it alongside #477’s fencing capabilities.
Additional context
In terms of prior art, Issue #512 tracks this lifecycle mismatch, and PR #533 proposed an ordering exception but did not add a request signal preserving why the CO bypassed node cleanup.
changed the title [-]Signal node out of service state in ControllerUnpublishVolume[/-][+]Signal node-out-of-service state in ControllerUnpublishVolume[/+]on Aug 31, 2026
@bswartz and I talked offline through three broad ControllerUnpublish cases:
The ordinary spec path.
The timeout path.
The node out-of-service path.
Ben suggested modeling these cases with an enum rather than only carrying the out-of-service signal, which I agree with.
Importantly, whatever we do here should leave the door open for future work in this area, such as #477. This proposal is not a replacement for that, its about giving the CO a way to signal to the SP that its bypassing the node side operations.
Our SP implemented a similar feature for force detach. We can force attach the regional-disk to another zone and fence the previous node even when the previous detach cannot finish.
We currently enable this mode by detecting the volume is simutaniously unpublishing from one node and publishing to another node.
This is somewhat hacky IMO. Kubernetes RWO protection currently ignores nodes failing unpublish (uncertain in ad-controller), but this is not documented.
And this behaviour is actually causing another issue for us: After publish failed, Kubernetes can issue ControllerPublishVolume to multiple nodes even for RWO volume. Not sure if this is a bug.
Activity
[-]Signal node out of service state in ControllerUnpublishVolume[/-][+]Signal node-out-of-service state in ControllerUnpublishVolume[/+]torredil commentedon Sep 1, 2026
/assign
torredil commentedon Sep 1, 2026
@bswartz and I talked offline through three broad
ControllerUnpublishcases:Ben suggested modeling these cases with an enum rather than only carrying the out-of-service signal, which I agree with.
Importantly, whatever we do here should leave the door open for future work in this area, such as #477. This proposal is not a replacement for that, its about giving the CO a way to signal to the SP that its bypassing the node side operations.
huww98 commentedon Oct 7, 2026
Our SP implemented a similar feature for force detach. We can force attach the regional-disk to another zone and fence the previous node even when the previous detach cannot finish.
We currently enable this mode by detecting the volume is simutaniously unpublishing from one node and publishing to another node.
This is somewhat hacky IMO. Kubernetes RWO protection currently ignores nodes failing unpublish (uncertain in ad-controller), but this is not documented.
And this behaviour is actually causing another issue for us: After publish failed, Kubernetes can issue ControllerPublishVolume to multiple nodes even for RWO volume. Not sure if this is a bug.