Skip to content

Signal node-out-of-service state in ControllerUnpublishVolume #610

Open

Description

@torredil

Is your feature request related to a problem?/Why is this needed

When a node is shut down or otherwise unrecoverable, NodeUnpublishVolume and NodeUnstageVolume cannot run, but a CO may still need to call ControllerUnpublishVolume. For example, today Kubernetes reaches this state in two ways:

  1. The attach/detach controller's maximum wait for the volume to be safely unmounted has expired.
  2. The node is explicitly taken out of service.

The first path says nothing about whether the node can still write to the volume, while the second follows the CO's explicit decision to take the node out of service. These are meaningfully different but that distinction is lost today because both produce the same ControllerUnpublishVolumeRequest, so the SP cannot handle them differently.

Describe the solution you'd like in detail

Add one field to ControllerUnpublishVolumeRequest:

message ControllerUnpublishVolumeRequest {
  string volume_id = 1;
  string node_id = 2;
  map<string, string> secrets = 3 [(csi_secret) = true];

  // Indicates that the CO has explicitly taken the node out of service
  // and is calling `ControllerUnpublishVolume` without completing
  // node-side volume cleanup.
  // The CO MUST NOT set this field to true solely because the node is
  // unreachable or a timeout waiting for node-side cleanup has expired.
  // This field is OPTIONAL.
  // The default value is false.
  bool node_out_of_service = 4;
}

An older SP will ignore the field and continue handling ControllerUnpublishVolume as it does today, so I don't think a new capability is needed.

Describe alternatives you've considered

In practice, an SP may use node_out_of_service=true when deciding whether to issue a force detach after the CO has taken the node out of service. PR #477 covers the case where the node may still be running, so the SP must fence the volume before it can be reused. This field does not provide that guarantee, but an SP could support it alongside #477’s fencing capabilities.

Additional context

In terms of prior art, Issue #512 tracks this lifecycle mismatch, and PR #533 proposed an ordering exception but did not add a request signal preserving why the CO bypassed node cleanup.

Activity

  1. changed the title [-]Signal node out of service state in ControllerUnpublishVolume[/-] [+]Signal node-out-of-service state in ControllerUnpublishVolume[/+] on Aug 31, 2026
  2. torredil commented on Sep 1, 2026

    @torredil
    Author

    /assign

  3. torredil commented on Sep 1, 2026

    @torredil
    Author

    @bswartz and I talked offline through three broad ControllerUnpublish cases:

    1. The ordinary spec path.
    2. The timeout path.
    3. The node out-of-service path.

    Ben suggested modeling these cases with an enum rather than only carrying the out-of-service signal, which I agree with.

    Importantly, whatever we do here should leave the door open for future work in this area, such as #477. This proposal is not a replacement for that, its about giving the CO a way to signal to the SP that its bypassing the node side operations.

  4. huww98 commented on Oct 7, 2026

    @huww98
    Contributor

    Our SP implemented a similar feature for force detach. We can force attach the regional-disk to another zone and fence the previous node even when the previous detach cannot finish.

    We currently enable this mode by detecting the volume is simutaniously unpublishing from one node and publishing to another node.
    This is somewhat hacky IMO. Kubernetes RWO protection currently ignores nodes failing unpublish (uncertain in ad-controller), but this is not documented.

    And this behaviour is actually causing another issue for us: After publish failed, Kubernetes can issue ControllerPublishVolume to multiple nodes even for RWO volume. Not sure if this is a bug.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions