Skip to content

Consumers can't see why a workload fails to start #278

Open

Description

@scotwells

What needs to happen

When a workload fails to start, nothing a consumer can see tells them why. The workload and its deployment report that instances are "being provisioned" — indefinitely, with no indication that anything has gone wrong. The instance underneath reports a single failure reason that is frequently not the actual cause.

Image pulls are the clearest gap. When an image cannot be retrieved there is no consumer-visible field naming the image, the registry that was contacted, or the error that registry returned. Compute also emits no events into the project, so there is nowhere else for a consumer to look.

Workload status needs to carry enough for a consumer to diagnose a failed start on their own.

Why this matters

A live example from staging: a workload has reported "instances are being provisioned" for seven hours. Its instance reports a network interface allocation failure — while, on that same object, the network interface reports it allocated successfully and holds every address it asked for. The real cause is that the image could not be pulled. A consumer following the only signal available to them would spend the afternoon debugging networking.

That user had attached a valid image pull secret, so nothing looked wrong from their side either. A private image that genuinely needs credentials the user has not supplied fails in exactly the same silent way, so this is not limited to any one mistake.

Both the consumer-facing view in #164 and the provider troubleshooting view in #183 will render whatever status carries. Today that is "provisioning" and a misleading error, so this has to be fixed underneath them rather than in the presentation layer.

Desired outcome

  • A consumer can tell the difference between "still working on it" and "this has failed and will not recover on its own."
  • When an image cannot be retrieved, the consumer can see which image, from which registry, and what that registry said.
  • The reason surfaced on an instance reflects what is actually blocking it.
  • Diagnosing a failed workload does not require access to infrastructure the consumer does not have.

Related to #277
Related to #164
Related to #183

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Fields

Priority

None yet

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions