Summary
Two alerts shipped in this bundle are reported by production as having selected no series for over a week, which reads identically to a rule whose metric was renamed or never shipped.
Both are correct as written. The controller pod is running and not crash-looping, and the membership readiness rule reads 545 real series that are all currently ready, so counting the unready ones yields no series at all rather than a zero.
Nothing on either rule records that the emptiness is intended, so they sit in the same list as the genuinely dead ones and the reader cannot tell which is which.
What success looks like
Background
Found while clearing the alerts that production reports as matching no data, tracked in https://github.com/datum-cloud/infra/issues/4909. The mechanism is a label the compliance exporter reads, described in https://github.com/datum-cloud/infra/blob/main/docs/runbooks/alerting/rule-matches-no-data.md.
https://claude.ai/code/session_01DirbefA5FTdhqHDUwneqYp
Summary
Two alerts shipped in this bundle are reported by production as having selected no series for over a week, which reads identically to a rule whose metric was renamed or never shipped.
Both are correct as written. The controller pod is running and not crash-looping, and the membership readiness rule reads 545 real series that are all currently ready, so counting the unready ones yields no series at all rather than a zero.
Nothing on either rule records that the emptiness is intended, so they sit in the same list as the genuinely dead ones and the reader cannot tell which is which.
What success looks like
Background
Found while clearing the alerts that production reports as matching no data, tracked in https://github.com/datum-cloud/infra/issues/4909. The mechanism is a label the compliance exporter reads, described in https://github.com/datum-cloud/infra/blob/main/docs/runbooks/alerting/rule-matches-no-data.md.
https://claude.ai/code/session_01DirbefA5FTdhqHDUwneqYp