Summary
Every readiness check in milo sits on a recording rule shaped like count(milo_<kind>_status_condition{type="Ready"} == 0), and count() over an empty set emits no sample rather than a zero.
A healthy check therefore publishes nothing, which is exactly what a rule with a typo, a dead exporter, or a vanished input publishes.
The alerts still fire when an object breaks, but neither a reader nor the no-data detector can tell a working rule from a dead one, and a dashboard panel reads "No data" where it should read zero.
Adding or vector(0) to each recording rule would make the healthy case publish a zero and leave the alerting behaviour unchanged.
Acceptance criteria
The rules
In notification.readiness.1m, feeding ContactsNotReady, ContactGroupsNotReady, EmailTemplatesNotReady and EmailBroadcastsNotReady.
In iam.readiness.1m, feeding PolicyBindingsNotReady.
What each environment shows
Staging, read 16 September 2026 at 22:56Z: notification_miloapis_com:contacts:not_ready fetched 221 series and produced zero samples, and notification_miloapis_com:contactgroups:not_ready fetched 1 series and did the same. Nothing under the notification_miloapis_com: prefix has published a sample there in seven days.
Production, read 17 September 2026 at 04:00Z: iam_miloapis_com:policybindings:not_ready reports health ok, an empty lastError, 2,566 series fetched, and lastSamples: 0. Its input metric is unbroken and growing across the last month, so the rule is working perfectly and saying nothing.
Why the zero earns its place here
Production's policy bindings were all Ready from 9 September, so the series vanished. It has since come back three times, for under two minutes each: one binding on 11 September, 22 on 12 September, and 178 on 16 September at 22:24Z.
Those bursts are the only evidence that anything happens in this system at all, and they sit in a series that is otherwise absent. On a graph they are indistinguishable from a rule that has been broken for a week and briefly returned.
A rule publishing a zero proves at every evaluation that it is still watching. The escape-hatch label only declares that silence is expected, which is why the floor is worth more than the label here.
Background
Found while clearing the alert rules that select no data, tracked in https://github.com/datum-cloud/infra/issues/5086. The alternative mechanism, a label the compliance exporter reads, is described in https://github.com/datum-cloud/infra/blob/main/docs/runbooks/alerting/rule-matches-no-data.md.
The production behaviour above is tracked in https://github.com/datum-cloud/infra/issues/3443, which also covers whether a burst this short should page.
Summary
Every readiness check in milo sits on a recording rule shaped like
count(milo_<kind>_status_condition{type="Ready"} == 0), andcount()over an empty set emits no sample rather than a zero.A healthy check therefore publishes nothing, which is exactly what a rule with a typo, a dead exporter, or a vanished input publishes.
The alerts still fire when an object breaks, but neither a reader nor the no-data detector can tell a working rule from a dead one, and a dashboard panel reads "No data" where it should read zero.
Adding
or vector(0)to each recording rule would make the healthy case publish a zero and leave the alerting behaviour unchanged.Acceptance criteria
The rules
In
notification.readiness.1m, feedingContactsNotReady,ContactGroupsNotReady,EmailTemplatesNotReadyandEmailBroadcastsNotReady.In
iam.readiness.1m, feedingPolicyBindingsNotReady.What each environment shows
Staging, read 16 September 2026 at 22:56Z:
notification_miloapis_com:contacts:not_readyfetched 221 series and produced zero samples, andnotification_miloapis_com:contactgroups:not_readyfetched 1 series and did the same. Nothing under thenotification_miloapis_com:prefix has published a sample there in seven days.Production, read 17 September 2026 at 04:00Z:
iam_miloapis_com:policybindings:not_readyreports healthok, an emptylastError, 2,566 series fetched, andlastSamples: 0. Its input metric is unbroken and growing across the last month, so the rule is working perfectly and saying nothing.Why the zero earns its place here
Production's policy bindings were all Ready from 9 September, so the series vanished. It has since come back three times, for under two minutes each: one binding on 11 September, 22 on 12 September, and 178 on 16 September at 22:24Z.
Those bursts are the only evidence that anything happens in this system at all, and they sit in a series that is otherwise absent. On a graph they are indistinguishable from a rule that has been broken for a week and briefly returned.
A rule publishing a zero proves at every evaluation that it is still watching. The escape-hatch label only declares that silence is expected, which is why the floor is worth more than the label here.
Background
Found while clearing the alert rules that select no data, tracked in https://github.com/datum-cloud/infra/issues/5086. The alternative mechanism, a label the compliance exporter reads, is described in https://github.com/datum-cloud/infra/blob/main/docs/runbooks/alerting/rule-matches-no-data.md.
The production behaviour above is tracked in https://github.com/datum-cloud/infra/issues/3443, which also covers whether a burst this short should page.