Alert rule tests pass against a copy, not the rule that ships

[ecv]

Summary

The alert rule unit tests are green while the alert that actually ships behaves differently, so a passing suite is no longer evidence that the deployed alert works. The tests run against a hand maintained second copy of the rules instead of the ones that deploy, nothing checks that the two stay in step, and a grace period added to the project creation alert months ago never reached the copy. Every assertion in that suite fails when it is pointed at the rule now running, and the suite is not wired into continuous integration either.

Note

The duplicate copy is a reasonable workaround, because the test runner cannot read the format the rules ship in, so the defect is the absence of anything keeping the copy honest.

Acceptance criteria

• The suite exercises the rules that ship, or a check fails when the copy and the shipped rule diverge
• The existing grace period drift is resolved
• The suite runs automatically on every change that touches alert rules
• A failing check names which alert drifted, so a reviewer can act on it without hunting

[ecv]

Evidence

Verified at fa05453ae861e937ab14b7c40cf1cf719acc8abe on main, read only, with promtool 3.13.2.

The two copies. The rule that deploys is a PrometheusRule custom resource at config/telemetry/alerts/resources-manager/projects.yaml. The test copy is bare Prometheus rule YAML at test/prometheus-rules/resources-manager/projects/projects-slo-rules.yaml. The suite at projects-slo-tests.yaml names rule_files: [projects-slo-rules.yaml], so it reads the sibling copy and never touches the shipped resource.

Both define one alert, ProjectStuckCreatingSLOViolation, with byte identical expr, labels and annotations. The only difference is for: 2m shipped, 0s in the test copy.

The drift has a commit. 314d6805 ("fix: add grace period to project stuck creating alert", 2026-03-28) changed for: 0s to for: 2m in one file only, per git show --stat. The test copy's git log --follow does not contain that commit. The stated intent was to stop false pages during monitoring disruptions, so the untested value is the one that matters operationally.

What that costs. Extracting .spec from the shipped resource into the path the suite reads turns SUCCESS into three failed assertions, all got:[], because at eval_time: 1m a for: 2m alert is still pending. The suite does not merely skip the grace period, it would fail outright against the shipped rule.

No generator. test/prometheus-rules/ is not written by anything. There is no go:generate in the tree, no Makefile, no hack/ or bin/, and scripts/ holds only datum-mcp.sh. Every generate:* task in Taskfile.yaml (generate:code, generate:openapi:identity, generate:docs, generate:test-docs) covers deepcopy, CRDs, OpenAPI and Chainsaw docs. None reads or writes config/telemetry or test/prometheus-rules.

No drift check. Nothing in the repo diffs the two. Searched Taskfile.yaml and all of .github/ for promtool, prometheus-rules, prometheus_rules, telemetry, diff, drift and sync. There is no pre-commit, lefthook or husky config, and no non GitHub CI config.

The suite never runs. task test-prometheus-rules exists at Taskfile.yaml:520 but no workflow calls it. The four workflows are build-apiserver, fix-newlines, test-environment-validation and testing; grepping every task invocation across .github/workflows/ returns no match. testing.yaml runs go test ./... on Go path changes only. So the drift above could not have been caught even in principle.

Reproduce

git clone https://github.com/milo-os/milo.git && cd milo
cp test/prometheus-rules/resources-manager/projects/*.yaml /tmp/p/ && cd /tmp/p
promtool test rules projects-slo-tests.yaml            # SUCCESS
yq '.spec' <repo>/config/telemetry/alerts/resources-manager/projects.yaml > projects-slo-rules.yaml
promtool test rules projects-slo-tests.yaml            # FAILED, 3 assertions, got:[]

Note on the shorter for

A shortened for in a fixture is a normal way to keep test series short, so 0s may well have been deliberate when the copy was written, when the shipped rule was also 0s. That is not what makes this a defect. The defect is that the shipped value changed afterwards and nothing existed to notice. Any field can drift the same way, and the annotation drift in the sibling repo (datum-cloud/network-services-operator#353) shows it is not confined to for.