Operational visibility into the billing pipeline (metering → pricing → invoicing → charge)

[kevwilliams]

Parent Issue

Tracked by datum-cloud/enhancements#763 (Bill customers for Compute usage)

Summary

The billing pipeline is made of several independently-failing hops — CloudEvent ingestion, billing-consumer attribution, Amberflo sync, invoice-run (#85), Stripe charge — and today none of them are visible to on-call. If a meter stops flowing, a customer either gets under-billed silently or the platform never notices until a support ticket. This task defines the dashboards, alerts, and runbooks needed to run the pipeline in production, and is the "operationalize before GA" gate for Compute billing.

Goals

• Define the SLIs for the billing pipeline (e.g. event-to-Amberflo lag, invoice-run success rate, Stripe charge success rate, meter drop rate).
• Dashboards showing pipeline health per stage: ingestion → attribution → Amberflo sync → invoice-run → Stripe charge.
• Alerting on stuck/failed invoice runs, meter delivery gaps, and Stripe charge failures above baseline decline rate.
• Runbook for the most likely failure modes: Amberflo sync backlog, invoice-run controller crash mid-run, Stripe outage, credit double-spend suspicion.
• Reconciliation check: a way to confirm metered usage in Amberflo matches what the platform emitted (catch silent drops).

Non-Goals

• Building the invoice-run controller itself (Invoice-run controller: turn rated Amberflo usage into a Stripe charge #85) — this issue covers observing it, not implementing it.
• Dunning / suspension policy for non-payment — separate enhancement.
• General platform-wide SLO framework — scoped to the billing pipeline specifically, though it should plug into whatever cross-cutting SLO/alerting convention the Reliability & Production Readiness workstream settles on.

Open Questions

• Where do billing pipeline dashboards live — alongside the general Observability workstream's Grafana boards, or a dedicated billing board?
• Who is on-call for billing pipeline failures — platform team or a dedicated billing rotation?
• What's the acceptable lag between usage occurring and it landing in Amberflo before we alert?