Spend limits + budgets via Amberflo rated usage

[mattdjenkinson]

Parent enhancement: platform/billing/initial-scope.md.
Depends on: datum-cloud/tmp-2026-07-08-Pricing-via-Amberflo-478608ed#1 (Pricing via Amberflo) — you can't enforce a dollar cap without rated $ subtotals.
Related: datum-cloud/tmp-2026-07-08-Credit-ledger-via-Am-6bd3efb0#1 (credit ledger), datum-cloud/enhancements#730 (onboarding flow). Originated in @zsmithnyc's comment on datum-cloud/tmp-2026-07-08-Pricing-via-Amberflo-478608ed#1.

Summary

Today the platform has no way for a customer or operator to say "this org should never spend more than $X this period." Quota gates units (tokens, CPU-seconds), not dollars — a 100k-token quota and a $100 spend cap are different primitives because the rate (and therefore the cost of one token) lives in the Offer, not the meter. This enhancement introduces a SpendLimit primitive: dollar-denominated, period-scoped, with configurable actions (notify / block further usage / suspend). A controller polls Amberflo's rated-usage API on a schedule and fires the configured action when the period subtotal crosses a threshold.

Motivation

Three concrete user / operator stories that today have no clean answer:

• Fraud / abuse cap. "Don't let any single org accrue more than $100 in usage before someone reviews the account." Today this requires manual monitoring or quota guesses that don't track price changes.
• Customer self-imposed budget. "Notify me at $80, block at $100 — I'm dogfooding and don't want a surprise bill." AWS Budgets / GCP Billing budgets cover this for hyperscalers; we have no equivalent.
• Staff make-good cap on a comped account. "Customer X is on a trial with $500 of credit — block them at $500 even though their card is on file." Credits (#747) alone don't enforce this; once credit is exhausted the card would silently take over.

Quota addresses unit caps (which is what datum-cloud/tmp-2026-07-08-Pricing-via-Amberflo-478608ed#1 + service-catalog's OrganizationDefaultsReconciler already do for BA count, projects, etc.); it does not address monetary caps. The two primitives compose — a SpendLimit enforces dollars, quota enforces units — but neither replaces the other.

Goals (v1)

• SpendLimit CRD in service-catalog, BA-scoped: spec.billingAccountRef, spec.amount (decimal + currency), spec.period (BillingPeriod aligned with BA.spec.paymentTerms.invoiceFrequency, or fixed Monthly), spec.thresholds[] (one or more, each with percent + action).
• Threshold actions. Notify, Block, Suspend. Multiple thresholds on a single SpendLimit (e.g. notify at 80%, block at 100%) so customers don't need a separate CR per action.
• Evaluation controller in milo-os/amberflo-provider (or a sibling milo-os/billing controller — see open Q2). Polls Amberflo's rated-usage API on a configurable cadence, evaluates against active SpendLimits, transitions status.thresholdsCrossed[] and fires actions.
• Notify action. Emits a notification.miloapis.com/Notification (or equivalent) addressed to the BA's contact and to the org's notification routing. Idempotent — one Notification per threshold per period boundary.
• Block action. Patches the BA's active BillingEntitlement.status (or sets a condition) so the existing quota fan-out can refuse new resource creation. Block stays in force until the period resets or the limit is raised.
• Suspend action. Same as Block plus marks the BA Phase: Suspended. Existing resources keep running for the grace window declared in spec.suspendGraceSeconds; after that they're stopped via the existing workload-controller suspend path. Out-of-scope-from-here: the actual workload stop logic — we just emit the suspend signal.
• Period reset. Limits reset at the period boundary (driven by BA.spec.paymentTerms.invoiceDayOfMonth + the invoice run). Crossed thresholds clear, blocks lift, suspensions lift unless held open by a separate operator action.
• Visibility.
  • Cloud-portal billing page: current period spend, active limits, thresholds + status, history of past triggers.
  • Staff-portal: same plus the ability to create / edit / disable a SpendLimit on any BA (with audit-log entry).
• Composition with credits (#747). Credit consumption does not count toward SpendLimit — the limit measures dollars the customer is spending out of pocket. A $100 limit on a BA with $50 of credit still blocks at $100 of post-credit charges. Confirm with datum-cloud/tmp-2026-07-08-Credit-ledger-via-Am-6bd3efb0#1 before locking.

Non-Goals (v1)

• Per-Service or per-Project spend limits. v1 is BA-scoped only; finer-grained budgets follow once a customer asks for them.
• Forecasted / projected spend alerts ("at current rate you'll hit $X by end of period"). Useful follow-up but needs a forecasting model — out for v1.
• Hard real-time enforcement. Polling cadence will be measurable lag (15s–1m); customers who exceed the limit by a small margin between polls aren't refunded the overage. We're solving fraud / surprise-bill, not penny-perfect billing.
• Multi-currency. USD only (matches the umbrella's out-of-scope list).
• Cross-account budgets (one limit spanning multiple BAs in an org). One BA per limit for v1.
• Workload stop logic itself — Suspend emits the signal, the existing workload controllers consume it. That separation already exists for other lifecycle events.

Design Sketch

Amberflo rated usage API
        │
        │ poll on cadence (per BA)
        ▼
SpendLimit evaluation controller
        │
        ├── status.currentSpend       ◄── persisted; cloud-portal reads this
        ├── status.thresholdsCrossed  ◄── append-only audit
        │
        ▼ on threshold cross
   ┌────────────────┬──────────────────┬────────────────────┐
   │                │                  │                    │
   ▼                ▼                  ▼                    ▼
Notification    Patch BA           Patch BA              (Period reset
emitted       .status.spend-      .status.spend-          on invoice run
(addressed    Blocked=True        Suspended=True          → clear flags,
to BA          ▼                   ▼                       lift block)
contacts)    Quota admission     Workload controller
             refuses new          enters suspend
             resource creates     grace window

Polling cadence: default 1 minute, configurable per-environment via ServiceConfiguration.spec.spendLimitPollInterval (new field on the billing ServiceConfiguration). Tighter cadence costs more Amberflo API calls; looser leaves more room for over-spend between polls. 1 minute is the rough middle ground for fraud-defense vs. API cost.

State machine: SpendLimit.status.phase ∈ Active | ThresholdCrossed | Suspended | Expired. Transitions are append-only on status.thresholdsCrossed[] so the history is auditable.

Open Questions

1. Block mechanism. Two candidates: (a) emit a condition on BillingEntitlement, which the existing quota fan-out reads as a "no new grants" signal; (b) patch each AllowanceBucket to limit=0 directly. (a) is cleaner — one condition gates everything — but couples this enhancement to quota-controller behaviour. Lean (a).
2. Where the evaluation controller lives. milo-os/amberflo-provider (it already talks to Amberflo) or a new milo-os/billing controller (the "policy" stays closer to billing semantics)? Lean amberflo-provider — single Amberflo client, one place to rate-limit API calls.
3. Multiple SpendLimits per BA. Strictly one (simpler), or unlimited (more flexible — overlay a customer self-budget on top of a staff fraud cap)? Lean unlimited; the controller evaluates the most-restrictive at any moment.
4. Action ordering / mutual exclusion. What happens if Notify at 80% and Block at 80% both fire — order? Probably notify-first-then-block in the same reconcile, but worth pinning.
5. Per-environment defaults. Should every new BA carry an implicit fraud cap (e.g. "block at $1000") by default, or do SpendLimits have to be created explicitly? Lean explicit-only for v1; auto-fraud-caps follow as a separate enhancement once we have signal on what threshold is reasonable.
6. Credit interaction (with datum-cloud/tmp-2026-07-08-Credit-ledger-via-Am-6bd3efb0#1). Confirm sequence: rated subtotal → credit drawdown → remainder counts toward SpendLimit. Need datum-cloud/tmp-2026-07-08-Credit-ledger-via-Am-6bd3efb0#1's reviewer (cc @scotwells) to validate before locking.

Acceptance Criteria

• A staff user can create a SpendLimit on any BA with multiple thresholds (e.g. 80% notify, 100% block).
• The evaluation controller polls Amberflo every minute and updates status.currentSpend within one poll of the period subtotal moving.
• At 80% of the limit, a Notification fires to the BA contacts.
• At 100%, the BA's active BillingEntitlement carries a SpendBlocked=True condition; subsequent resource-creation admission requests are refused with a clear error pointing at the SpendLimit.
• At the period boundary (invoice run), status.thresholdsCrossed[] clears, the SpendBlocked condition lifts, and the next poll re-establishes baseline.
• Cloud-portal billing page surfaces the limit, current spend, and crossed thresholds. Staff-portal can edit/disable any SpendLimit with an audit-log entry.

Suggested Implementation Phases

1. SpendLimit CRD + evaluation controller. Wire the polling, the currentSpend status, and the Notify action (lowest blast radius). Validate the polling cadence in staging.
2. Block action. Add the BA / Entitlement condition and the quota-fan-out integration that honours it.
3. Suspend action. Wire to the workload-suspend signal; verify with one product (compute) before others.
4. Cloud-portal + staff-portal UI. Surface limits, status, and (staff-only) editing.

[drewr]

Moved from https://github.com/datum-cloud/enhancements/issues/759