A user signing up going through the onboarding flow in the portal can hit an "unauthorised" error within seconds when adding a payment method. We saw this concretely on 24.08.26: a brand-new org got 8 straight 403s from milo-apiserver on POST to allowancebuckets, the same time its owner PolicyBinding was still being reconciled into an OpenFGA tuple. The org had the grant it just hadn't fully propagated yet.
Every hop in the chain is fast and healthy on its own (reconcile p99 was under half a second, no errors, no queue backlog). The problem is that nothing guarantees the chain finishes before a client is allowed to act on the new org. If a request lands in that window, it just fails outright.
Potential fix: extend the existing admission-wait pattern to authz checks. When a request 403s because a relation tuple isn't resolvable yet for a resource that was just created, hold briefly and retry before failing, the same way the waiter already does for missing resources.
Worth flagging though, this is brittle under exactly the load it's meant to cover. The propagation time we measured (~1-3s) is from a calm system. A signup burst changes both sides of the race: more PolicyBinding CRs for auth-provider-openfga's controller to get through (queue depth won't stay near zero like it was here), and more requests polling OpenFGA for the tuple, adding read load right when it's busiest writing. Every held-open request also ties up a milo-apiserver goroutine/connection for the wait, so more signups at once means more waiters competing for the same capacity the reconciling needs. The busier it gets, the longer propagation takes, and the fix ends up holding more requests open for longer instead of failing fast, right when that's worst.
Should probably ship with a few guardrails, not just the wait itself:
- a cap on total concurrent waiters, failing fast with a retryable error past that cap instead of queuing without limit
- backoff with jitter instead of fixed-interval polling, so a burst doesn't hammer OpenFGA in lockstep
- a metric for "wait ceiling exceeded" (same idea as the quota waiter's own gauge) so a load-driven regression shows up before users hit it
- a load test simulating an actual signup burst, since the numbers above are the best case, not the case this is meant to protect against
We'll need to also need to make sure the portal submission has a long enough time out and some internal retry mechanism.
A user signing up going through the onboarding flow in the portal can hit an "unauthorised" error within seconds when adding a payment method. We saw this concretely on 24.08.26: a brand-new org got 8 straight 403s from
milo-apiserveron POST toallowancebuckets, the same time its owner PolicyBinding was still being reconciled into an OpenFGA tuple. The org had the grant it just hadn't fully propagated yet.Every hop in the chain is fast and healthy on its own (reconcile p99 was under half a second, no errors, no queue backlog). The problem is that nothing guarantees the chain finishes before a client is allowed to act on the new org. If a request lands in that window, it just fails outright.
Potential fix: extend the existing admission-wait pattern to authz checks. When a request 403s because a relation tuple isn't resolvable yet for a resource that was just created, hold briefly and retry before failing, the same way the waiter already does for missing resources.
Worth flagging though, this is brittle under exactly the load it's meant to cover. The propagation time we measured (~1-3s) is from a calm system. A signup burst changes both sides of the race: more PolicyBinding CRs for
auth-provider-openfga's controller to get through (queue depth won't stay near zero like it was here), and more requests polling OpenFGA for the tuple, adding read load right when it's busiest writing. Every held-open request also ties up amilo-apiservergoroutine/connection for the wait, so more signups at once means more waiters competing for the same capacity the reconciling needs. The busier it gets, the longer propagation takes, and the fix ends up holding more requests open for longer instead of failing fast, right when that's worst.Should probably ship with a few guardrails, not just the wait itself:
We'll need to also need to make sure the portal submission has a long enough time out and some internal retry mechanism.