[ivanauth]

When users start SpiceDB without running migrations first, they get a confusing error like:

unable to list namespaces: ERROR: relation "namespace_config" does not exist (SQLSTATE 42P01)

This doesn't tell them what went wrong or how to fix it.

Now the error message is clear and actionable:

datastore error: the database schema has not been initialized; please run "spicedb datastore migrate": ERROR: relation "namespace_config" does not exist (SQLSTATE 42P01)

This applies to all SQL datastores (postgres, mysql, crdb) and covers read operations, write operations, and statistics queries.

Closes #2749

Readiness gate latches once ready

The readiness middleware now acts as a startup migration gate: after the datastore first reports ready, the gate remains open for the lifetime of the process. This matches the existing health manager, which marks services serving and stops checking after the datastore and dispatcher become ready. Previously, the gate rechecked readiness every 500ms, so an older running pod could begin rejecting requests after a newer SpiceDB version advanced the datastore to a revision the older binary did not recognize. Latching prevents that additional rejection during rolling upgrades and removes readiness queries from the normal request path after startup. Before the gate opens, migration-related failures still return FailedPrecondition with ERROR_REASON_DATASTORE_NOT_MIGRATED.

[codecov]

Codecov Report

❌ Patch coverage is 96.31902% with 6 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
pkg/middleware/readiness/readiness.go	91.67%	4 Missing and 2 partials ⚠️

📢 Thoughts on this report? Let us know!

[miparnisari]

Used Postman and tried all the APIs against CRDB and Spanner. Notes:

• we're missing this change for Spanner. You can use docker-compose -f docker-compose.spanner.yaml up --build to test. E.g.
  • call to WriteSchema I get "unable to list caveats: spanner: code = \"NotFound\", desc = \"Table not found: caveat\", requestID = \"1.23ff9b5081a896f8.1.1.70.1\"",.
  • call to DeleteRelationships I get "object definition doc not found". This line has to be updated to account for when the error description says "table"

    spicedb/internal/datastore/spanner/reader.go

    Line 300 in f41f641

    if spanner.ErrCode(err) == codes.NotFound {

• in many cases, the errors returned to the user are prefixed with error encoding zedtoken: unable to read unique ID which makes no sense to me, and probably not to the end user 😆 It comes from

  spicedb/pkg/zedtoken/zedtoken.go

  Lines 71 to 74 in 01d72a1

  	datastoreUniqueID, err := ds.UniqueID(ctx)
  	if err != nil {
  	return nil, fmt.Errorf(errEncodeError, err)
  	}

  . i'm very tempted to remove that error encoding zedtoken and to clarify what the "unique ID" is
• in spanner, if you didn't run migrations, calling Watch api gives a HTTP 400. But against CRDB, the watch service is completely disabled: unknown service authzed.api.v1.WatchService, which means that you need to re-boot the server after you ran the migrations. I wonder if the server is supposed to be hable to handle migrations running at the same time ?

Anyway. Could you update

spicedb/internal/datastore/crdb/crdb.go

Line 576 in 3370198

features.Watch.Reason = "Range feeds must be enabled in CockroachDB and the user must have permission to create them in order to enable the Watch API: " + err.Error()

to set a different error message in this case?

• when you call ImportBulk, you get this

{
    "code": 2,
    "message": "statement description failed: ERROR: relation \"relation_tuple\" does not exist (SQLSTATE 42P01)",
    "details": []
}

, seems like we missed an error handling here:

spicedb/internal/datastore/postgres/common/bulk.go

Line 79 in 3370198

copied, err := tx.CopyFrom(ctx, pgx.Identifier{tupleTableName}, colNames, adapter)

[miparnisari]

nit: please create the right error code

[miparnisari]

Suggested change

	fmt.Errorf("datastore error: the database schema has not been initialized; please run \"spicedb datastore migrate\": %w", underlying),
	fmt.Errorf("%w. please run \"spicedb datastore migrate\", underlying),

[miparnisari]

Suggested change

	if wrappedErr := pgxcommon.WrapMissingTableError(err); wrappedErr != nil {
	return nil, datastore.NoRevision, wrappedErr
	if pgxcommon.IsMissingTableError(err) {
	err = common.NewSchemaNotInitializedError(err)
	}

to keep the same pattern as the lines before

[miparnisari]

there are two constants with the same value defined elsewhere: postgresMissingTableErrorCode

please remove them in favor of this one

[miparnisari]

Suggested change

	return newXID, newSnapshot, timestamp, wrappedErr
	cterr = ....

[miparnisari]

Suggested change

	return nil, wrappedErr
	err = wrappedErr

otherwise you lose the errUnableToListNamespaces below

[miparnisari]

I also wonder if instead of littering all the datastore implementations with error handling, we could just prevent the server from coming up online when you have never ran migrate.

E.g. here

spicedb/internal/services/health/health.go

Lines 112 to 115 in 70cd757

	if !dsReady.IsReady {
	log.Ctx(ctx).Warn().Bool("datastoreReady", false).Msgf("datastore failed readiness checks: %s", dsReady.Message)
	return false
	}

if dsReady.message includes datastore is not migrated: currently at revision \"\", then we could tell people to run migrate. we'd need to write a generic test ensuring that all datastores return empty revision when they haven't been migrated.

I'm sure there are more changes required than this, perhaps an update to Server.Complete to include this check, but @josephschorr does this sound reasonable?

[ivanauth]

I replaced scattered IsMissingTableError checks with a single gRPC readiness middleware that blocks ALL requests until the datastore is migrated.

Before: 15 files with scattered IsMissingTableError checks (easy to miss spots)
After: One middleware gates ALL requests until migrated

[ivanauth]

Squashed to 2 commits and addressed all feedback:

• Replaced WrapMissingTableError with IsMissingTableError and NewSchemaNotInitializedError across datastores to preserve caller context
• Added case-insensitive “table not found” matching for Spanner
• Switched readiness gate to resenje.org/singleflight to fix lint
• Error code remains UNSPECIFIED until api#159 lands

Zedtoken prefix and Watch API behavior seem like separate concerns - happy to tackle in follow-up PRs.

Unit test failure (TestCertWatcherStart in pkg/x509util) is flaky and unrelated.