Repository navigation
Spec: Add named storage configurations to the management API - #5721
Draft
sririshindra wants to merge 1 commit into
Draft
sririshindra wants to merge 1 commit into
sririshindra wants to merge 1 commit into
Conversation
Adds an optional `storageConfigInfos` array to `Catalog` and
`UpdateCatalogRequest`, the `StorageConfigInfos` list schema, and the
`/catalogs/{catalogName}/storage-configs[/{storageConfigName}]` endpoints
for managing one named storage configuration at a time.
This is the API part of apache#5556. The server does not implement it yet: the
new endpoints answer with the generated 501 default, and the server
ignores `storageConfigInfos`.
`storageConfigInfos` is `nullable: true` and `openApiNullable` is disabled
for the model module, so an omitted array stays `null` instead of becoming
an empty list, and the field keeps a plain `List<StorageConfigInfo>` type.
Existing callers of the all-args constructors pass `null` for the new
argument.
sririshindra
force-pushed
the
named-storage-spec
branch
from
October 6, 2026 23:49
41070bc to
9ff2c1f
Compare
sririshindra
commented
Oct 7, 2026
| storageConfigInfos: | ||
| type: array | ||
| nullable: true | ||
| description: Optional list of additional named storage configurations held by the catalog, alongside its default storageConfigInfo. Each entry is identified by its storageName, which is required. Names are trimmed, must match ^[a-zA-Z0-9_-]{1,128}$, are case-sensitive, must be unique within the list, and must differ from the storageName of the default storageConfigInfo. Unlike the default storageConfigInfo, each entry must list at least one allowedLocations entry. |
Contributor
Author
There was a problem hiding this comment.
A note on why named entries must list at least one allowed location, while the default configuration doesn't have to:
- When the default
storageConfigInfohas noallowedLocations, Polaris uses the catalog'sdefault-base-locationinstead (CatalogEntity.Builder#processStorageConfigurationInfo). A named entry has no equivalent to fall back on. It usually points at different storage than the catalog default (another bucket, another account, sometimes another storage type), so borrowing the catalog's base location would either allow the wrong place or fail the storage-type prefix check. - An empty list doesn't mean "anywhere".
StorageLocationValidator.validateAllowedLocationsaccepts a location only if it is under one of the allowed locations. With an empty list, every table create, commit and credential request that uses the entry would fail with 403. It's better to reject the entry with a 400 when it is defined than to have it fail later on every table that uses it. - The cross-catalog overlap check also works from
allowedLocations, so an entry without any would never be checked against other catalogs.
Happy to change this if people would prefer a different rule.
Contributor
|
Is this ready for review? 😅 |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
#5556 adds named storage configurations to catalogs, but it is large. This PR takes only its API
change so the shape of the management API can be agreed on first. The implementation will follow
in smaller PRs built on this one. #5556 stays open for anyone who wants to see the whole change.
Background: today a catalog has exactly one storage configuration. The design discussed on dev@
(thread) lets a catalog also
hold named storage configurations next to its default one, so that later on namespaces and
tables can use a different bucket, role or cloud than the catalog default.
Related: #5556, #4023, #3409.
What changes
spec/polaris-management-service.ymlCatalogandUpdateCatalogRequestget an optionalstorageConfigInfosarray. The existingstorageConfigInfostays the catalog's default configuration.storageName. Names are trimmed and must match^[a-zA-Z0-9_-]{1,128}$. They are case-sensitive, must be unique within the list, and mustdiffer from the default configuration's
storageName. Unlike the default, every named entrymust list at least one allowed location, because there is no base location to fall back to.
storageConfigInfosout keeps the current set. Sending an array replacesthe whole set, and an empty array removes them all.
resending all the others:
GET /catalogs/{catalogName}/storage-configslists the named configurations (not thedefault).
GET /catalogs/{catalogName}/storage-configs/{storageConfigName}PUT /catalogs/{catalogName}/storage-configs/{storageConfigName}creates or replaces oneentry. If the body sets
storageName, it must match the path; if the body leaves it out,the path name is used.
DELETE /catalogs/{catalogName}/storage-configs/{storageConfigName}api/management-model/build.gradle.ktsSets
openApiNullable=false. The update rule above needs "field left out" to be different from"empty array". Marking the field
nullable: truemakes the generated defaultnullinstead ofan empty list. With
openApiNullabledisabled, the Java type stays a plainList<StorageConfigInfo>instead ofJsonNullable<...>. The catalog API and the semantic-modelsextension already use the same setting. In this spec, only the two new fields are
nullable, sono other generated model changes.
Existing callers
The new field adds a trailing argument to the all-args constructors of
CatalogandUpdateCatalogRequest. Every existing call passesnull, which accounts for all of the one-linetest changes.
Behavior after this PR
PolarisCatalogsApiService.storageConfigInfosin a request but ignores it, and doesn't include it inresponses. Because the models use
JsonInclude.NON_NULL, catalog responses look exactly asbefore.
Next steps
storageConfigInfoson catalog create, update and get. This will be behinda new feature flag,
ENABLE_NAMED_STORAGE_CONFIGURATIONS, which defaults to off./storage-configsendpoints.Using a named configuration for a namespace or table, and vending credentials from it, comes after
that and is not part of this spec.
Testing
No new tests, because this PR has no server behavior. Existing tests that build
CatalogorUpdateCatalogRequestnow pass the extra argument. CI covers the full build.Checklist
CHANGELOG.md(if needed): not yet; the entry comes with the implementationsite/content/in-dev/unreleased(if needed): not yet; nothing is usable until the implementation lands