Skip to content

Spec: Add named storage configurations to the management API - #5721

Draft
sririshindra wants to merge 1 commit into
apache:mainfrom
sririshindra:named-storage-spec
Draft

sririshindra wants to merge 1 commit into
apache:mainfrom
sririshindra:named-storage-spec

Conversation

@sririshindra

@sririshindra sririshindra commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

Why

#5556 adds named storage configurations to catalogs, but it is large. This PR takes only its API
change so the shape of the management API can be agreed on first. The implementation will follow
in smaller PRs built on this one. #5556 stays open for anyone who wants to see the whole change.

Background: today a catalog has exactly one storage configuration. The design discussed on dev@
(thread) lets a catalog also
hold named storage configurations next to its default one, so that later on namespaces and
tables can use a different bucket, role or cloud than the catalog default.

Related: #5556, #4023, #3409.

What changes

spec/polaris-management-service.yml

  • Catalog and UpdateCatalogRequest get an optional storageConfigInfos array. The existing
    storageConfigInfo stays the catalog's default configuration.
  • Each entry is identified by its storageName. Names are trimmed and must match
    ^[a-zA-Z0-9_-]{1,128}$. They are case-sensitive, must be unique within the list, and must
    differ from the default configuration's storageName. Unlike the default, every named entry
    must list at least one allowed location, because there is no base location to fall back to.
  • On update, leaving storageConfigInfos out keeps the current set. Sending an array replaces
    the whole set, and an empty array removes them all.
  • New endpoints for managing one entry at a time, so that changing one entry doesn't require
    resending all the others:
    • GET /catalogs/{catalogName}/storage-configs lists the named configurations (not the
      default).
    • GET /catalogs/{catalogName}/storage-configs/{storageConfigName}
    • PUT /catalogs/{catalogName}/storage-configs/{storageConfigName} creates or replaces one
      entry. If the body sets storageName, it must match the path; if the body leaves it out,
      the path name is used.
    • DELETE /catalogs/{catalogName}/storage-configs/{storageConfigName}

api/management-model/build.gradle.kts

Sets openApiNullable=false. The update rule above needs "field left out" to be different from
"empty array". Marking the field nullable: true makes the generated default null instead of
an empty list. With openApiNullable disabled, the Java type stays a plain
List<StorageConfigInfo> instead of JsonNullable<...>. The catalog API and the semantic-models
extension already use the same setting. In this spec, only the two new fields are nullable, so
no other generated model changes.

Existing callers

The new field adds a trailing argument to the all-args constructors of Catalog and
UpdateCatalogRequest. Every existing call passes null, which accounts for all of the one-line
test changes.

Behavior after this PR

  • The new endpoints return 501 Not Implemented, the generated default in
    PolarisCatalogsApiService.
  • The server accepts storageConfigInfos in a request but ignores it, and doesn't include it in
    responses. Because the models use JsonInclude.NON_NULL, catalog responses look exactly as
    before.

Next steps

  1. Store and validate storageConfigInfos on catalog create, update and get. This will be behind
    a new feature flag, ENABLE_NAMED_STORAGE_CONFIGURATIONS, which defaults to off.
  2. Include named configurations in the check for overlapping catalog locations.
  3. Implement the /storage-configs endpoints.

Using a named configuration for a namespace or table, and vending credentials from it, comes after
that and is not part of this spec.

Testing

No new tests, because this PR has no server behavior. Existing tests that build Catalog or
UpdateCatalogRequest now pass the extra argument. CI covers the full build.

Checklist

  • 🛡️ Don't disclose security issues! (contact security@apache.org)
  • 🔗 Clearly explained why the changes are needed, or linked related issues: Related to Support multiple named storage configurations per catalog #5556
  • 🧪 Added/updated tests with good coverage, or manually tested (and explained how)
  • 💡 Added comments for complex logic
  • 🧾 Updated CHANGELOG.md (if needed): not yet; the entry comes with the implementation
  • 📚 Updated documentation in site/content/in-dev/unreleased (if needed): not yet; nothing is usable until the implementation lands

Adds an optional `storageConfigInfos` array to `Catalog` and
`UpdateCatalogRequest`, the `StorageConfigInfos` list schema, and the
`/catalogs/{catalogName}/storage-configs[/{storageConfigName}]` endpoints
for managing one named storage configuration at a time.

This is the API part of apache#5556. The server does not implement it yet: the
new endpoints answer with the generated 501 default, and the server
ignores `storageConfigInfos`.

`storageConfigInfos` is `nullable: true` and `openApiNullable` is disabled
for the model module, so an omitted array stays `null` instead of becoming
an empty list, and the field keeps a plain `List<StorageConfigInfo>` type.
Existing callers of the all-args constructors pass `null` for the new
argument.
storageConfigInfos:
type: array
nullable: true
description: Optional list of additional named storage configurations held by the catalog, alongside its default storageConfigInfo. Each entry is identified by its storageName, which is required. Names are trimmed, must match ^[a-zA-Z0-9_-]{1,128}$, are case-sensitive, must be unique within the list, and must differ from the storageName of the default storageConfigInfo. Unlike the default storageConfigInfo, each entry must list at least one allowedLocations entry.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A note on why named entries must list at least one allowed location, while the default configuration doesn't have to:

  • When the default storageConfigInfo has no allowedLocations, Polaris uses the catalog's default-base-location instead (CatalogEntity.Builder#processStorageConfigurationInfo). A named entry has no equivalent to fall back on. It usually points at different storage than the catalog default (another bucket, another account, sometimes another storage type), so borrowing the catalog's base location would either allow the wrong place or fail the storage-type prefix check.
  • An empty list doesn't mean "anywhere". StorageLocationValidator.validateAllowedLocations accepts a location only if it is under one of the allowed locations. With an empty list, every table create, commit and credential request that uses the entry would fail with 403. It's better to reject the entry with a 400 when it is defined than to have it fail later on every table that uses it.
  • The cross-catalog overlap check also works from allowedLocations, so an entry without any would never be checked against other catalogs.

Happy to change this if people would prefer a different rule.

@dimas-b

dimas-b commented Oct 7, 2026

Copy link
Copy Markdown
Contributor

Is this ready for review? 😅

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants