feat(rest): support rest catalog for databases, tables and snapshots

Add a REST catalog selected via the "metastore=rest" option:

- RestHttpClient: blocking libcurl client with exponential-backoff
  retries. 429/503 are retried for every method, a transient transport
  error only for an idempotent one; a server closing the connection
  without responding is not retried. A Retry-After header is honored in
  both the delta-seconds and the HTTP-date form with a
  locale-independent parse, and is never shortened: the overall retry
  budget bounds it while the per-sleep cap bounds only the backoff. Easy
  handles are pooled, so a request reuses the previous connection
  instead of paying a handshake every time; TLS below 1.2 is refused and
  requests carry a default user agent. Every transported request is
  debug-logged with the server request id.
- RestApi: HTTP + JSON protocol layer. "/v1/config" merges the server
  options into the client ones (overrides > client options > defaults,
  an override sent as null unsets the client option), "header." options
  are sent as request headers and a listing follows the page tokens to
  the last page. A failed request maps to a Status by the code of the
  parsed error body, falling back to the http status, and carries a
  RestErrorDetail with that code so callers can tell e.g. an
  authentication failure from another IO error. A response body is never
  echoed into an error, since it may carry credentials, so a body that
  is not an error object is only reported as unparsable.
- RestCatalog: database, table and snapshot operations, with
  "table-default." option defaults and a schema conversion computing the
  highest field id (ids at or above the new
  SpecialFieldIds::SYSTEM_FIELD_ID_START are excluded, duplicated ids
  are rejected at any nesting level). A branch stays in the object name
  so the server resolves it and returns the branch's own schema, the
  default branch "main" being matched ignoring case and addressed as the
  bare table. The "sys" database serves the local global system tables
  like FileSystemCatalog.
- Bear token authentication provider ("bear" is the protocol's
  historical spelling of "bearer"). RestAuthParameter::Create
  url-encodes the query parameter values a signing provider signs.
- CatalogOptions: public catalog-level option keys (metastore, uri,
  token, token.provider, "table-default." prefix), kept separate from
  the table-level CoreOptions.
- CatalogUtils: system database, system table and branch checks shared
  by FileSystemCatalog and RestCatalog.
- RestUtil::ExtractRequestId: the request id lookup shared by the debug
  log line and the error message, falling back to any header whose name
  carries a request id since a gateway may report it under its own name.

The pieces overlapping with the object store file systems move to
common/utils: UrlUtils carries both URL encoding flavors
(form-urlencoded for the REST api, RFC 3986 for S3), the generic HTTP
client moves there from common/fs, and the new SensitiveConfigUtils
redacts credentials for every surface that may expose them - the
sys.catalog_options table masks the value of a credential-carrying
option key, keeping at most a short tail of an identifier-like value,
and a server error message carrying a secret marker is redacted as a
whole.

libcurl is used from the system, so the new PAIMON_ENABLE_REST option
defaults to OFF like the other optional components with external
requirements; its CMake detection is a single find_package guarded by
PAIMON_ENABLE_S3 OR PAIMON_ENABLE_REST.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Author: SteNicholas

CMakeLists.txt

@@ -68,9 +68,17 @@ option(PAIMON_ENABLE_LUMINA "Whether to enable lumina vector index" OFF)
 option(PAIMON_ENABLE_LUCENE "Whether to enable lucene index" OFF)
 option(PAIMON_ENABLE_TANTIVY
        "Whether to enable tantivy-fulltext global index (Rust FFI, experimental)" OFF)
+option(PAIMON_ENABLE_REST "Whether to enable the rest catalog (requires libcurl)" OFF)
 if(PAIMON_ENABLE_ORC)
     add_definitions(-DPAIMON_ENABLE_ORC)
 endif()
+if(PAIMON_ENABLE_REST)
+    add_definitions(-DPAIMON_ENABLE_REST)
+endif()
+# libcurl backs the HTTP client shared by the S3 file system and the rest catalog.
+if(PAIMON_ENABLE_S3 OR PAIMON_ENABLE_REST)
+    find_package(CURL REQUIRED)
+endif()
 if(PAIMON_ENABLE_AVRO)
     add_definitions(-DPAIMON_ENABLE_AVRO)
 endif()
@@ -498,9 +506,6 @@ install(FILES "${CMAKE_CURRENT_BINARY_DIR}/PaimonConfig.cmake"
 
 config_summary_message()
 
-if(PAIMON_ENABLE_S3)
-    find_package(CURL REQUIRED)
-endif()
 add_subdirectory(src/paimon)
 add_subdirectory(src/paimon/fs/local)
 if(PAIMON_ENABLE_JINDO)

ci/scripts/build_paimon.sh

@@ -146,6 +146,7 @@ CMAKE_ARGS=(
     "-DPAIMON_ENABLE_LUMINA=${ENABLE_LUMINA}"
     "-DPAIMON_ENABLE_LUCENE=ON"
     "-DPAIMON_ENABLE_TANTIVY=${ENABLE_TANTIVY}"
+    "-DPAIMON_ENABLE_REST=ON"
     "-DPAIMON_LINT_GIT_TARGET_COMMIT=${lint_git_target_commit}"
 )
 

docs/source/building.rst

@@ -126,6 +126,7 @@ boolean flags to ``cmake``.
 * ``-DPAIMON_ENABLE_LUMINA=ON``: Support for the Lumina vector index.
 * ``-DPAIMON_ENABLE_LUCENE=ON``: Support for Lucene full-text search indexes
 * ``-DPAIMON_ENABLE_TANTIVY=ON``: Enable the experimental Tantivy full-text index Rust FFI.
+* ``-DPAIMON_ENABLE_REST=ON``: Support for the REST catalog (``metastore=rest``), requires the libcurl development package.
 
 Third-party dependency source
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

docs/source/user_guide/catalog.rst

@@ -23,15 +23,63 @@ C++ Paimon provides a :ref:`Catalog abstraction <cpp-api-catalog>` to manage the
 abstraction provides a series of ways to help you better integrate with computing engines. We always
 recommend that you use Catalog to access the Paimon table.
 
+C++ Paimon supports two metastores, selected with the catalog option
+``metastore``: the filesystem metastore (default) and the REST metastore.
+
 Filesystem Catalog
 ~~~~~~~~~~~~~~~~~~
-C++ Paimon catalog currently support one types of metastores filesystem metastore (default),
-which stores both metadata and table files in filesystems.
+The filesystem metastore (``metastore=filesystem``, the default) stores both
+metadata and table files in filesystems. The ``root_path`` argument of
+``Catalog::Create`` is the warehouse directory holding the databases and tables.
+
+REST Catalog
+~~~~~~~~~~~~
+With the REST metastore (``metastore=rest``), catalog metadata is managed by a
+remote catalog server exposed through a REST API; table data itself is still read
+and written through the table paths returned by the server. See `Java Paimon REST
+Catalog <https://paimon.apache.org/docs/master/concepts/rest/>`_ for the concept
+and the server-side protocol.
+
+REST catalog support is an optional build component: configure the build with
+``-DPAIMON_ENABLE_REST=ON`` (see :ref:`cpp_build_optional_components`).
+
+When ``metastore=rest``, the ``root_path`` argument of ``Catalog::Create`` is not
+a filesystem path but the warehouse (instance) name under which the tables are
+registered on the REST server. The catalog is configured through the
+``CatalogOptions`` keys:
+
+* ``metastore``: must be ``rest`` to select the REST catalog.
+* ``uri``: server url of the REST catalog server.
+* ``token.provider``: authentication provider of the REST catalog; currently only
+  ``bear`` is supported (the protocol's historical spelling of "bearer").
+* ``token``: token of the ``bear`` token provider.
+* ``table-default.<key>``: table option defaults applied when a created table
+  left ``<key>`` unset.
+* ``header.<name>``: sent as the ``<name>`` http header on every request to the
+  server. The server may configure headers of its own through the ``/v1/config``
+  endpoint, which are merged with these as any other option is.
+
+.. code-block:: cpp
+
+   std::map<std::string, std::string> options = {
+       {"metastore", "rest"},
+       {"uri", "http://127.0.0.1:8080"},
+       {"token.provider", "bear"},
+       {"token", "<token>"},
+   };
+   PAIMON_ASSIGN_OR_RAISE(std::unique_ptr<paimon::Catalog> catalog,
+                          paimon::Catalog::Create(/*root_path=*/"my_instance", options));
+
+On creation the catalog queries the server's ``/v1/config`` endpoint and merges
+its response with the options above: the server's overrides win over the client
+options, which in turn win over the server's defaults.
 
-.. note::
+Databases and tables are then created, listed, loaded, renamed and dropped
+through the regular ``Catalog`` API, and table snapshots can be listed through
+``Catalog::ListSnapshots``.
 
-  Current C++ Paimon only supports filesystem catalog. In the future, we will
-  support REST catalog.
-  By using the Paimon REST catalog, changes to the catalog will be directly stored
-  in a remote catalog server which exposed through REST API. See `Java Paimon REST
-  Catalog <https://paimon.apache.org/docs/master/concepts/rest/overview/>`_.
+The C++ REST catalog covers the database, table and snapshot operations of the
+``Catalog`` API. The parts of the Java REST catalog that have no C++ counterpart
+yet — altering a database or a table, views, functions, partitions, tags, branch
+management and consumers — are not supported, and neither is the ``dlf`` token
+provider.

include/paimon/catalog/catalog.h

@@ -58,7 +58,9 @@ class PAIMON_EXPORT Catalog {
 
     /// %Factory method for creating a `Catalog` instance.
     ///
-    /// @param root_path Path to the root directory where the catalog is located.
+    /// @param root_path Path to the root directory where the catalog is located. For the
+    ///                  REST catalog (`CatalogOptions::METASTORE` set to "rest") this is
+    ///                  instead the warehouse (instance) name registered on the server.
     /// @param options Configuration options for catalog initialization.
     /// @param file_system Specifies the file system for file operations.
     ///                    If not set, use default file system (configured in

include/paimon/catalog_options.h

@@ -0,0 +1,45 @@
+/*
+ * Copyright 2026-present Alibaba Inc.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+#include "paimon/visibility.h"
+
+namespace paimon {
+
+/// Catalog-level configuration option keys; table-level keys live in `Options`.
+struct PAIMON_EXPORT CatalogOptions {
+    /// "metastore" - Metastore of the paimon catalog.
+    /// Supported values are "filesystem" (default) and "rest".
+    static const char METASTORE[];
+
+    /// "uri" - Server url of the REST catalog. Only used when METASTORE is "rest".
+    static const char URI[];
+
+    /// "token" - Token of the "bear" token provider of the REST catalog.
+    static const char TOKEN[];
+
+    /// "token.provider" - Authentication provider of the REST catalog. Only "bear" is
+    /// supported ("bear" is the protocol's historical spelling of "bearer", do not "fix" it).
+    static const char TOKEN_PROVIDER[];
+
+    /// "table-default." - Prefix of the catalog options that provide table option
+    /// defaults: "table-default.<key>=<value>" applies "<key>=<value>" to a created
+    /// table when the caller left "<key>" unset.
+    static const char TABLE_DEFAULT_OPTION_PREFIX[];
+};
+
+}  // namespace paimon

include/paimon/utils/special_field_ids.h

@@ -42,6 +42,10 @@ class SpecialFieldIds {
 
     /// Special field ID reserved for index score. Value: CPP_FIELD_ID_END - 1
     inline static constexpr int32_t INDEX_SCORE = CPP_FIELD_ID_END - 1;
+
+    /// Lowest field ID reserved for system fields; IDs at or above it are excluded from the
+    /// highest field ID of a schema. Value: INT32_MAX / 2
+    inline static constexpr int32_t SYSTEM_FIELD_ID_START = std::numeric_limits<int32_t>::max() / 2;
 };
 
 }  // namespace paimon

src/paimon/CMakeLists.txt

@@ -15,6 +15,7 @@
 # limitations under the License.
 
 set(PAIMON_COMMON_SRCS
+    common/catalog_options.cpp
     common/compression/block_compression_factory.cpp
     common/compression/block_compressor.cpp
     common/compression/block_decompressor.cpp
@@ -179,13 +180,20 @@ set(PAIMON_COMMON_SRCS
     common/utils/roaring_bitmap32.cpp
     common/utils/roaring_bitmap64.cpp
     common/utils/row_range_index.cpp
+    common/utils/sensitive_config_utils.cpp
     common/utils/status.cpp
-    common/utils/string_utils.cpp)
+    common/utils/string_utils.cpp
+    common/utils/url_utils.cpp)
 
+# The shared HTTP client is used by both the object store file systems and the
+# rest catalog.
+set(PAIMON_CURL_LINK_LIBS)
+if(PAIMON_ENABLE_S3 OR PAIMON_ENABLE_REST)
+    list(APPEND PAIMON_COMMON_SRCS common/utils/http_client.cpp)
+    set(PAIMON_CURL_LINK_LIBS CURL::libcurl)
+endif()
 if(PAIMON_ENABLE_S3)
-    list(APPEND PAIMON_COMMON_SRCS common/fs/http_client.cpp
-         common/fs/object_store_file_system.cpp)
-    set(PAIMON_OBJECT_STORE_LINK_LIBS CURL::libcurl)
+    list(APPEND PAIMON_COMMON_SRCS common/fs/object_store_file_system.cpp)
 endif()
 
 set(PAIMON_CORE_SRCS
@@ -228,6 +236,7 @@ set(PAIMON_CORE_SRCS
     core/casting/timestamp_to_timestamp_cast_executor.cpp
     core/casting/casting_utils.cpp
     core/catalog/catalog.cpp
+    core/catalog/catalog_utils.cpp
     core/catalog/file_system_catalog.cpp
     core/catalog/identifier.cpp
     core/core_options.cpp
@@ -405,6 +414,18 @@ set(PAIMON_CORE_SRCS
     core/utils/snapshot_manager.cpp
     core/utils/tag_manager.cpp)
 
+if(PAIMON_ENABLE_REST)
+    list(APPEND
+         PAIMON_CORE_SRCS
+         rest/rest_http_client.cpp
+         rest/resource_paths.cpp
+         rest/rest_api.cpp
+         rest/rest_auth.cpp
+         rest/rest_catalog.cpp
+         rest/rest_messages.cpp
+         rest/rest_util.cpp)
+endif()
+
 add_paimon_lib(paimon
                SOURCES
                ${PAIMON_COMMON_SRCS}
@@ -418,7 +439,7 @@ add_paimon_lib(paimon
                xxhash
                Threads::Threads
                RapidJSON
-               ${PAIMON_OBJECT_STORE_LINK_LIBS}
+               ${PAIMON_CURL_LINK_LIBS}
                STATIC_LINK_LIBS
                arrow
                tbb
@@ -428,14 +449,21 @@ add_paimon_lib(paimon
                xxhash
                Threads::Threads
                RapidJSON
-               ${PAIMON_OBJECT_STORE_LINK_LIBS}
+               ${PAIMON_CURL_LINK_LIBS}
                SHARED_LINK_FLAGS
                ${PAIMON_VERSION_SCRIPT_FLAGS})
 
 add_subdirectory(common/file_index)
 add_subdirectory(common/global_index)
 
 if(PAIMON_BUILD_TESTS)
+    # The shared HTTP client is compiled only for the components that need libcurl,
+    # so its test follows the same gate.
+    set(PAIMON_COMMON_HTTP_CLIENT_TEST_SRCS)
+    if(PAIMON_ENABLE_S3 OR PAIMON_ENABLE_REST)
+        set(PAIMON_COMMON_HTTP_CLIENT_TEST_SRCS common/utils/http_client_test.cpp)
+    endif()
+
     add_paimon_test(memory_test
                     SOURCES
                     common/memory/memory_pool_test.cpp
@@ -593,10 +621,13 @@ if(PAIMON_BUILD_TESTS)
                     common/io/cache/lru_cache_test.cpp
                     common/utils/byte_range_combiner_test.cpp
                     common/utils/scope_guard_test.cpp
+                    common/utils/sensitive_config_utils_test.cpp
                     common/utils/serialization_utils_test.cpp
                     common/utils/status_test.cpp
                     common/utils/stream_utils_test.cpp
                     common/utils/string_utils_test.cpp
+                    common/utils/url_utils_test.cpp
+                    ${PAIMON_COMMON_HTTP_CLIENT_TEST_SRCS}
                     common/utils/range_test.cpp
                     common/utils/uuid_test.cpp
                     common/utils/decimal_utils_test.cpp
@@ -874,4 +905,20 @@ if(PAIMON_BUILD_TESTS)
                     EXTRA_INCLUDES
                     ${JINDOSDK_INCLUDE_DIR})
 
+    if(PAIMON_ENABLE_REST)
+        add_paimon_test(rest_test
+                        SOURCES
+                        rest/rest_http_client_test.cpp
+                        rest/mock_rest_server.cpp
+                        rest/resource_paths_test.cpp
+                        rest/rest_catalog_test.cpp
+                        rest/rest_messages_test.cpp
+                        rest/rest_util_test.cpp
+                        STATIC_LINK_LIBS
+                        paimon_shared
+                        test_utils_static
+                        ${TEST_STATIC_LINK_LIBS}
+                        ${GTEST_LINK_TOOLCHAIN})
+    endif()
+
 endif()

src/paimon/common/catalog_options.cpp

@@ -0,0 +1,27 @@
+/*
+ * Copyright 2026-present Alibaba Inc.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "paimon/catalog_options.h"
+
+namespace paimon {
+
+const char CatalogOptions::METASTORE[] = "metastore";
+const char CatalogOptions::URI[] = "uri";
+const char CatalogOptions::TOKEN[] = "token";
+const char CatalogOptions::TOKEN_PROVIDER[] = "token.provider";
+const char CatalogOptions::TABLE_DEFAULT_OPTION_PREFIX[] = "table-default.";
+
+}  // namespace paimon