[Improvement] Document and guard the URI-shaped warehouse quirk on the Iceberg REST catalog backend

[nevzheng]

What would you like to be improved?

Purpose of this issue: document a known quirk and open discussion on how (or whether) to guard against it — not to mandate a specific fix.

Suggested priority: P3 (usability / minor). No data loss, no blocked functionality, and a trivial workaround exists (remove the property). The cost is purely a confusing first-run experience. Could be argued up to P2 given it's demo-visible and hits anyone copying a hive/jdbc example.

On the lakehouse-iceberg catalog with catalog-backend=rest, the warehouse property is a catalog selector (a name) — it's forwarded to the remote IRC as GET /v1/config?warehouse=<value>. On the hive/jdbc backends the same property is a storage-location URI. Because the name means two different things by backend, users copying a hive/jdbc example naturally set a URI like warehouse: "s3://warehouse/" on a REST catalog.

The catalog is then created without complaint (the value is accepted, never validated), and the failure is deferred to the first table/namespace operation:

org.apache.iceberg.exceptions.NoSuchWarehouseException:
Couldn't find Iceberg configuration for catalog s3://warehouse/

The error gives no hint that the fix is to remove the property (or set it to a catalog name). The value is forwarded verbatim: the REST branch of the required-check is skipped (IcebergCatalogWrapper.java:93), and the receiving IRC treats it as a catalog name (IcebergConfigOperations.java:128 -> IcebergCatalogWrapperManager.java:123).

From the end-user's perspective (UX matrix)

The confusion is best seen as a matrix over two dimensions — backend (rest vs hive/jdbc) and whether warehouse is a URI, a name, or omitted — comparing what the user intends against the outcome:

Backend	warehouse	User intends	What it means here	Outcome
hive/jdbc	s3://warehouse/ (URI)	"where data lives"	storage location ✅ matches	✅ Works — correct usage
hive/jdbc	(omitted)	"use a default"	required, no default	❌ Rejected at create, clear msg: The 'warehouse' parameter must have a value.
rest	(omitted)	"just connect"	default catalog	✅ Works — recommended REST setup
rest	sales (a name)	"use my sales catalog"	catalog selector ✅ matches	✅ Works — multi-catalog usage
rest	s3://warehouse/ (URI)	"where data lives" (copied from hive/jdbc)	catalog name to look up ✗ mismatch	❌ The trap. Create → 200; first op → NoSuchWarehouseException: Couldn't find Iceberg configuration for catalog s3://warehouse/
rest	"" (empty)	(accidental)	blank → default catalog	✅ Works

The whole issue is one cell — rest + URI. It's dangerous because on hive/jdbc that same URI value is required and correct, so the user's instinct ("warehouse = my S3 path") is right everywhere except here — and unlike the hive/jdbc mistake, this one is silently accepted at create and only fails later.

Flow (what actually happens)

flowchart TD
    U(["User creates a lakehouse-iceberg catalog<br/>with a 'warehouse' value"]) --> B{catalog-backend?}

    B -->|hive / jdbc| H["'warehouse' = STORAGE LOCATION"]
    H --> HV{value provided?}
    HV -->|"s3://warehouse/ (URI)"| HOK["✅ used as the data location — matches intent"]
    HV -->|omitted| HERR["❌ rejected at CREATE, clear message:<br/>'The warehouse parameter must have a value'"]

    B -->|rest| R["Gravitino forwards the value verbatim:<br/>GET /v1/config?warehouse=VALUE"]
    R --> IRC["IRC interprets VALUE as a CATALOG NAME to look up"]
    IRC --> F{name resolves?}
    F -->|"omitted → default, or real name e.g. 'sales'"| ROK["✅ 200 ConfigResponse — catalog selected"]
    F -->|"s3://warehouse/ (URI) — no such name"| RERR["❌ 404 → NoSuchWarehouseException<br/>CREATE returned 200; surfaces only on first use"]

    classDef good fill:#e6ffed,stroke:#2ea043,color:#003300;
    classDef bad fill:#ffebe9,stroke:#cf222e,color:#330000;
    class HOK,ROK good;
    class HERR,RERR bad;

Loading

The same s3://warehouse/ edge lands on green on the hive/jdbc side and red on the rest side — that mirror is the bug in one picture.

Reproduction (Docker quickstart)

Environment: apache/gravitino:latest (native API on :8090) + apache/gravitino-iceberg-rest:latest (IRC on :9001), as reported on the gravitino-irc-quickstart setup.

# Register a REST-backend catalog with a URI-shaped warehouse (the natural mistake
# when copying a jdbc/hive example). This SUCCEEDS — nothing rejects it.
curl -s -X POST http://localhost:8090/api/metalakes/v3check/catalogs \
  -H 'Accept: application/vnd.gravitino.v1+json' \
  -H 'Content-Type: application/json' \
  -d '{
    "name": "irc_probe",
    "type": "RELATIONAL",
    "provider": "lakehouse-iceberg",
    "comment": "probe with URI warehouse",
    "properties": {
      "catalog-backend": "rest",
      "uri": "http://gravitino-irc:9001/iceberg",
      "warehouse": "s3://warehouse/"
    }
  }'
# -> HTTP 200, catalog created.

# Any subsequent operation triggers the IRC handshake and fails:
curl -s http://localhost:8090/api/metalakes/v3check/catalogs/irc_probe/schemas
# -> {"code":1006,"type":"NoSuchWarehouseException",
#     "message":"... Couldn't find Iceberg configuration for catalog s3://warehouse/"}

# Removing 'warehouse' (or setting it to a real catalog name) works:
curl -s -X POST http://localhost:8090/api/metalakes/v3check/catalogs \
  -H 'Accept: application/vnd.gravitino.v1+json' \
  -H 'Content-Type: application/json' \
  -d '{
    "name": "irc_probe_ok",
    "type": "RELATIONAL",
    "provider": "lakehouse-iceberg",
    "properties": {
      "catalog-backend": "rest",
      "uri": "http://gravitino-irc:9001/iceberg"
    }
  }'
curl -s http://localhost:8090/api/metalakes/v3check/catalogs/irc_probe_ok/schemas   # -> 200

Tests that expose the issue

The server-side behavior is already covered — a URI warehouse is just another unknown catalog name and returns 404 (which the Iceberg REST client rethrows as NoSuchWarehouseException). Adding the URI case to the existing parameterized test makes it explicit:

// iceberg/iceberg-rest-server/.../rest/TestIcebergConfig.java  (testConfigWithNonExistentWarehouses)
@ValueSource(strings = {"invalid-catalog", "warehouse_123", "s3://warehouse/"})  // s3:// -> 404

Verified locally: the added "s3://warehouse/" case passes (returns 404) unchanged, confirming the value is treated as a catalog name, not a location.

Related: #5756 made warehouse optional for REST (fixed); #9810 proposes hiding it in the Web UI; PR #9811 documented the distinction. This issue covers the remaining gap — the REST-API / catalog-init path gives no runtime hint when a URI-shaped value is used.

How should we improve?

Filing to surface the quirk and gather maintainer input. Some options, in ascending effort / intrusiveness:

• Option A — Improve the error message only. Catch NoSuchWarehouseException on the REST path and append: "on the REST backend, warehouse selects a catalog by name; remove it or set a catalog name." Low effort, but the hint arrives late (first operation, not creation) and threads through Iceberg's exception.

• Option B — WARN at catalog init when warehouse is URI-shaped. In the branch that already special-cases REST (IcebergCatalogWrapper.java:93):

  } else if (IcebergCatalogBackend.REST.equals(catalogBackend)) {
    String warehouse = icebergConfig.get(IcebergConfig.CATALOG_WAREHOUSE);
    if (StringUtils.isNotBlank(warehouse) && warehouse.contains("://")) {
      LOG.warn(
          "The 'warehouse' value '{}' looks like a storage-location URI, but for the REST "
              + "catalog backend 'warehouse' selects a catalog by name. It is sent to the Iceberg "
              + "REST server as a catalog selector and will likely fail with "
              + "NoSuchWarehouseException. Remove 'warehouse' to use the default catalog, or set "
              + "it to the target catalog's name.",
          warehouse);
    }
  }

  Non-breaking, fires at creation, correctly scoped (hive/jdbc take the other branch). Pairs with a unit test asserting construction succeeds (no throw) plus the URI case in TestIcebergConfig.

• Option C — Reject a URI-shaped warehouse at creation. Fail fast with a clear message. Best UX for the common case, but warehouse is a server-defined opaque identifier per the Iceberg REST spec, and some non-Gravitino IRC servers legitimately use URI/ARN-shaped identifiers — a hard reject risks breaking valid federation.

• Option D — Hide/omit warehouse for REST in the Web UI. Tracked separately in [Improvement] remove warehouse properties in Iceberg catalog WEB UI if the catalog backend is REST #9810; complements but doesn't cover the REST-API path.

Leaning toward Option B as the smallest non-breaking change that fires early and gives the exact hint the raw error omits — while avoiding Option C's risk to the legitimate opaque-identifier case. But the intent here is to document the quirk and discuss; happy to go with whatever the maintainers prefer (including doc-only, since PR #9811 already covers part of it).

Reported by Mark Hoerth; filed on his behalf.

[nevzheng]

Proposal

Two concrete changes, plus an honest note on priority.

1) Clarify warehouse in the docs and the API contract. This is Mark's original option (1), in two layers:

(1a) User docs (markdown properties table) — already landed in PR #9811. The properties table now says "use catalog name for REST catalog backend," and the REST Catalog Backend section explains warehouse maps to the catalog name (empty = default). Small enhancement worth making: state explicitly that on REST it is not a storage location, and that some servers (e.g. S3 Tables) expect an opaque id/ARN.

(1b) API contract (OpenAPI) — still missing. CatalogCreateRequest.properties is an untyped map<string,string> with no documented keys, so warehouse's per-backend meaning lives only in prose/code.

The property bag stays a free-form map — we're just documenting one of its possible keys while leaving everything else open (additionalProperties is untouched, so it's non-breaking and the plugin model is unaffected). Concretely, we add the JSON Schema properties keyword inside the map object, alongside additionalProperties:

properties:                # the catalog property bag (unchanged: still an open map)
  type: object
  description: A map of properties for the catalog
  nullable: true
  default: { }
  properties:              # document ONE known key; the map stays open
    warehouse:
      type: string
      description: >
        Behavior depends on catalog-backend (lakehouse-iceberg):
          • hive / jdbc: a STORAGE LOCATION (e.g. s3a://bucket/wh, hdfs://ns/path). Required.
          • rest: a CATALOG SELECTOR forwarded to the Iceberg REST server as ?warehouse=
            (a catalog name for Gravitino's IRC; an opaque id/ARN for some servers, e.g. S3 Tables).
            Optional; omit for the default catalog. NOT a storage location.
  additionalProperties:    # unchanged: every other key is still a free string
    type: string

(Sketch, not a final shape — open to modeling it as a oneOf/discriminator keyed on catalog-backend instead, or docs-only. Flagging that the OpenAPI here is doc-only; enforcement lives in IcebergCatalogPropertiesMetadata, which already types these keys. Feedback on the modeling approach welcome.)

This makes the contract match what the code already declares and removes the surprise (POLA).

2) Validate warehouse at create for all backends — no delayed failures. The core bug is that create returns 200 and the mistake only surfaces on the first operation. Gravitino already has the mechanism (IcebergCatalogOperations.testConnection → listNamespace, which forces the resolve); the fix is to run that validation as part of create so any warehouse problem fails at create, with an actionable hint — not later. Crucially, validation must resolve, not pattern-match the value: a URI/ARN warehouse is legitimately correct for S3 Tables, so we let the backend be the oracle rather than rejecting on shape.

Compact flow (proposed create-time validation):

flowchart TD
    C["POST /catalogs (create)"] --> B{catalog-backend?}
    B -->|hive / jdbc| L{warehouse present<br/>& reachable as a location?}
    L -->|yes| OK1["✅ create"]
    L -->|no| E1["❌ fail fast:<br/>'warehouse required; must be a storage location'"]
    B -->|rest| V["eager resolve: GET /v1/config?warehouse=…"]
    V --> R{resolves?}
    R -->|yes / omitted → default| OK2["✅ create"]
    R -->|no| E2["❌ fail fast:<br/>'warehouse did not resolve as a catalog selector —<br/>remove it for the default catalog, or use a catalog name'"]

    classDef good fill:#e6ffed,stroke:#2ea043,color:#003300;
    classDef bad fill:#ffebe9,stroke:#cf222e,color:#330000;
    class OK1,OK2 good;
    class E1,E2 bad;

Loading

Same principle for every backend: attempt to use warehouse at create, so it fails now instead of later.

On priority — yes, this is an edge case. The workaround is trivial (omit or rename the property), so this is low-priority, not urgent. But it's the kind of paper cut that's cheap to prevent and disproportionately confusing when hit — silent accept + a cryptic delayed NoSuchWarehouseException, on the REST surface, in a demo/eval setting. Filing it as the right small fix, not a fire. Even part (1) alone (docs) is worth landing on its own.

References

• Spec today: CatalogCreateRequest.properties — untyped string map
• Why the surprise is the bug: Principle of Least Astonishment
• Ideal long-term model (typed per backend): OpenAPI polymorphism / discriminator
• Why validation must resolve, not reject on shape (URI/ARN warehouse is valid here): AWS S3 Tables Iceberg REST — warehouse = bucket ARN
• Related: [Bug report] The API for creating an Iceberg catalog has a deprecated parameter "warehouse" that is required in the HTTP payload #5756 (made warehouse optional for REST), [Improvement] remove warehouse properties in Iceberg catalog WEB UI if the catalog backend is REST #9810 (hide warehouse in Web UI for REST), PR [MINOR]: doc(iceberg): add warehouse document for REST catalog backend for Iceberg #9811 (documented the distinction in the catalog docs)

[roryqi]

Agree with adding the documents. I noticed this issue before. I revised other REST catalog services. None of them use a storage path as the warehouse, too.