Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
77 commits
Select commit Hold shift + click to select a range
eafed3d
[CORE] Specification for an HTTP REST catalog
kbendick Nov 16, 2021
fb81cfb
Round out the comments a bit
kbendick Nov 16, 2021
dda40a3
Add a description for tableExists route
kbendick Nov 17, 2021
28a4f9f
Fix some typos and remove unnecessary field from a schema definition
kbendick Nov 17, 2021
495c4c5
Regenerate the docs
kbendick Nov 17, 2021
08cc6a9
Remove unnecessary comment block
kbendick Nov 17, 2021
eebb775
Specify that v1 represents the catalog API version and use POST /v1/c…
kbendick Nov 17, 2021
484a573
Make version 1 instead of specifying v1 in the title so the generated…
kbendick Nov 17, 2021
12a61be
Change descriptions to Unauthorized
kbendick Nov 22, 2021
dea7a62
Remove prod from example namespaces
kbendick Nov 22, 2021
54263ae
Update comment
kbendick Nov 22, 2021
3a1bb09
Rename route to /v1/namespaces/{namespace}/tables
kbendick Nov 22, 2021
4aecdda
Change renamTable from POST /v1/tables/renameTable to PUT /v1/namespa…
kbendick Nov 22, 2021
42834b3
Remove /list from GET /v1/namespaces, leaving namespace as query para…
kbendick Nov 22, 2021
fd1591c
Update response for getNamespaceMetadata and remove redundant getName…
kbendick Nov 22, 2021
a4e2e59
Update dropTable to use purgeRequested given that servers dont necess…
kbendick Nov 22, 2021
20aa1c1
Remove unused POST config route for now
kbendick Nov 22, 2021
46f8c1a
Use parents as query parameter to listNamespaces and remove no_such_i…
kbendick Nov 22, 2021
cee5dc0
Update configuration route description
kbendick Dec 1, 2021
58f94e3
Remove namespace description and references to pagination
kbendick Dec 1, 2021
3fa7bcd
Use new table name description
kbendick Dec 1, 2021
28d44a2
Remove reference to multiple catalogs
kbendick Dec 1, 2021
8044e56
Remove commit table - will handle transacions in a follow up
kbendick Dec 1, 2021
460a57b
Change renameTable back to POST /v1/tables/renameTable
kbendick Dec 1, 2021
1af20b3
Add UnsupportedOperationException to the same table
kbendick Dec 1, 2021
56fff46
Move loadNamespaceMetadata to GET /v1/namesapces/{namespace} and remo…
kbendick Dec 1, 2021
b2c10c9
Remove redundant PUT properties
kbendick Dec 1, 2021
651c419
Update removeProperties to be DELETE /v1/namespaces/{namespace}/prope…
kbendick Dec 1, 2021
f3e65df
Allow setProperties to return UnsupportedOperationException per the i…
kbendick Dec 1, 2021
12622c7
Move createNamespace route to POST /v1/namespaces and update properti…
kbendick Dec 1, 2021
5e342fd
Remove tableLocation from CreateTableResponse
kbendick Dec 1, 2021
4fe74d2
Change removeProperties to use put for now as delete cannot have a re…
kbendick Dec 2, 2021
e3cc497
Add in 202 - Accepted for dropTable
kbendick Dec 2, 2021
2b396b7
Revert to using 404 instead
kbendick Dec 6, 2021
03a3708
Get rid of CreateTableRequest - will handle table operations in a fol…
kbendick Dec 6, 2021
6ef7944
Remove duplicated response code map keys after reverting to 404s - re…
kbendick Dec 6, 2021
80501ff
Fix validation errors on multiple examples
kbendick Dec 6, 2021
06d1de6
Further define the error classes
kbendick Dec 6, 2021
bf42455
Add in IcebergHttpResponse object to more easily discuss the generic …
kbendick Dec 6, 2021
a71b6c9
Add in IcebergHttpResponse object to more easily discuss the generic …
kbendick Dec 6, 2021
7f0e848
Regenerate the docs
kbendick Dec 6, 2021
fd77e47
Add a note about ServerVariable so user's can configure the server ro…
kbendick Dec 6, 2021
2e87e72
Begin organizing the routes in the API doc
kbendick Dec 6, 2021
e11605e
Change schema from boolean to existing IcebergResponseObject
kbendick Dec 6, 2021
e95b5ab
Move up /namespaces/{ns}/properties for now - will see what happens w…
kbendick Dec 6, 2021
efc6acf
Fix schemas and doc
kbendick Dec 6, 2021
1f35862
regenerate the docs
kbendick Dec 6, 2021
33129b8
Reorg more with more examples
kbendick Dec 6, 2021
540af23
Remove getTable for now
kbendick Dec 6, 2021
73cfb0d
Regenerate the markdown
kbendick Dec 6, 2021
db52c0a
Remove HEAD response bodies
kbendick Dec 6, 2021
1171e08
Add missing 404
kbendick Dec 6, 2021
716c421
Revert code to be 3 digit HTTP response code, as type gives us all th…
kbendick Dec 6, 2021
85ed09f
Missing quotes
kbendick Dec 6, 2021
5768a72
Merge setProperties and removeProperties into one route
kbendick Dec 6, 2021
d698f44
Formaty JSON examples better - will move off of concrete error types …
kbendick Dec 6, 2021
c7df485
Regenerate the docs
kbendick Dec 6, 2021
87a96ce
Add in 422 - Unprocessable Entity for Update Properties if it has bot…
kbendick Dec 6, 2021
a0084ec
Update examples... considering making namespace and identifier a sing…
kbendick Dec 6, 2021
7f36e31
Add DropTableResponse and begin to reformat the schemas
kbendick Dec 6, 2021
03db9b7
Make the server object configurable and introduce responses to reduce…
kbendick Dec 6, 2021
e67f9a8
Update description of config endpoint
kbendick Dec 7, 2021
a15ffba
Regenerate docs without metadata in standard error response
kbendick Dec 7, 2021
7a1dd9a
Change description
kbendick Dec 7, 2021
5292a1d
Add note on the Java class that we're keeping it for discuccsion only
kbendick Dec 9, 2021
c83a78f
Fix 200 description for namespaceExists from 200 to Namespace Exists …
kbendick Dec 13, 2021
a6bf1ea
Move the info.version to 0.0.1 as we've added /v1 back to the route d…
kbendick Dec 13, 2021
55db1ea
Fix missing space typo in list namespaces description
kbendick Dec 15, 2021
19ed45b
Make the operationId for set and remove proprties updateProperties to…
kbendick Dec 15, 2021
9771cd9
Regenerate the markdown
kbendick Dec 15, 2021
9bbea49
Remove test server URL and make 400 the minimum for the error code pr…
kbendick Dec 15, 2021
9c388ab
Remove refeence to exception class when defining error.type's descrip…
kbendick Dec 15, 2021
291fc3f
Fix typo in Configuration schema, but leave the schema as is as it's …
kbendick Dec 15, 2021
ea301eb
Add in one more required property while refactoring before opening ne…
kbendick Dec 15, 2021
17ed2df
Rename IcebergConfiguration to CatalogConfiguration and then change s…
kbendick Dec 15, 2021
9d1b3de
Fix query parameter example
kbendick Dec 15, 2021
0d998e5
Update one of the namespace path descriptions to indicate that null b…
kbendick Dec 15, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,121 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/

package org.apache.iceberg.rest.http;

import com.fasterxml.jackson.annotation.JsonCreator;
import com.fasterxml.jackson.annotation.JsonInclude;
import com.fasterxml.jackson.annotation.JsonProperty;

/**
*
* NOTE: This document isn't intended as part of the Spec, and was
* intended for showing something to somebody. As there is on-going
* discssion, I'm leaving it, but then I will likely remove it for now.
*
* All responses for the first version of the Iceberg REST API will be JSON objects, a top-level envelope object with two fields,
* `error` and `data`, which are themselves JSON objects.
*
* data: represents the JSON encoded response for a successful path or for a valid / expected failure. JSON representation of various response types
* error: a standardized obect on error, containing the error code, message, type, and additional metadata (an optional JSON object of metadata) as defined below.
*
* All responses for the REST catalog should be wrapped this way, vs using primitives. For example, for a ListTableResponse, listing tables under a namespace "accounting",
* we'd get a JSON object back like the following:
* (the field `error`, having a value of null, has not been serialized).
*
* { "data": { "identifiers": [ "accounting.tax", "accounting.currency_conversions"] } }
*
* NOTE: The direction now is to head to just using HTTP response codes and
* not internal codes, as we have `Type`. If anybody feels strongly about
* using internal codes, please bring that up.
*
* If the namesapce `accounting` didn't request, the response from that call would have a body like the following,
* where the `code` 40401 is a two-part identifier:
* - HTTP response code: 404
* - Two digit internal application defined error code for further detail: 01 - Namespace not found.
*
* { "data": {}, "error": { "message": "Failed to list tables. The Namespace 'accounting' does not exist", "type": "NamespaceNotFoundException", "code": 40401 }
*
* We could also embed the HTTP response code plainly by itself, without internally documented codes, as a separate field. I have found having a documented list of internal
* error codes to be very helpful previously, but am open to discussion on this.
Comment on lines +55 to +56
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could it be clearer if the HTTP error code were separate from the internal error code? This would allow free form internal code without having to be confined to 100 values under an HTTP code prefix, and at the same time maintain reference to the more generic HTTP set of codes. It would also make it easier to extract HTTP error codes for processing.

Copy link
Contributor Author

@kbendick kbendick Dec 9, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So since we have type, which can be used to enumerate all of the possible sub fields, we've moved in the direction of just using the 3-digit HTTP response code in the body.

This file isn't part of the spec at all, as mentioned below, so I forgot to update it. I will remove it eventually but leave it so we can discuss.

Do you have input on using a dedicated int code (perhaps not constrained to HTTP code space as you mentioned) instead of just using the standardized value in type (which is a string)?

If we used code, the lookup table in my experience brings you to type. But I have previously used the way you described, with numeric error codes that we define a lookup table for that are independent. Do you have feedback on moving to just type?

Copy link
Contributor

@dimas-b dimas-b Dec 9, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IMHO, type is preferable to numeric code, but it might be harder to define strictly and there's risk that type names deviate from actual exception class names. Plus fixing class names in the API spec might affect non-java implementations in a negative way and complicate java code refactoring too.

In Nessie, as an example, we have an enum of non-numeric error codes. The API returns enum names, but they can be mapped to java exceptions upon parsing. Python client has its own mapping too. cf. ErrorCode.java

Copy link
Contributor Author

@kbendick kbendick Dec 9, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok. Then we'll stick with type for now and use the http response codes for code.

For the moment, there's no hard requirement that they map to exception names. And this class isn't actually part of the spec, it was more to show an idea to somebody.

We don't have to use class names / exception names. For now, let's just say that they are a set of well-defined names (at least for any values that we might want the client to react to).

Though implementations could in theory use a different set of names and they might have benefit for logging purposes / monitoring purposes, but the client wouldn't be able react to any custom names (because it wouldn't know about them and how to react to them). We don't ideally want to restrict the spec so much that people can't make it suit their needs (for example, if having a richer set of type would help them with observability in the future).

We can cover what those values are called in a separate discussion later on (this is far from the only discussion / reviewable that will be had about this so I think it's safe to punt it down the road, with that knowledge in mind that that's the general goal for now).

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Excluding this class from the spec and using only OpenAPI yaml SGTM 👍

Re: type values, I personally tend to think that if the field itself is part of the spec, the spec would have to define how to interpret it. I am not sure keeping it "opaque" is going to be that much useful from the spec's perspective.

Is this spec intended to define just a common part of the protocol between a Catalog impl. on the Iceberg side and the HTTP server that offers the REST API? Are the client and server parts supposed to be interoperable or are they assumed to work knowing the exact nature and version of the other side?

Copy link
Contributor Author

@kbendick kbendick Dec 15, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re: type values, I personally tend to think that if the field itself is part of the spec, the spec would have to define how to interpret it. I am not sure keeping it "opaque" is going to be that much useful from the spec's perspective.

I agree. Since we're using code to just be the HTTP status response code, I would say that type should be a well defined set we can look up.

We might also choose to allow people to use others that will get ignored by the client but are logged or something.

But for now I would say let's plan for it to be a well defined set of values, but not necessarily attempt to enumerate all of them just yet since we don't have about half of the basic Catalog AP in here yet. It's currently mostly just around namespaces and some of the table related ones will be handled next.

Sound good?

*/
@JsonInclude(JsonInclude.Include.NON_NULL)
public class IcebergHttpResponse<T> {

private final T data;
private final Error error;

@JsonCreator
public IcebergHttpResponse(
@JsonProperty("data") T data,
@JsonProperty("error") Error error) {
this.data = data;
this.error = error;
}

public Error error() {
return error;
}

public T data() {
return data;
}

/**
* An error object embedded in every HTTP response.
*
* On error, this contains:
* - message: A short, human-readable description of the error.
* - type: Type of exception - more specifically a class name, e.g. NamespaceNotFoundException)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this have to be part of the spec? In other words are specific exception class names going to be part of the spec in addition and/or in parallel to "internal" error codes?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No this isn't part of the spec at all.

I uploaded this here to share it with somebody I was speaking to, but this file is not part of the spec whatsoever. Sorry about the confusion!

* - code: An (optional) application specific error code, to distinguish between causes
* of the same HTTP response code (eg possibly different types of Unauthorized exceptions).
*
* #################### Optional fields to consider ######################################
* - status: HTTP response code (optional).
* - traceId: Unique specific identifier for this error and request, for monitoring purposes.
* Presumably this would be an OpenTracing Span (optional).
* Will almost certainly add tracing headers as an optional follow-up.
* - metadata: Further map of optional metadata (such as further directions to users etc) (optional - unsure?).
* #######################################################################################
*
* Example:
* "error": {
* "message": "Authorization denied: Missing Bearer header",
* "type": "OAuthException",
* "code": 40102
* }
*/
public static class Error {

private final String message;
private final String type;
private final int code;

@JsonCreator
public Error(
@JsonProperty("message") String message,
@JsonProperty("type") String type,
@JsonProperty("code") int code) {
this.message = message;
this.type = type;
this.code = code;
}
}
}

2 changes: 2 additions & 0 deletions dev/.rat-excludes
Original file line number Diff line number Diff line change
Expand Up @@ -24,3 +24,5 @@ gradle/*
package-list
sitemap.xml
derby.log
rest_docs/*

Loading