Skip to content

Python: First version of the rest catalog #5287

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
rdblue merged 25 commits into from
Jul 30, 2022
Merged

Python: First version of the rest catalog #5287

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
01842e3
Python: Add REST Catalog
Fokko Jun 20, 2022
5a73bd4
Merge branch 'master' of https://github.com/apache/iceberg into fd-re…
Fokko Jul 5, 2022
776a9ea
First few are working
Fokko Jul 5, 2022
ba29add
Merge branch 'master' of https://github.com/apache/iceberg into fd-re…
Fokko Jul 12, 2022
d55fc81
.
Fokko Jul 13, 2022
7568a3b
Add load table
Fokko Jul 13, 2022
671cc07
Merge branch 'master' of https://github.com/apache/iceberg into fd-re…
Fokko Jul 13, 2022
f1a986e
Python: First version of the REST catalog
Fokko Jul 14, 2022
a51f853
Process comments
Fokko Jul 17, 2022
931c3e3
Remove the rough edges
Fokko Jul 17, 2022
736df3f
Merge branch 'master' of https://github.com/apache/iceberg into fd-re…
Fokko Jul 17, 2022
49acfb0
Revert unrelated changes
Fokko Jul 17, 2022
c6dc7cd
Merge branch 'master' of https://github.com/apache/iceberg into fd-re…
Fokko Jul 21, 2022
4f373f2
Add the missing alias
Fokko Jul 21, 2022
1a7faa9
pre-commit
Fokko Jul 21, 2022
9ddc4e7
Comments
Fokko Jul 22, 2022
34fe404
Merge branch 'master' of https://github.com/apache/iceberg into fd-re…
Fokko Jul 22, 2022
3f538c6
Merge branch 'master' of https://github.com/apache/iceberg into fd-re…
Fokko Jul 22, 2022
b184a48
Merge branch 'master' of https://github.com/apache/iceberg into fd-re…
Fokko Jul 25, 2022
37763bf
Add version to the headers
Fokko Jul 25, 2022
c27ef21
Merge branch 'master' of https://github.com/apache/iceberg into fd-re…
Fokko Jul 26, 2022
5283691
Bit more cleanup
Fokko Jul 26, 2022
62d823b
Just return the table
Fokko Jul 26, 2022
b48722f
Allow both client_{id,secret} and token
Fokko Jul 27, 2022
dc79b27
Comments
Fokko Jul 28, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/open-api.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,4 +45,4 @@ jobs:
python-version: 3.9
- name: Install
working-directory: ./open-api
run: pip install openapi-spec-validator && openapi-spec-validator rest-catalog-open-api.yaml
run: pip install openapi-spec-validator && openapi-spec-validator rest-catalog-open-api.yaml
2 changes: 1 addition & 1 deletion python/.pre-commit-config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ repos:
rev: v0.971
hooks:
- id: mypy
args: [--config=python/pyproject.toml]
args: [ --install-types, --non-interactive, --config=python/pyproject.toml]
- repo: https://github.com/hadialqattan/pycln
rev: v2.0.4
hooks:
Expand Down
256 changes: 103 additions & 153 deletions python/poetry.lock
View file
Open in desktop

Large diffs are not rendered by default.

19 changes: 15 additions & 4 deletions python/pyiceberg/catalog/base.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,11 +18,20 @@
from __future__ import annotations

from abc import ABC, abstractmethod
from dataclasses import dataclass

from pyiceberg.catalog import Identifier, Properties
from pyiceberg.schema import Schema
from pyiceberg.table.base import Table
from pyiceberg.table.partitioning import PartitionSpec
from pyiceberg.table.partitioning import UNPARTITIONED_PARTITION_SPEC, PartitionSpec
from pyiceberg.table.sorting import UNSORTED_SORT_ORDER, SortOrder


@dataclass
class PropertiesUpdateSummary:
removed: list[str]
updated: list[str]
missing: list[str]


class Catalog(ABC):
Expand Down Expand Up @@ -57,7 +66,8 @@ def create_table(
identifier: str | Identifier,
schema: Schema,
location: str | None = None,
partition_spec: PartitionSpec | None = None,
partition_spec: PartitionSpec = UNPARTITIONED_PARTITION_SPEC,
sort_order: SortOrder = UNSORTED_SORT_ORDER,
properties: Properties | None = None,
Copy link
Contributor

@rdblue rdblue Jul 15, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we similarly default properties to {}?

Copy link
Contributor Author

@Fokko Fokko Jul 17, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would like that, but that's actually a code smell in Python (or a quirk in the language). The default {} will be a reference to a single object, if you mutate that one, the next time the default value is being assigned, it will give the reference to the same object. More info here: https://florimond.dev/en/posts/2018/08/python-mutable-defaults-are-the-source-of-all-evil/ So the recommended way is just to set it to null, and then do properties or {} in the code. This {} will then always initialize an empty dict.

Copy link
Contributor

@rdblue rdblue Jul 26, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's no immutable dict?

Copy link
Contributor Author

@Fokko Fokko Jul 26, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not in the standard lib, there is a package, or we can implement it ourselves.

Copy link
Contributor

@rdblue rdblue Jul 26, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe later, but this is fine for now.

) -> Table:
"""Create a table
Expand All @@ -66,7 +76,8 @@ def create_table(
identifier: Table identifier.
schema: Table's schema.
location: Location for the table. Optional Argument.
partition_spec: PartitionSpec for the table. Optional Argument.
partition_spec: PartitionSpec for the table.
sort_order: SortOrder for the table.
properties: Table properties that can be a string based dictionary. Optional Argument.

Returns:
Expand Down Expand Up @@ -195,7 +206,7 @@ def load_namespace_properties(self, namespace: str | Identifier) -> Properties:
@abstractmethod
def update_namespace_properties(
self, namespace: str | Identifier, removals: set[str] | None = None, updates: Properties | None = None
) -> None:
) -> PropertiesUpdateSummary:
"""Removes provided property keys and updates properties for a namespace.

Args:
Expand Down
Loading