[#8834] feat(catalogs): Add a table to store details information of Gravitino managed tables. by yuqi1129 · Pull Request #8847 · apache/gravitino

yuqi1129 · 2025-10-20T09:20:38Z

What changes were proposed in this pull request?

Add a new table to store table details for Gravitino-managed tables.

Why are the changes needed?

To support the managed catalog.

Fix: #8834

Does this PR introduce any user-facing change?

N/A.

How was this patch tested?

Test the process of create table locally.

yuqi1129 · 2025-10-20T09:21:34Z

@mchades
Please check if we need to modify the structure of schema_meta and table_meta.

yuqi1129 · 2025-10-20T09:24:49Z

The schema and catalog, we need to add locatation information at least, and whether we can store the information in properties.

mchades · 2025-10-20T10:22:24Z

+    `partition_info`  CLOB DEFAULT NULL COMMENT 'table partition info',
+    `index_info`      CLOB DEFAULT NULL COMMENT 'table index info',
+    `current_version` BIGINT(20) UNSIGNED COMMENT 'table current version',
+    `last_version`    BIGINT(20) UNSIGNED COMMENT 'table last version',


What's the usage of last_version

last_version marks the largest version that has been successfully committed to storage. In most cases, current_version equals last_version. However, if we want to roll back to a certain version, then we can change current_version to an old one. The following is an example:

current_version = 3
last_version = 3,

When we want to write a new version, then last_version++ and current_version = last_version(Now, they are both 4). All available versions are 1, 2, 3, and 4. We can change current_version to any value between 1 and 4 to use a specific version. After a series of operations, we can use last_version=4 to indicate that the next version is 5.

What is the difference between last_version here and last_version in the table_meta table?

They should be the same, let me check whether we can remove last_version as we have define it in table_meta.

yeah, I think it's unecessary here

"last_version" is quite misleading since started, actually it is the latest version. Besides, I agree with @mchades that this field is not so necessary.

jerryshao · 2025-10-22T06:55:15Z

+    `format`          VARCHAR(64) NOT NULL COMMENT 'table format, such as Lance, Iceberg and so on',
+    `location`        VARCHAR(512) NOT NULL COMMENT 'table storage location',
+    `external`        BOOLEAN NOT NULL DEFAULT FALSE COMMENT 'whether the table is external table',
+    `properties`      CLOB DEFAULT NULL COMMENT 'table properties',


Why do you use CLOB for this field?

CLOB is the for H2 and it is equivalent toTEXT type in MySQL

jerryshao · 2025-10-22T06:57:01Z

+    `format`          VARCHAR(64) NOT NULL COMMENT 'table format, such as Lance, Iceberg and so on',
+    `location`        VARCHAR(512) NOT NULL COMMENT 'table storage location',
+    `external`        BOOLEAN NOT NULL DEFAULT FALSE COMMENT 'whether the table is external table',


My feeling is that these fields can be property, what do you think? @mchades . IIRC, these fields are properties for Hive table, can you please check Iceberg and other tables? If so, I think we can follow the convention.

Yes, I would also suggest putting location in the table property.

Considering that there may be a need to filter the table according to the format later, the format can be stored as a separate field.

I will put location and external in properties and let format as a field.

Location is important. If we need to check the locations whether is used or duplicated.

jerryshao · 2025-10-22T07:00:20Z

+    `partitions`  MEDIUMTEXT DEFAULT NULL COMMENT 'table partition info',
+    `distribution` MEDIUMTEXT DEFAULT NULL COMMENT 'table distribution info',
+    `sort_orders` MEDIUMTEXT DEFAULT NULL COMMENT 'table sort order info',
+    `indexes_info`      MEDIUMTEXT DEFAULT NULL COMMENT 'table index info',


All these fields will be serialized into json string and write into the DB, right?

Yeah, I intend to do so.

2. Rename `current_version` to `version`.

mchades · 2025-10-22T12:22:11Z

+    `table_id`        BIGINT(20) UNSIGNED NOT NULL COMMENT 'table id',
+    `format`          VARCHAR(64) NOT NULL COMMENT 'table format, such as Lance, Iceberg and so on',
+    `properties`      CLOB DEFAULT NULL COMMENT 'table properties',
+    `partitions`  MEDIUMTEXT DEFAULT NULL COMMENT 'table partition info',


should use Partitioning?

…n of Gravitino managed tables. (apache#8847) ### What changes were proposed in this pull request? Add a new table to store table details for Gravitino-managed tables. ### Why are the changes needed? To support the managed catalog. Fix: apache#8834 ### Does this PR introduce _any_ user-facing change? N/A. ### How was this patch tested? Test the process of create table locally.

Add a table to store details information of Gravitino managed tables.

9461656

mchades reviewed Oct 20, 2025

View reviewed changes

Comment thread scripts/h2/schema-1.1.0-h2.sql Outdated

mchades reviewed Oct 20, 2025

View reviewed changes

fix

bc16f98

mchades reviewed Oct 20, 2025

View reviewed changes

Comment thread scripts/h2/schema-1.1.0-h2.sql

yuqi1129 added 5 commits October 20, 2025 22:13

fix

438f0e6

fix

4ad2b7e

fix

c0e40ec

fix

22d1a34

Merge branch 'branch-lance-namepspace-dev' into issue_8834

3c5f4e5

yuqi1129 closed this Oct 22, 2025

yuqi1129 reopened this Oct 22, 2025

jerryshao reviewed Oct 22, 2025

View reviewed changes

yuqi1129 added 3 commits October 22, 2025 15:26

1. Remove external and location field.

bd9d3ff

2. Rename `current_version` to `version`.

Merge remote-tracking branch 'me/issue_8834' into issue_8834

84819d4

fix

23d1a90

mchades reviewed Oct 22, 2025

View reviewed changes

Change field partitions to partitioning.

4dbcf9a

yuqi1129 mentioned this pull request Oct 22, 2025

[#8838] feat(catalogs): Support create/load/list table operation for lance table. #8879

Merged

mchades approved these changes Oct 23, 2025

View reviewed changes

mchades assigned yuqi1129 Oct 23, 2025

mchades merged commit 924acbb into apache:branch-lance-namepspace-dev Oct 23, 2025
28 checks passed

Conversation

yuqi1129 commented Oct 20, 2025

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

yuqi1129 commented Oct 20, 2025

Uh oh!

yuqi1129 commented Oct 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mchades Oct 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jerryshao Oct 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

yuqi1129 commented Oct 20, 2025 •

edited

Loading

mchades Oct 20, 2025 •

edited

Loading

jerryshao Oct 22, 2025 •

edited

Loading