[FEATURE] Support paimon rest server

[neuyilan]

Describe the feature

Since paimon implements the rest catalog[1], do we plan to support paimon's rest server like iceberg?

[1] https://paimon.apache.org/docs/1.2/concepts/rest/overview/

Motivation

No response

Describe the solution

No response

Additional context

No response

[FANNG1]

Great idea, call for the voluntary user to implement the Paimon REST catalog, anyone interested could refer to RESTService for the Iceberg REST server.

[lsyulong]

@FANNG1 Can you assign this task to me? I want to study it，tks

[FANNG1]

Sure, could you propose a design doc or draft PR first? Then we could have some discussion on it.

[lsyulong]

OK

[GuoZhaoY]

Hello, may I ask if this feature has started development yet? We also need to develop this feature now

[lsyulong]

Sure, could you propose a design doc or draft PR first? Then we could have some discussion on it.

Hello, I am referring to the design document written by the open API of the official website of Paimon and the iceberg rest server implemented by Gravitino. Because there is a problem with my Google account at present, I cannot use Google documents, so I will use QQ documents instead. If there are any inappropriate designs in the documents, please help me take a look, and we can discuss it together. thank you very much! https://docs.qq.com/doc/DS3dKdFRib25FTEtk?no_promotion=1 cc @FANNG1 @neuyilan @GuoZhaoY

[neuyilan]

Hi @lsyulong , thanks for propose the degisn.
I'm wondering if we can abstract the REST server interface, so that Paimon, Iceberg, or other engines like Lance can implement this interface. At the same time, we can perform auditing, metadata management, and other tasks within the interface.
However, currently, the REST specifications of various engines are different, so it may not be easy to achieve this.

[lsyulong]

Hi @lsyulong , thanks for propose the degisn. I'm wondering if we can abstract the REST server interface, so that Paimon, Iceberg, or other engines like Lance can implement this interface. At the same time, we can perform auditing, metadata management, and other tasks within the interface. However, currently, the REST specifications of various engines are different, so it may not be easy to achieve this.

@neuyilan teacher,Thank you for proposing the abstraction of REST server interfaces! This is indeed an architectural question worth deep discussion.

I think there are two feasible approaches:

Approach 1: Abstract Unified Interface

Advantages: Code reuse, unified management, good maintainability
Challenges: May require refactoring the existing Iceberg REST server, designing an abstraction layer for REST servers
Approach 2: Independent REST Server Implementation

Advantages: Simple implementation, high flexibility, risk isolation
Challenges: Code duplication, maintenance costs
Considering Gravitino's current development stage and community resources, I suggest we start with Approach 2 for the following reasons:

Incremental Development: We can get the Paimon REST server up and running quickly to gain experience
Risk Reduction: Avoid refactoring that could impact the existing stable Iceberg REST server - would we need additional testing for the current Iceberg REST server?
Requirements Validation: Implement multiple REST servers first, then abstract a better, more flexible abstraction layer that fits Gravitino's architecture, allowing us to better validate and test based on actual experience
Of course, these are just my personal thoughts. If you have better ideas or designs for the abstract unified interface, we can discuss them together - I'd love to learn from you!

[GuoZhaoY]

Hi @lsyulong , thanks for propose the degisn. I'm wondering if we can abstract the REST server interface, so that Paimon, Iceberg, or other engines like Lance can implement this interface. At the same time, we can perform auditing, metadata management, and other tasks within the interface. However, currently, the REST specifications of various engines are different, so it may not be easy to achieve this.

@neuyilan teacher,Thank you for proposing the abstraction of REST server interfaces! This is indeed an architectural question worth deep discussion.

I think there are two feasible approaches:

Approach 1: Abstract Unified Interface

Advantages: Code reuse, unified management, good maintainability Challenges: May require refactoring the existing Iceberg REST server, designing an abstraction layer for REST servers Approach 2: Independent REST Server Implementation

Advantages: Simple implementation, high flexibility, risk isolation Challenges: Code duplication, maintenance costs Considering Gravitino's current development stage and community resources, I suggest we start with Approach 2 for the following reasons:

Incremental Development: We can get the Paimon REST server up and running quickly to gain experience Risk Reduction: Avoid refactoring that could impact the existing stable Iceberg REST server - would we need additional testing for the current Iceberg REST server? Requirements Validation: Implement multiple REST servers first, then abstract a better, more flexible abstraction layer that fits Gravitino's architecture, allowing us to better validate and test based on actual experience Of course, these are just my personal thoughts. If you have better ideas or designs for the abstract unified interface, we can discuss them together - I'd love to learn from you!

Whether using Plan 1 or Plan 2, if manpower is needed, I am more than happy to participate in the improvement of this feature

[neuyilan]

Hi, @lsyulong
Thank you for the detailed explanation. I think we can start with Approach 2. As you said, we should first ensure the functionality is available, and then, based on the accumulated experience, encapsulate and abstract the code to avoid premature optimization.

[FANNG1]

Sure, could you propose a design doc or draft PR first? Then we could have some discussion on it.

Hello, I am referring to the design document written by the open API of the official website of Paimon and the iceberg rest server implemented by Gravitino. Because there is a problem with my Google account at present, I cannot use Google documents, so I will use QQ documents instead. If there are any inappropriate designs in the documents, please help me take a look, and we can discuss it together. thank you very much! https://docs.qq.com/doc/DS3dKdFRib25FTEtk?no_promotion=1 cc @FANNG1 @neuyilan @GuoZhaoY

The design LGTM, I would suggest we could support the basic code skeleton and one single Paimon interface first, which contains the basic framework, simple UT and IT if required. For the version of Paimon I prefer to upgrade it first not do some compatibility works, WDYT?

[lizc9]

Sure, could you propose a design doc or draft PR first? Then we could have some discussion on it.

Hello, I am referring to the design document written by the open API of the official website of Paimon and the iceberg rest server implemented by Gravitino. Because there is a problem with my Google account at present, I cannot use Google documents, so I will use QQ documents instead. If there are any inappropriate designs in the documents, please help me take a look, and we can discuss it together. thank you very much! https://docs.qq.com/doc/DS3dKdFRib25FTEtk?no_promotion=1 cc @FANNG1 @neuyilan @GuoZhaoY

Hi, @lsyulong , Does this include lock design? For example, when modifying table metadata or committing snapshots, locks need to be added. Or does the underlying layer still use the Hive Catalog lock?

[lsyulong]

Thank you @lizc9 for raising the locking mechanism question! This is indeed a key consideration for the Paimon REST Server implementation.

I see that Gravitino already has a TreeLock mechanism, so we can implement based on this rather than relying on the underlying Hive Catalog locks. Here are the reasons:

Architectural Consistency
Gravitino already has a complete hierarchical locking system - TreeLock.

Existing Implementation Reference
TreeLock is already widely used in Gravitino's core Dispatcher layer.

Existing Iceberg REST Server Locking Mechanism Reference

The Iceberg REST Server itself doesn't directly implement locks - it indirectly benefits from TreeLock protection by calling Gravitino's core Dispatcher layer.

So, can the Paimon REST Server leverage previous experience during implementation?

What do you think of this approach? Welcome further discussion!

[lizc9]

@lsyulong LGTM