[Feature] Improve Paimon Spec

[Xuanwo]

Search before asking

• I searched in the issues and found nothing similar.

Motivation

I'm working on paimon-rust now. I found paimon's specifications to be neither accurate nor detailed.

Take Schema as an example, the current spec only have:

The version of the schema file starts from 0 and currently retains all versions of the schema. There may be old files that rely on the old schema version, so its deletion should be done with caution.

Schema File is JSON, it includes:

fields: data field list, data field contains id, name, type, field id is used to support schema evolution.
partitionKeys: partition definition of the table, it cannot be modified.
primaryKeys: primary key definition of the table, it cannot be modified.
options: options of the table, including a lot of capabilities and optimizations.

I will have the following questions:

• What's the detailed json format of Schema?
• What's field?
• What's partition definition?
• What's primary key definition?
• How options is stored? What's it key/value type? Should it be ordered?

Solution

Please provide more detailed specifications for Paimon, possibly including pseudocode examples.

Anything else?

None

Are you willing to submit a PR?

• I'm willing to submit a PR!

[JingsongLi]

I will do this.