-
Notifications
You must be signed in to change notification settings - Fork 1.2k
[core][rest] support schema validation and infer for external paimon table #6501
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Is my understanding correct? There will be support for two ways to create paimon external tables:
|
Totally correct. Should be similar with HiveCatalog |
JingsongLi
approved these changes
Nov 3, 2025
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
+1
gmdfalk
added a commit
to gmdfalk/paimon
that referenced
this pull request
Nov 5, 2025
* master: (162 commits) [Python] Rename to BATCH_COMMIT_IDENTIFIER in snapshot.py [Python] Suppport multi prepare commit in the same TableWrite (apache#6526) [spark] Fix drop temporary view (apache#6529) [core] skip validate main branch before orphan files cleaning (apache#6524) [core][spark] Introduce upper transform (apache#6521) [Python] Keep the variable names of Identifier consistent with Java (apache#6520) [core] Remove hash lookup to simplify interface (apache#6519) [core][format] Format Table plan partitions should ignore hidden & illegal dirs (apache#6522) [hotfix] Print partition spec and type when error in InternalRowPartitionComputer [hotfix] Add more informat to check partition spec in InternalRowPartitionComputer [hotfix] Use deleteDirectoryQuietly in TempFileCommitter.clean [core] format table: support write file in _temporary at first (apache#6510) [core] Support non null column with write type (apache#6513) [core][fix] Blob with rolling file failed (apache#6518) [core][rest] Support schema validation and infer for external paimon table (apache#6501) [hotfix] Correct visitors for TransformPredicate [hotfix] Rename to copy from withNewInputs in TransformPredicate [core][spark] Support push down transform predicate (apache#6506) [spark] Implement SupportsReportStatistics for PaimonFormatTableBaseScan (apache#6515) [docs] add docs for auto-clustering of historical partitions (apache#6516) ...
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
[core][rest] Add schema validation and inference for REST catalog external tables
Purpose
Currently, REST catalog supported external tables but did not validate schema consistency between the filesystem and server-side schema. External tables always required explicit schema definition even when schema already existed in the filesystem.
This PR
Enhances REST catalog external table creation with schema inference and validation:
in the location
Examples:
CREATE TABLE t2 (id INT, name STRING) USING paimon LOCATION 'path' (explicit schema)CREATE TABLE t2 USING paimon LOCATION 'path' (schema inference from filesystem, newly supported)other fix about external table
sys.tablesto avoid failure caused by external paimon tableTests
PaimonExternalTableTest
RESTCatalogTest.testCreateExternalTableWithSchemaInference
RESTCatalogTest.testReadSystemTablesWithExternalTable
API and Format
Documentation