(feat: persistence) Add schema-metrics-v1.sql for metrics tables (#3337) by obelix74 · Pull Request #3523 · apache/polaris

obelix74 · 2026-01-24T00:16:15Z

Add new schema version 4 with tables for storing scan and commit metrics reports as first-class entities.

New tables:

scan_metrics_report: Stores scan metrics with trace correlation
scan_metrics_report_roles: Junction table for principal roles
commit_metrics_report: Stores commit metrics with trace correlation
commit_metrics_report_roles: Junction table for principal roles

Key design decisions:

PRIMARY KEY (realm_id, report_id) for multi-tenancy
Junction tables with CASCADE DELETE for roles
Timestamp index for retention cleanup
JSONB metadata column for extensibility (Postgres), TEXT for H2

Checklist

🛡️ Don't disclose security issues! (contact security@apache.org)
🔗 Clearly explained why the changes are needed, or linked related issues: Fixes #
🧪 Added/updated tests with good coverage, or manually tested (and explained how)
💡 Added comments for complex logic
🧾 Updated CHANGELOG.md (if needed)
📚 Updated documentation in site/content/in-dev/unreleased (if needed)

persistence/relational-jdbc/src/main/resources/postgres/schema-v4.sql

dimas-b · 2026-01-26T22:58:35Z

@obelix74 @singhpk234 : WDYT about starting an RFC doc + dev thread on this? I believe a structured overview of this feature would be good to set the stage for PRs :) (apologies if I missed it)

singhpk234 · 2026-01-26T23:37:26Z

@dimas-b there is a dev thread already please ref : https://lists.apache.org/thread/c83jnkvlwc2k3swm65cmvl4t0mt7p799
thanks @obelix74 for the the writing this up !

obelix74 · 2026-01-27T00:23:05Z

@obelix74 @singhpk234 : WDYT about starting an RFC doc + dev thread on this? I believe a structured overview of this feature would be good to set the stage for PRs :) (apologies if I missed it)

I am trying to solve two sets of asks from my product folks with this.

Metrics - what tables were accessed by a client principal
Auditing - which user accessed what data and why

From the metrics perspective, today, with 1.3.0, I want to be able to report on table metrics based on:

Track table scan operations:

by table
by snapshot
by time range
by realm
by user principal
by engine

For commit report queries:

by operation type
data growth
file churn
storage analysis

Also many operational dashboards, and filtering by user, realm, engine name, version etc.

I have not thought about roles in this flow at all, perhaps it will be useful. @singhpk234 recommended adding roles and I added them. I normalized the roles tables from a RDBMS perspective, but I didn't realize there are other similar fields stored as JSON already.

persistence/relational-jdbc/src/main/resources/h2/schema-v4.sql

dimas-b · 2026-02-04T18:24:56Z

@obelix74 : please rebase to fix CI

dimas-b · 2026-02-04T18:25:44Z

Let's hold final review until #3616 is resolved... Intermediate comments are welcome, of course :)

persistence/relational-jdbc/src/main/resources/postgres/schema-metrics-v1.sql

...rc/main/java/org/apache/polaris/persistence/relational/jdbc/JdbcMetaStoreManagerFactory.java

dimas-b

LGTM. Let's give this PR a couple more days in review for other people to comment if they want.

persistence/relational-jdbc/src/main/resources/postgres/schema-v4.sql

...rc/main/java/org/apache/polaris/persistence/relational/jdbc/JdbcMetaStoreManagerFactory.java

...lational-jdbc/src/main/java/org/apache/polaris/persistence/relational/jdbc/DatabaseType.java

persistence/relational-jdbc/src/main/resources/postgres/schema-metrics-v1.sql

persistence/relational-jdbc/src/main/resources/h2/schema-metrics-v1.sql

runtime/admin/src/main/java/org/apache/polaris/admintool/BootstrapCommand.java

persistence/relational-jdbc/src/main/resources/h2/schema-metrics-v1.sql

dimas-b

LGTM 👍

@singhpk234 : WDYT?

persistence/relational-jdbc/src/main/resources/postgres/schema-metrics-v1.sql

polaris-core/src/main/java/org/apache/polaris/core/persistence/MetaStoreManagerFactory.java

.../org/apache/polaris/persistence/relational/jdbc/RelationalJdbcProductionReadinessChecks.java

…tation Add checkMetricsPersistenceBootstrapped() to RelationalJdbcProductionReadinessChecks that properly verifies the MetricsPersistence implementation as requested in PR apache#3523. The check verifies: 1. The MetricsPersistence implementation is JdbcMetricsPersistence (not NoOpMetricsPersistence) 2. The JdbcMetricsPersistence.supportsMetricsPersistence() returns true (schema >= v4) This addresses the PR comment from dimas-b suggesting we should test the actual Metrics Persistence implementation rather than just checking configuration flags.

runtime/common/build.gradle.kts

This commit adds database schema support for metrics persistence tables as part of the JDBC persistence backend. Key changes: - schema-metrics-v1.sql for PostgreSQL and H2 with metrics tables - DatabaseType: Add metrics schema resource path support - JdbcBasePersistenceImpl: Add metrics schema bootstrap capability - JdbcBootstrapUtils: Support for metrics schema initialization - QueryGenerator: Add metrics-related query generation - SchemaOptions: Add metrics schema option for bootstrap command - BootstrapCommand: Support --metrics flag for schema bootstrap Testing: - MetricsPersistenceBootstrapValidationTest: Validates schema bootstrap - JdbcBootstrapUtilsTest: Tests for bootstrap utilities - QueryGeneratorTest: Tests for metrics query generation

github-project-automation bot added this to Basic Kanban Board Jan 24, 2026

github-project-automation bot moved this to PRs In Progress in Basic Kanban Board Jan 24, 2026

obelix74 mentioned this pull request Jan 24, 2026

feat(persistence): Add JDBC persistence layer for Iceberg metrics reporting (#3337) #3385

Open

6 tasks

singhpk234 reviewed Jan 24, 2026

View reviewed changes

persistence/relational-jdbc/src/main/resources/postgres/schema-v4.sql Outdated Show resolved Hide resolved

persistence/relational-jdbc/src/main/resources/postgres/schema-v4.sql Show resolved Hide resolved

singhpk234 requested a review from dimas-b January 24, 2026 00:23

dimas-b reviewed Jan 26, 2026

View reviewed changes

persistence/relational-jdbc/src/main/resources/postgres/schema-v4.sql Outdated Show resolved Hide resolved

obelix74 requested review from dimas-b and singhpk234 January 27, 2026 00:35

singhpk234 reviewed Jan 27, 2026

View reviewed changes

persistence/relational-jdbc/src/main/resources/h2/schema-v4.sql Show resolved Hide resolved

obelix74 force-pushed the feat-3337-schema-v4 branch 3 times, most recently from 7771396 to 65e8cb4 Compare February 5, 2026 23:44

obelix74 requested a review from singhpk234 February 5, 2026 23:50

obelix74 commented Feb 5, 2026

View reviewed changes

persistence/relational-jdbc/src/main/resources/postgres/schema-metrics-v1.sql Outdated Show resolved Hide resolved

obelix74 commented Feb 5, 2026

View reviewed changes

...rc/main/java/org/apache/polaris/persistence/relational/jdbc/JdbcMetaStoreManagerFactory.java Show resolved Hide resolved

dimas-b previously approved these changes Feb 6, 2026

View reviewed changes

persistence/relational-jdbc/src/main/resources/postgres/schema-v4.sql Show resolved Hide resolved

...rc/main/java/org/apache/polaris/persistence/relational/jdbc/JdbcMetaStoreManagerFactory.java Show resolved Hide resolved

github-project-automation bot moved this from PRs In Progress to Ready to merge in Basic Kanban Board Feb 6, 2026

obelix74 dismissed dimas-b’s stale review via 3eb9912 February 6, 2026 03:35

obelix74 commented Feb 6, 2026

View reviewed changes

...lational-jdbc/src/main/java/org/apache/polaris/persistence/relational/jdbc/DatabaseType.java Show resolved Hide resolved

obelix74 commented Feb 6, 2026

View reviewed changes

persistence/relational-jdbc/src/main/resources/postgres/schema-metrics-v1.sql Outdated Show resolved Hide resolved

singhpk234 reviewed Feb 6, 2026

View reviewed changes

persistence/relational-jdbc/src/main/resources/h2/schema-metrics-v1.sql Outdated Show resolved Hide resolved

runtime/admin/src/main/java/org/apache/polaris/admintool/BootstrapCommand.java Show resolved Hide resolved

dimas-b changed the title ~~(feat: persistence) Add schema-v4.sql for metrics tables (#3337)~~ (feat: persistence) Add schema-metrics-v1.sql for metrics tables (#3337) Feb 6, 2026

obelix74 commented Feb 6, 2026

View reviewed changes

persistence/relational-jdbc/src/main/resources/h2/schema-metrics-v1.sql Outdated Show resolved Hide resolved

dimas-b previously approved these changes Feb 6, 2026

View reviewed changes

persistence/relational-jdbc/src/main/resources/postgres/schema-metrics-v1.sql Outdated Show resolved Hide resolved

obelix74 dismissed dimas-b’s stale review via 7d382ee February 6, 2026 20:13

dimas-b previously approved these changes Feb 6, 2026

View reviewed changes

dimas-b requested a review from singhpk234 February 9, 2026 18:29

obelix74 dismissed dimas-b’s stale review via 1a3d7fb February 12, 2026 18:37

dimas-b reviewed Feb 12, 2026

View reviewed changes

polaris-core/src/main/java/org/apache/polaris/core/persistence/MetaStoreManagerFactory.java Outdated Show resolved Hide resolved

dimas-b reviewed Feb 12, 2026

View reviewed changes

.../org/apache/polaris/persistence/relational/jdbc/RelationalJdbcProductionReadinessChecks.java Outdated Show resolved Hide resolved

obelix74 requested a review from dimas-b February 12, 2026 20:36

dimas-b reviewed Feb 12, 2026

View reviewed changes

runtime/common/build.gradle.kts Outdated Show resolved Hide resolved

obelix74 force-pushed the feat-3337-schema-v4 branch from df4d3e7 to ff9d552 Compare February 12, 2026 21:48

dimas-b approved these changes Feb 12, 2026

View reviewed changes

obelix74 force-pushed the feat-3337-schema-v4 branch from 3e553a3 to 64dca5c Compare February 14, 2026 15:11

dimas-b merged commit 4ffb98a into apache:main Feb 17, 2026
15 checks passed

github-project-automation bot moved this from Ready to merge to Done in Basic Kanban Board Feb 17, 2026

Conversation

obelix74 commented Jan 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Checklist

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dimas-b commented Jan 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

singhpk234 commented Jan 26, 2026

Uh oh!

obelix74 commented Jan 27, 2026

Uh oh!

Uh oh!

dimas-b commented Feb 4, 2026

Uh oh!

dimas-b commented Feb 4, 2026

Uh oh!

Uh oh!

Uh oh!

dimas-b left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dimas-b left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

obelix74 commented Jan 24, 2026 •

edited

Loading

dimas-b commented Jan 26, 2026 •

edited

Loading