perf(refactor): optimizing JdbcBasePersistenceImpl.listEntities

[binarycat0]

org.apache.polaris.persistence.relational.jdbc.JdbcBasePersistenceImpl.listEntities

• Reduced Column Selection: Only 6 columns instead of 16
• Eliminated Object Creation Overhead: Direct conversion to EntityNameLookupRecord without intermediate PolarisBaseEntity
• minor refactoring

Issue: #2352

---

Query EXPLAIN plan shown in the comment:
#2465 (comment)

Follow-ups

• Rethink design of persistence.relational.jdbc.models.Converter #2487
• After dev ML discussion need to decide do we need to add an extra Persistance DB Index or not #2488

[singhpk234]

Thank you for this change @binarycat0 do you have some perf numbers on how much this benefits compared to existing

[binarycat0]

@singhpk234 Hello! Good question. I did not do any performance measurements, but potencially it should help reduce ram and cpu consumption. Could you share a link/clue how I can perform perf testing and compare previous and the current solution?

[dimas-b]

LGTM overall 👍

Re: perf. testing, from my side, I'd ask for a test that progressively adds entities and shows various API response times as a function of time (or equivalently of the number of entities).

The hardware does not have to be very fast. The point is to compare old code with new code on the same hardware.

[binarycat0]

@singhpk234 My colleagues and I conducted several different experiments and we didn't notice significant improvement on the API side. This is most likely primarily dependent on EntityCache, which greatly improves performance in the current implementation.
Therefore, I can say that the current changes don't significantly improve API performance, but the additional DB INDEX improves SQL experience and reduces query execution time.
Also, in my tests I didn't notice a difference when measuring EXPLAIN PLAN for (select *) versus (select colA, colB...)

[binarycat0]

Test ENV

OS: 24.5.0 Darwin Kernel Version 24.5.0: Tue Apr 22 19:53:27 PDT 2025; root:xnu-11417.121.6~2/RELEASE_ARM64_T6041 arm64
Storage: Docker image postgres:17
Rows in total = 899040.
Expected rows result = 200005.

Results

Main build version

EXPLAIN ANALYZE
(SELECT id, catalog_id, parent_id, type_code, name, entity_version, sub_type_code, create_timestamp, drop_timestamp, purge_timestamp, to_purge_timestamp, last_update_timestamp, properties, internal_properties, grant_records_version, location_without_scheme
 FROM POLARIS_SCHEMA.ENTITIES
 WHERE catalog_id = 6911759911282783572 AND sub_type_code = 0 AND realm_id = 'POLARIS' AND parent_id = 6911759911282783572 AND type_code = 6);

Bitmap Heap Scan on entities  (cost=3960.68..22260.05 rows=44861 width=140) (actual time=16.287..43.851 rows=200005 loops=1)
  Recheck Cond: ((realm_id = 'POLARIS'::text) AND (catalog_id = '6911759911282783572'::bigint) AND (parent_id = '6911759911282783572'::bigint) AND (type_code = 6))
  Filter: (sub_type_code = 0)
  Heap Blocks: exact=3848
  ->  Bitmap Index Scan on constraint_name  (cost=0.00..3949.47 rows=44861 width=0) (actual time=15.792..15.792 rows=200005 loops=1)
        Index Cond: ((realm_id = 'POLARIS'::text) AND (catalog_id = '6911759911282783572'::bigint) AND (parent_id = '6911759911282783572'::bigint) AND (type_code = 6))
Planning Time: 0.239 ms
Execution Time: 49.253 ms

New build version

CREATE INDEX IF NOT EXISTS idx_entities_lookup
    ON entities USING btree (realm_id, catalog_id, parent_id, type_code, sub_type_code)
    INCLUDE (id, name);

EXPLAIN ANALYZE
(SELECT id, catalog_id, parent_id, type_code, name, sub_type_code 
 FROM POLARIS_SCHEMA.ENTITIES
 WHERE catalog_id = 6911759911282783572 AND sub_type_code = 0 AND realm_id = 'POLARIS' AND parent_id = 6911759911282783572 AND type_code = 6);

Index Only Scan using idx_entities_lookup on entities  (cost=0.42..2975.89 rows=42744 width=50) (actual time=0.030..29.235 rows=200005 loops=1)
  Index Cond: ((realm_id = 'POLARIS'::text) AND (catalog_id = '6911759911282783572'::bigint) AND (parent_id = '6911759911282783572'::bigint) AND (type_code = 6) AND (sub_type_code = 0))
  Heap Fetches: 0
Planning Time: 0.089 ms
Execution Time: 35.317 ms

Conclusion

Using Index Only Scan instead of Bitmap Heap Scan helps:

• reduce query execution time up to 28%;
• reducing query cost up to 87%;

cc: @dimas-b @singhpk234

[XN137]

thanks for working on this, left some minor comments but looks good overall now.

imo since this addresses an existing TODO it shouldnt be strictly required to demonstrate elaborate perf numbers.

what we would need though is a comparison of the postgres explain plan before and after these changes for operations that use PolarisMetaStoreManager.listEntities i.e. listing tables below a parent namespace.
previously is should be an inefficient "table scan" where-as now it is an index-only scan ...
whether that shows an immediately measurable performance improvements depends on a lot of details about benchmark design and postgres internals.

[dimas-b]

Thanks for the perf. update @binarycat0 ! However, the execution times are too small and the sample size is small too (I'm afraid) so the perf. numbers are not decisive, IMHO 😅

At the same time the new index will have some storage and maintenance overhead.

Let's open a dev ML discussion about this and see what people who use Polaris with Postgres think.

The java code changes LGTM 👍

[dimas-b]

These java code change look like a nice improvement to me.

As for the new index (that was in the earlier revisions of this PR). Let's decide on that based on dev ML discussion.

[dimas-b]

@binarycat0 : please remove Database Index Optimization: Covering index enables index-only scans from the description as the new index is no longer in this PR 😅