[#9021] feat(lance-rest): Support create & list index for LRS

[yuqi1129]

What changes were proposed in this pull request?

Add create and list index APIs in the Lance rest server.

Why are the changes needed?

It's a feature that benefits users.

Fix: #9021

Does this PR introduce any user-facing change?

N/A.

How was this patch tested?

UTs and ITs

[Copilot]

Pull request overview

This PR adds create and list index APIs to the Lance REST server, enabling users to create various types of indexes (BTREE, BITMAP, vector indexes like IVF_FLAT, IVF_PQ, etc.) and list existing indexes on Lance tables.

Key Changes:

• Added createTableIndex, listTableIndices, and describeTableIndexStats REST endpoints and implementations
• Extended the Index and TableChange.AddIndex APIs to support index properties
• Implemented Lance-specific index creation logic with support for scalar and vector index types
• Refactored table operation tests into a dedicated TestLanceTableOperations class

Reviewed changes

Copilot reviewed 24 out of 28 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
lance/lance-rest-server/src/main/java/org/apache/gravitino/lance/service/rest/LanceTableOperations.java	Added REST endpoints for creating and listing table indexes
lance/lance-common/src/main/java/org/apache/gravitino/lance/common/ops/LanceTableOperations.java	Extended interface with index operation methods
lance/lance-common/src/main/java/org/apache/gravitino/lance/common/ops/gravitino/GravitinoLanceTableOperations.java	Implemented index creation and listing logic using Gravitino catalog operations
catalogs/catalog-generic-lakehouse/src/main/java/org/apache/gravitino/catalog/lakehouse/lance/LanceCatalogOperations.java	Added Lance dataset index creation with support for various index types and parameters
api/src/main/java/org/apache/gravitino/rel/indexes/Index.java	Added properties support to Index interface and new index types (IVF_HNSW_PQ, FTS)
api/src/main/java/org/apache/gravitino/rel/indexes/Indexes.java	Extended factory methods to support index properties
api/src/main/java/org/apache/gravitino/rel/TableChange.java	Extended AddIndex change to include index properties
common/src/main/java/org/apache/gravitino/dto/rel/indexes/IndexDTO.java	Added properties field to index DTO
common/src/main/java/org/apache/gravitino/json/JsonUtils.java	Updated serialization/deserialization to handle index properties
lance/lance-rest-server/src/test/java/org/apache/gravitino/lance/service/rest/TestLanceTableOperations.java	Added comprehensive unit tests for table operations including index creation and listing
lance/lance-rest-server/src/test/java/org/apache/gravitino/lance/integration/test/LanceRESTServiceIT.java	Added integration tests for index creation with various index types
gradle/libs.versions.toml	Updated lance-namespace version from 0.0.19 to 0.0.20
lance/lance-rest-server/build.gradle.kts	Added dependency exclusions for commons-lang3 and guava
core/src/main/java/org/apache/gravitino/storage/relational/utils/POConverters.java	Fixed table property deserialization to handle null properties
bin/gravitino-lance-rest-server.sh.template	Added startup script for Lance REST server
build.gradle.kts	Added build tasks for Lance REST server distribution

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 25 out of 29 changed files in this pull request and generated 13 comments.

[Copilot]

Missing handling for IVF_HNSW_PQ index type. The enum is defined in the API but is not handled in either the getIndexType() method (line 328) or the generateIndexParams() method (line 345). This will cause an IllegalArgumentException when users try to create an IVF_HNSW_PQ index.

Suggested change

	default -> throw new IllegalArgumentException("Unsupported index type: " + indexType);
	}
	case IVF_HNSW_PQ -> builder.setVectorIndexParams(
	new VectorIndexParams.Builder(new IvfBuildParams.Builder().build())
	.setDistanceType(toLanceDistanceType(request.getMetricType()))
	.setHnswParams(new HnswBuildParams.Builder().build())
	.setPqParams(
	new PQBuildParams.Builder()
	.setNumSubVectors(1) // others use default value.
	.build())
	.build());
	default -> throw new IllegalArgumentException("Unsupported index type: " + indexType);

[Copilot]

Missing handling for LABEL_LIST index type. The enum is defined in the API (line 101) but is not handled in the getIndexType() or generateIndexParams() methods, which will cause failures when users try to create this type of index.

[yuqi1129]

It's not supported by the client side.

[Copilot]

Invalid switch statement syntax. The switch cases use arrow syntax (->) but are missing the arrow for the first case (line 346-347). The correct syntax should be either traditional case SCALAR: case BTREE: case INVERTED: case BITMAP: with breaks, or case SCALAR, BTREE, INVERTED, BITMAP -> with an arrow. The current mixed syntax will cause compilation errors.

[yuqi1129]

      case SCALAR, BTREE, INVERTED, BITMAP -> builder.setScalarIndexParams(
          ScalarIndexParams.create(indexType.name()));

The code is as above, so what's the problem.

[Copilot]

The test expects FTS index to fail with a specific error message about "No enum constant com.lancedb.lance.index.IndexType.FTS". However, the FTS enum has been added to Index.IndexType in the API changes. This test should be updated to either handle FTS properly if Lance now supports it, or the FTS enum should be removed from the API if it's not yet supported.

[yuqi1129]

This is thirty-party API and I can't remove it.

[yuqi1129]

listIndexes has been merged into the main by lance-format/lance#5447, and I'm pushing describe index PR to merge to main