Skip to content

[#8891] feat(lance-rest-server): Add standalone and auxiliary mode support for Lance REST service#8903

Merged
jerryshao merged 5 commits intoapache:branch-lance-namepspace-devfrom
youngyjd:package-standalone-auxiliary-mode-lance
Oct 24, 2025
Merged

[#8891] feat(lance-rest-server): Add standalone and auxiliary mode support for Lance REST service#8903
jerryshao merged 5 commits intoapache:branch-lance-namepspace-devfrom
youngyjd:package-standalone-auxiliary-mode-lance

Conversation

@youngyjd
Copy link
Copy Markdown
Contributor

@youngyjd youngyjd commented Oct 23, 2025

What changes were proposed in this pull request?

  1. Created service discovery file (META-INF/services/org.apache.gravitino.auxiliary.GravitinoAuxiliaryService)
  2. Added unit tests (TestLanceConfig.java)

Why are the changes needed?

Fix: #8891

To enable flexible deployment of Lance REST service:

Standalone mode (independent JVM):

  • Lance needs to connect to Gravitino server via HTTP to access catalog/schema/table metadata
  • GRAVITINO_URI and GRAVITINO_METALAKE configs specify the connection parameters
  • Use case: Deploy Lance REST separately for isolation or scaling

Auxiliary mode (embedded in Gravitino server):

  • Lance runs in the same JVM as Gravitino, enabling direct memory access (zero overhead)
  • META-INF/services file enables automatic service discovery via Java ServiceLoader
  • Use case: Simplified deployment with shared resources and unified monitoring

Does this PR introduce any user-facing change?

Yes - New configuration properties

For Standalone Mode:

# Required: Gravitino server URI
gravitino.lance-rest.gravitino-uri=http://gravitino-server:8090

# Required: Metalake name
gravitino.lance-rest.gravitino-metalake=production

For Auxiliary Mode:

# Enable Lance REST as auxiliary service
gravitino.auxService.names=iceberg-rest,lance-rest

# Classpath for Lance REST JAR
gravitino.lance-rest.classpath=lance/lance-rest-server/build/libs

Deployment modes supported:
Standalone: Lance REST runs independently, connects to Gravitino via HTTP
Auxiliary: Lance REST runs embedded in Gravitino, uses direct memory access
No breaking changes - all changes are additive and backward compatible.

How was this patch tested?

  1. unit test
  2. manual test

Standalone mode test:

junda@java(junda.devpod-us-or) ~/gravitino/distribution/package/bin
 % ./gravitino.sh start
Gravitino Server is running[PID:3907790]
junda@java(junda.devpod-us-or) ~
 % curl http://localhost:8090/api/version
{"code":0,"version":{"version":"1.1.0-SNAPSHOT","compileDate":"23/10/2025 21:03:17","gitCommit":"35bd819975badea2493b4dbd9d0de68425229945"}}%   

junda@java(junda.devpod-us-or) ~/gravitino/distribution/package/bin
 % ./gravitino-lance-rest-server.sh start
GravitinoLanceRESTServer is running[PID:3930096]
junda@java(junda.devpod-us-or) ~
 % curl "http://localhost:9101/lance/v1/namespace/root/list"
{"id":"root","delimiter":"$","namespaces":["default"]}%   

Auxiliary mode test:

junda@java(junda.devpod-us-or) ~/gravitino/distribution/package
 % cd /home/user/gravitino && cat >> distribution/package/conf/gravitino.conf << 'EOF'

# ===== Lance REST Auxiliary Service =====
gravitino.auxService.names = iceberg-rest,lance-rest
gravitino.lance-rest.classpath = /home/user/gravitino/lance/lance-rest-server/build/libs
gravitino.lance-rest.catalog-name = lance_catalog
EOF

junda@java(junda.devpod-us-or) ~/gravitino/distribution/package
 % ./bin/gravitino.sh start          
Gravitino Server is running[PID:274396]

junda@java(junda.devpod-us-or) ~/gravitino/distribution/package
 % tail -50 logs/gravitino-server.log | grep -i lance | grep -i started
2025-10-23 21:34:41.495 INFO [main] [org.apache.gravitino.server.web.JettyServer.start(JettyServer.java:202)] - lance-rest web server started on host 0.0.0.0 port 9101.
2025-10-23 21:34:41.495 INFO [main] [org.apache.gravitino.lance.LanceRESTService.serviceStart(LanceRESTService.java:77)] - Lance REST service started

@youngyjd youngyjd changed the title update-for-standalone-mode-and-auxiliary-mode feat(lance): Add standalone and auxiliary mode support for Lance REST service Oct 23, 2025
@youngyjd youngyjd marked this pull request as ready for review October 23, 2025 20:34
@youngyjd youngyjd closed this Oct 23, 2025
@youngyjd youngyjd reopened this Oct 23, 2025
@mchades
Copy link
Copy Markdown
Contributor

mchades commented Oct 24, 2025

LGTM. Could you plz resolve the conflicts? thx

@youngyjd youngyjd force-pushed the package-standalone-auxiliary-mode-lance branch from e78757f to ca98524 Compare October 24, 2025 03:09
@mchades mchades changed the title feat(lance): Add standalone and auxiliary mode support for Lance REST service [#8891] feat(lance-rest-server): Add standalone and auxiliary mode support for Lance REST service Oct 24, 2025
Copy link
Copy Markdown
Contributor

@mchades mchades left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm.

@youngyjd youngyjd closed this Oct 24, 2025
@youngyjd youngyjd reopened this Oct 24, 2025
@jerryshao jerryshao merged commit f6d49d9 into apache:branch-lance-namepspace-dev Oct 24, 2025
49 of 50 checks passed
jerryshao pushed a commit to jerryshao/gravitino that referenced this pull request Nov 11, 2025
…ode support for Lance REST service (apache#8903)

### What changes were proposed in this pull request?

1. Created service discovery file
(META-INF/services/org.apache.gravitino.auxiliary.GravitinoAuxiliaryService)
2. Added unit tests (TestLanceConfig.java)

### Why are the changes needed?
Fix: apache#8891 

To enable flexible deployment of Lance REST service:

Standalone mode (independent JVM):
- Lance needs to connect to Gravitino server via HTTP to access
catalog/schema/table metadata
- GRAVITINO_URI and GRAVITINO_METALAKE configs specify the connection
parameters
- Use case: Deploy Lance REST separately for isolation or scaling

Auxiliary mode (embedded in Gravitino server):
- Lance runs in the same JVM as Gravitino, enabling direct memory access
(zero overhead)
- META-INF/services file enables automatic service discovery via Java
ServiceLoader
- Use case: Simplified deployment with shared resources and unified
monitoring


### Does this PR introduce _any_ user-facing change?

Yes - New configuration properties

For Standalone Mode:
```
# Required: Gravitino server URI
gravitino.lance-rest.gravitino-uri=http://gravitino-server:8090

# Required: Metalake name
gravitino.lance-rest.gravitino-metalake=production
```

For Auxiliary Mode:
```
# Enable Lance REST as auxiliary service
gravitino.auxService.names=iceberg-rest,lance-rest

# Classpath for Lance REST JAR
gravitino.lance-rest.classpath=lance/lance-rest-server/build/libs
```

Deployment modes supported:
Standalone: Lance REST runs independently, connects to Gravitino via
HTTP
Auxiliary: Lance REST runs embedded in Gravitino, uses direct memory
access
No breaking changes - all changes are additive and backward compatible.

### How was this patch tested?

1. unit test
2. manual test

**Standalone mode test:**
```
junda@java(junda.devpod-us-or) ~/gravitino/distribution/package/bin
 % ./gravitino.sh start
Gravitino Server is running[PID:3907790]
junda@java(junda.devpod-us-or) ~
 % curl http://localhost:8090/api/version
{"code":0,"version":{"version":"1.1.0-SNAPSHOT","compileDate":"23/10/2025 21:03:17","gitCommit":"35bd819975badea2493b4dbd9d0de68425229945"}}%   

junda@java(junda.devpod-us-or) ~/gravitino/distribution/package/bin
 % ./gravitino-lance-rest-server.sh start
GravitinoLanceRESTServer is running[PID:3930096]
junda@java(junda.devpod-us-or) ~
 % curl "http://localhost:9101/lance/v1/namespace/root/list"
{"id":"root","delimiter":"$","namespaces":["default"]}%   
```

**Auxiliary mode test:**
```
junda@java(junda.devpod-us-or) ~/gravitino/distribution/package
 % cd /home/user/gravitino && cat >> distribution/package/conf/gravitino.conf << 'EOF'

# ===== Lance REST Auxiliary Service =====
gravitino.auxService.names = iceberg-rest,lance-rest
gravitino.lance-rest.classpath = /home/user/gravitino/lance/lance-rest-server/build/libs
gravitino.lance-rest.catalog-name = lance_catalog
EOF

junda@java(junda.devpod-us-or) ~/gravitino/distribution/package
 % ./bin/gravitino.sh start          
Gravitino Server is running[PID:274396]

junda@java(junda.devpod-us-or) ~/gravitino/distribution/package
 % tail -50 logs/gravitino-server.log | grep -i lance | grep -i started
2025-10-23 21:34:41.495 INFO [main] [org.apache.gravitino.server.web.JettyServer.start(JettyServer.java:202)] - lance-rest web server started on host 0.0.0.0 port 9101.
2025-10-23 21:34:41.495 INFO [main] [org.apache.gravitino.lance.LanceRESTService.serviceStart(LanceRESTService.java:77)] - Lance REST service started
```
youngyjd added a commit to youngyjd/gravitino that referenced this pull request Nov 18, 2025
…ode support for Lance REST service (apache#8903)

### What changes were proposed in this pull request?

1. Created service discovery file
(META-INF/services/org.apache.gravitino.auxiliary.GravitinoAuxiliaryService)
2. Added unit tests (TestLanceConfig.java)

### Why are the changes needed?
Fix: apache#8891 

To enable flexible deployment of Lance REST service:

Standalone mode (independent JVM):
- Lance needs to connect to Gravitino server via HTTP to access
catalog/schema/table metadata
- GRAVITINO_URI and GRAVITINO_METALAKE configs specify the connection
parameters
- Use case: Deploy Lance REST separately for isolation or scaling

Auxiliary mode (embedded in Gravitino server):
- Lance runs in the same JVM as Gravitino, enabling direct memory access
(zero overhead)
- META-INF/services file enables automatic service discovery via Java
ServiceLoader
- Use case: Simplified deployment with shared resources and unified
monitoring


### Does this PR introduce _any_ user-facing change?

Yes - New configuration properties

For Standalone Mode:
```
# Required: Gravitino server URI
gravitino.lance-rest.gravitino-uri=http://gravitino-server:8090

# Required: Metalake name
gravitino.lance-rest.gravitino-metalake=production
```

For Auxiliary Mode:
```
# Enable Lance REST as auxiliary service
gravitino.auxService.names=iceberg-rest,lance-rest

# Classpath for Lance REST JAR
gravitino.lance-rest.classpath=lance/lance-rest-server/build/libs
```

Deployment modes supported:
Standalone: Lance REST runs independently, connects to Gravitino via
HTTP
Auxiliary: Lance REST runs embedded in Gravitino, uses direct memory
access
No breaking changes - all changes are additive and backward compatible.

### How was this patch tested?

1. unit test
2. manual test

**Standalone mode test:**
```
junda@java(junda.devpod-us-or) ~/gravitino/distribution/package/bin
 % ./gravitino.sh start
Gravitino Server is running[PID:3907790]
junda@java(junda.devpod-us-or) ~
 % curl http://localhost:8090/api/version
{"code":0,"version":{"version":"1.1.0-SNAPSHOT","compileDate":"23/10/2025 21:03:17","gitCommit":"35bd819975badea2493b4dbd9d0de68425229945"}}%   

junda@java(junda.devpod-us-or) ~/gravitino/distribution/package/bin
 % ./gravitino-lance-rest-server.sh start
GravitinoLanceRESTServer is running[PID:3930096]
junda@java(junda.devpod-us-or) ~
 % curl "http://localhost:9101/lance/v1/namespace/root/list"
{"id":"root","delimiter":"$","namespaces":["default"]}%   
```

**Auxiliary mode test:**
```
junda@java(junda.devpod-us-or) ~/gravitino/distribution/package
 % cd /home/user/gravitino && cat >> distribution/package/conf/gravitino.conf << 'EOF'

# ===== Lance REST Auxiliary Service =====
gravitino.auxService.names = iceberg-rest,lance-rest
gravitino.lance-rest.classpath = /home/user/gravitino/lance/lance-rest-server/build/libs
gravitino.lance-rest.catalog-name = lance_catalog
EOF

junda@java(junda.devpod-us-or) ~/gravitino/distribution/package
 % ./bin/gravitino.sh start          
Gravitino Server is running[PID:274396]

junda@java(junda.devpod-us-or) ~/gravitino/distribution/package
 % tail -50 logs/gravitino-server.log | grep -i lance | grep -i started
2025-10-23 21:34:41.495 INFO [main] [org.apache.gravitino.server.web.JettyServer.start(JettyServer.java:202)] - lance-rest web server started on host 0.0.0.0 port 9101.
2025-10-23 21:34:41.495 INFO [main] [org.apache.gravitino.lance.LanceRESTService.serviceStart(LanceRESTService.java:77)] - Lance REST service started
```
youngyjd added a commit to youngyjd/gravitino that referenced this pull request Nov 19, 2025
…ode support for Lance REST service (apache#8903)

### What changes were proposed in this pull request?

1. Created service discovery file
(META-INF/services/org.apache.gravitino.auxiliary.GravitinoAuxiliaryService)
2. Added unit tests (TestLanceConfig.java)

### Why are the changes needed?
Fix: apache#8891 

To enable flexible deployment of Lance REST service:

Standalone mode (independent JVM):
- Lance needs to connect to Gravitino server via HTTP to access
catalog/schema/table metadata
- GRAVITINO_URI and GRAVITINO_METALAKE configs specify the connection
parameters
- Use case: Deploy Lance REST separately for isolation or scaling

Auxiliary mode (embedded in Gravitino server):
- Lance runs in the same JVM as Gravitino, enabling direct memory access
(zero overhead)
- META-INF/services file enables automatic service discovery via Java
ServiceLoader
- Use case: Simplified deployment with shared resources and unified
monitoring


### Does this PR introduce _any_ user-facing change?

Yes - New configuration properties

For Standalone Mode:
```
# Required: Gravitino server URI
gravitino.lance-rest.gravitino-uri=http://gravitino-server:8090

# Required: Metalake name
gravitino.lance-rest.gravitino-metalake=production
```

For Auxiliary Mode:
```
# Enable Lance REST as auxiliary service
gravitino.auxService.names=iceberg-rest,lance-rest

# Classpath for Lance REST JAR
gravitino.lance-rest.classpath=lance/lance-rest-server/build/libs
```

Deployment modes supported:
Standalone: Lance REST runs independently, connects to Gravitino via
HTTP
Auxiliary: Lance REST runs embedded in Gravitino, uses direct memory
access
No breaking changes - all changes are additive and backward compatible.

### How was this patch tested?

1. unit test
2. manual test

**Standalone mode test:**
```
junda@java(junda.devpod-us-or) ~/gravitino/distribution/package/bin
 % ./gravitino.sh start
Gravitino Server is running[PID:3907790]
junda@java(junda.devpod-us-or) ~
 % curl http://localhost:8090/api/version
{"code":0,"version":{"version":"1.1.0-SNAPSHOT","compileDate":"23/10/2025 21:03:17","gitCommit":"35bd819975badea2493b4dbd9d0de68425229945"}}%   

junda@java(junda.devpod-us-or) ~/gravitino/distribution/package/bin
 % ./gravitino-lance-rest-server.sh start
GravitinoLanceRESTServer is running[PID:3930096]
junda@java(junda.devpod-us-or) ~
 % curl "http://localhost:9101/lance/v1/namespace/root/list"
{"id":"root","delimiter":"$","namespaces":["default"]}%   
```

**Auxiliary mode test:**
```
junda@java(junda.devpod-us-or) ~/gravitino/distribution/package
 % cd /home/user/gravitino && cat >> distribution/package/conf/gravitino.conf << 'EOF'

# ===== Lance REST Auxiliary Service =====
gravitino.auxService.names = iceberg-rest,lance-rest
gravitino.lance-rest.classpath = /home/user/gravitino/lance/lance-rest-server/build/libs
gravitino.lance-rest.catalog-name = lance_catalog
EOF

junda@java(junda.devpod-us-or) ~/gravitino/distribution/package
 % ./bin/gravitino.sh start          
Gravitino Server is running[PID:274396]

junda@java(junda.devpod-us-or) ~/gravitino/distribution/package
 % tail -50 logs/gravitino-server.log | grep -i lance | grep -i started
2025-10-23 21:34:41.495 INFO [main] [org.apache.gravitino.server.web.JettyServer.start(JettyServer.java:202)] - lance-rest web server started on host 0.0.0.0 port 9101.
2025-10-23 21:34:41.495 INFO [main] [org.apache.gravitino.lance.LanceRESTService.serviceStart(LanceRESTService.java:77)] - Lance REST service started
```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Subtask] Support for standalone mode and auxiliary mode of Lance REST Server

3 participants