From 654af681b6a2f4d13e22f8c54701347be876251b Mon Sep 17 00:00:00 2001 From: Alexandre Dutra Date: Tue, 21 Jan 2025 14:57:16 +0100 Subject: [PATCH 1/2] [WIP] Adapt main README.md to Quarkus --- README.md | 116 +++++++++++++++++++++++++++++++++++++++++------------- 1 file changed, 89 insertions(+), 27 deletions(-) diff --git a/README.md b/README.md index da1a27eb55..bf4bdd83a3 100644 --- a/README.md +++ b/README.md @@ -39,15 +39,38 @@ for contribution guidelines. ## Building and Running Apache Polaris is organized into the following modules: + - `polaris-core` - The main Polaris entity definitions and core business logic -- `polaris-server` - The Polaris REST API server -- `polaris-eclipselink` - The Eclipselink implementation of the MetaStoreManager interface +- API modules (implementing the Iceberg REST API and Polaris management API): + - `polaris-api-management-model` - The Polaris management model + - `polaris-api-management-service` - The Polaris management service + - `polaris-api-iceberg-service` - The Iceberg REST service +- Service modules: + - `polaris-service-common` - The main components of the Polaris server +- Quarkus runtime modules: + - `polaris-quarkus-service` - The Quarkus-specific components of the Polaris server + - `polaris-quarkus-defaults` - The Quarkus-specific configuration defaults + - `polaris-quarkus-server` - The Polaris server runtime + - `polaris-quarkus-admin-tool` - The Polaris admin & maintenance tool +- Persistence modules + - `polaris-jpa-model` - The JPA entity definitions + - `polaris-eclipselink` - The Eclipselink implementation of the MetaStoreManager interface Apache Polaris is built using Gradle with Java 21+ and Docker 27+. + - `./gradlew build` - To build and run tests. Make sure Docker is running, as the integration tests depend on it. - `./gradlew assemble` - To skip tests. - `./gradlew test` - To run unit tests and integration tests. -- `./gradlew runApp` - To run the Polaris server locally on localhost:8181. + +For local development, you can run the following commands: + +- `./gradlew polarisServerRun` - To run the Polaris server locally, with profile `prod`; the server + is reachable at localhost:8181. +- `java -Dquarkus.profile=test -jar quarkus/server/build/quarkus-app/quarkus-run.jar` - To run the + Polaris server locally, with profile `test`. With this profile, Polaris uses the `test` + Authenticator and `test` TokenBroker; this configuration is suitable for running regressions + tests, or for connecting with Spark. + - `./regtests/run_spark_sql.sh` - To connect from Spark SQL. Here are some example commands to run in the Spark SQL shell: ```sql create database db1; @@ -57,36 +80,75 @@ insert into db1.table1 values (1, 'a'); select * from db1.table1; ``` -Apache Polaris supports the following optional build options: -- `-PeclipseLink=true` – Enables the EclipseLink extension. -- `-PeclipseLinkDeps=[groupId]:[artifactId]:[version],...` – Specifies one or more additional dependencies for EclipseLink (e.g., JDBC drivers) separated by commas. - ### More build and run options -Running in Docker -- `docker build -t localhost:5001/polaris:latest .` - To build the image. - - Optional build options: - - `docker build -t localhost:5001/polaris:latest --build-arg ECLIPSELINK=true .` - Enables the EclipseLink extension. - - `docker build -t localhost:5001/polaris:latest --build-arg ECLIPSELINK=true --build-arg ECLIPSELINK_DEPS=[groupId]:[artifactId]:[version],... .` – Enables the EclipseLink extension with one or more additional dependencies for EclipseLink (e.g. JDBC drivers) separated by commas. -- `docker run -p 8181:8181 localhost:5001/polaris:latest` - To run the image in standalone mode. - -Running in Kubernetes -- `./run.sh` - To run Polaris as a mini-deployment locally. This will create one pod that bind itself to ports `8181` and `8182`. - - Optional run options: - - `./run.sh -b "ECLIPSELINK=true"` - Enables the EclipseLink extension. - - `./run.sh -b "ECLIPSELINK=true;ECLIPSELINK_DEPS=[groupId]:[artifactId]:[version],..."` – Enables the EclipseLink extension with one or more additional dependencies for EclipseLink (e.g. JDBC drivers) separated by commas. -- `kubectl port-forward svc/polaris-service -n polaris 8181:8181 8182:8182` - To create secure connections between a local machine and a pod within the cluster for both service and metrics endpoints. - - Currently supported metrics endpoints: - - localhost:8182/metrics - - localhost:8182/healthcheck + +#### Running in Docker + +Please note: there are no official Docker images for Apache Polaris yet. For now, you can build the +Docker images locally. + +To build the Polaris server Docker image locally: + +```shell +./gradlew clean :polaris-quarkus-server:assemble -Dquarkus.container-image.build=true +``` + +To run the Polaris server Docker image: + +```shell +docker run -p 8181:8181 -p 8182:8182 apache/polaris:latest +``` + +#### Running in Kubernetes + +- `./run.sh` - To run Polaris as a mini-deployment locally. This will create a Kind cluster, + then deploy one pod and one service. The service is available on ports `8181` and `8182`. +- `kubectl port-forward svc/polaris-service -n polaris 8181:8181 8182:8182` - To create secure + connections between a local machine and a pod within the cluster for both service and metrics + endpoints. + - Currently supported metrics and health endpoints: + - http://localhost:8182/q/metrics + - http://localhost:8182/q/health - `kubectl get pods -n polaris` - To check the status of the pods. - `kubectl get deployment -n polaris` - To check the status of the deployment. - `kubectl describe deployment polaris-deployment -n polaris` - To troubleshoot if things aren't working as expected. -Running regression tests -- `./regtests/run.sh` - To run regression tests in another terminal. -- `docker compose up --build --exit-code-from regtest` - To run regression tests in a Docker environment. +#### Running regression tests + +Regression tests can be run in a local environment or in a Docker environment. + +To run regression tests locally, you need to have a Polaris server running locally, with the +`test` Authenticator enabled. You can do this by running Polaris as below: + +```shell +java -Dquarkus.profile=test -jar quarkus/server/build/quarkus-app/quarkus-run.jar +``` + +Then, you can run the regression tests using the following command: + +```shell +POLARIS_HOST=localhost ./regtests/run.sh +``` + +To run regression tests in a Docker environment, you can use the following command: + +```shell +docker compose -f regtests/docker-compose.yml up --build --exit-code-from regtest +``` + +The above command will by default run Polaris with the Docker image `apache/polaris:latest`; if you +want to use a different image, you can modify the `docker-compose.yaml` file prior to running it; +alternatively, you can use the following commands to override the image: + +```shell +cat < /tmp/polaris-image.yml +services: { polaris: { image: localhost:5001/apache/polaris:latest } } +EOF +docker compose -f regtests/docker-compose.yml -f /tmp/polaris-image.yml up --build --exit-code-from regtest +``` + +#### Building docs -Building docs - Docs are generated using [Hugo](https://gohugo.io/) using the [Docsy](https://www.docsy.dev/docs/) theme. - To view the site locally, run ```bash From 5998e39d9defd317830697fe9664d3e7fc411819 Mon Sep 17 00:00:00 2001 From: Alexandre Dutra Date: Thu, 23 Jan 2025 10:41:30 +0100 Subject: [PATCH 2/2] review --- README.md | 44 +++++++++----------------------------------- 1 file changed, 9 insertions(+), 35 deletions(-) diff --git a/README.md b/README.md index bf4bdd83a3..1e1e9ed54a 100644 --- a/README.md +++ b/README.md @@ -52,7 +52,7 @@ Apache Polaris is organized into the following modules: - `polaris-quarkus-defaults` - The Quarkus-specific configuration defaults - `polaris-quarkus-server` - The Polaris server runtime - `polaris-quarkus-admin-tool` - The Polaris admin & maintenance tool -- Persistence modules +- Persistence modules: - `polaris-jpa-model` - The JPA entity definitions - `polaris-eclipselink` - The Eclipselink implementation of the MetaStoreManager interface @@ -61,9 +61,6 @@ Apache Polaris is built using Gradle with Java 21+ and Docker 27+. - `./gradlew build` - To build and run tests. Make sure Docker is running, as the integration tests depend on it. - `./gradlew assemble` - To skip tests. - `./gradlew test` - To run unit tests and integration tests. - -For local development, you can run the following commands: - - `./gradlew polarisServerRun` - To run the Polaris server locally, with profile `prod`; the server is reachable at localhost:8181. - `java -Dquarkus.profile=test -jar quarkus/server/build/quarkus-app/quarkus-run.jar` - To run the @@ -84,31 +81,19 @@ select * from db1.table1; #### Running in Docker -Please note: there are no official Docker images for Apache Polaris yet. For now, you can build the -Docker images locally. - -To build the Polaris server Docker image locally: - -```shell -./gradlew clean :polaris-quarkus-server:assemble -Dquarkus.container-image.build=true -``` - -To run the Polaris server Docker image: - -```shell -docker run -p 8181:8181 -p 8182:8182 apache/polaris:latest -``` +- `./gradlew clean :polaris-quarkus-server:assemble -Dquarkus.container-image.build=true` - To + build the image locally. +- `docker run -p 8181:8181 -p 8182:8182 apache/polaris:latest` - To run the image. #### Running in Kubernetes - `./run.sh` - To run Polaris as a mini-deployment locally. This will create a Kind cluster, then deploy one pod and one service. The service is available on ports `8181` and `8182`. - `kubectl port-forward svc/polaris-service -n polaris 8181:8181 8182:8182` - To create secure - connections between a local machine and a pod within the cluster for both service and metrics - endpoints. - - Currently supported metrics and health endpoints: - - http://localhost:8182/q/metrics - - http://localhost:8182/q/health + connections between a local machine and a pod within the cluster for both service and + health/metrics endpoints: + - http://localhost:8182/q/metrics + - http://localhost:8182/q/health - `kubectl get pods -n polaris` - To check the status of the pods. - `kubectl get deployment -n polaris` - To check the status of the deployment. - `kubectl describe deployment polaris-deployment -n polaris` - To troubleshoot if things aren't working as expected. @@ -127,7 +112,7 @@ java -Dquarkus.profile=test -jar quarkus/server/build/quarkus-app/quarkus-run.ja Then, you can run the regression tests using the following command: ```shell -POLARIS_HOST=localhost ./regtests/run.sh +env POLARIS_HOST=localhost ./regtests/run.sh ``` To run regression tests in a Docker environment, you can use the following command: @@ -136,17 +121,6 @@ To run regression tests in a Docker environment, you can use the following comma docker compose -f regtests/docker-compose.yml up --build --exit-code-from regtest ``` -The above command will by default run Polaris with the Docker image `apache/polaris:latest`; if you -want to use a different image, you can modify the `docker-compose.yaml` file prior to running it; -alternatively, you can use the following commands to override the image: - -```shell -cat < /tmp/polaris-image.yml -services: { polaris: { image: localhost:5001/apache/polaris:latest } } -EOF -docker compose -f regtests/docker-compose.yml -f /tmp/polaris-image.yml up --build --exit-code-from regtest -``` - #### Building docs - Docs are generated using [Hugo](https://gohugo.io/) using the [Docsy](https://www.docsy.dev/docs/) theme.