diff --git a/docker/README.md b/docker/README.md new file mode 100644 index 0000000000000..19293de188448 --- /dev/null +++ b/docker/README.md @@ -0,0 +1,93 @@ + + +# Docker Demo for Hudi + +This repo contains the docker demo resources for building docker demo images, set up the demo, and running Hudi in the +docker demo environment. + +## Repo Organization + +### Configs for assembling docker images - `/hoodie` + +The `/hoodie` folder contains all the configs for assembling necessary docker images. The name and repository of each +docker image, e.g., `apachehudi/hudi-hadoop_2.8.4-trinobase_368`, is defined in the maven configuration file `pom.xml`. + +### Docker compose config for the Demo - `/compose` + +The `/compose` folder contains the yaml file to compose the Docker environment for running Hudi Demo. + +### Resources and Sample Data for the Demo - `/demo` + +The `/demo` folder contains useful resources and sample data use for the Demo. + +## Build and Test Image locally + +To build all docker images locally, you can run the script: + +```shell +./build_local_docker_images.sh +``` + +To build a single image target, you can run + +```shell +mvn clean pre-integration-test -DskipTests -Ddocker.compose.skip=true -Ddocker.build.skip=false -pl : -am +# For example, to build hudi-hadoop-trinobase-docker +mvn clean pre-integration-test -DskipTests -Ddocker.compose.skip=true -Ddocker.build.skip=false -pl :hudi-hadoop-trinobase-docker -am +``` + +Alternatively, you can use `docker` cli directly under `hoodie/hadoop`. Note that, you need to manually name your local +image by using `-t` option to match the naming in the `pom.xml`, so that you can update the corresponding image +repository in Docker Hub (detailed steps in the next section). + +```shell +# Run under hoodie/hadoop, the is optional, "latest" by default +docker build -t /[:] +# For example, to build trinobase +docker build trinobase -t apachehudi/hudi-hadoop_2.8.4-trinobase_368 +``` + +After new images are built, you can run the following script to bring up docker demo with your local images: + +```shell +./setup_demo.sh dev +``` + +## Upload Updated Image to Repository on Docker Hub + +Once you have built the updated image locally, you can push the corresponding this repository of the image to the Docker +Hud registry designated by its name or tag: + +```shell +docker push /: +# For example +docker push apachehudi/hudi-hadoop_2.8.4-trinobase_368 +``` + +You can also easily push the image to the Docker Hub using Docker Desktop app: go to `Images`, search for the image by +the name, and then click on the three dots and `Push to Hub`. + +![Push to Docker Hub](push_to_docker_hub.png) + +Note that you need to ask for permission to upload the Hudi Docker Demo images to the repositories. + +You can find more information on [Docker Hub Repositories Manual](https://docs.docker.com/docker-hub/repos/). + +## Docker Demo Setup + +Please refer to the [Docker Demo Docs page](https://hudi.apache.org/docs/docker_demo). \ No newline at end of file diff --git a/docker/push_to_docker_hub.png b/docker/push_to_docker_hub.png new file mode 100644 index 0000000000000..faa431bd16784 Binary files /dev/null and b/docker/push_to_docker_hub.png differ diff --git a/docker/setup_demo.sh b/docker/setup_demo.sh index 634fe9e509bdd..9f0a100da6122 100755 --- a/docker/setup_demo.sh +++ b/docker/setup_demo.sh @@ -17,10 +17,14 @@ # limitations under the License. SCRIPT_PATH=$(cd `dirname $0`; pwd) +HUDI_DEMO_ENV=$1 WS_ROOT=`dirname $SCRIPT_PATH` # restart cluster HUDI_WS=${WS_ROOT} docker-compose -f ${SCRIPT_PATH}/compose/docker-compose_hadoop284_hive233_spark244.yml down -HUDI_WS=${WS_ROOT} docker-compose -f ${SCRIPT_PATH}/compose/docker-compose_hadoop284_hive233_spark244.yml pull +if [ "$HUDI_DEMO_ENV" != "dev" ]; then + echo "Pulling docker demo images ..." + HUDI_WS=${WS_ROOT} docker-compose -f ${SCRIPT_PATH}/compose/docker-compose_hadoop284_hive233_spark244.yml pull +fi sleep 5 HUDI_WS=${WS_ROOT} docker-compose -f ${SCRIPT_PATH}/compose/docker-compose_hadoop284_hive233_spark244.yml up -d sleep 15