From 5e54dd24ff524581b8ceb73bbb0c7a0e2357e177 Mon Sep 17 00:00:00 2001 From: Alexander Batashev Date: Mon, 18 Oct 2021 16:10:42 +0300 Subject: [PATCH 1/5] [Doc] Add documentation for Docker images --- .github/CODEOWNERS | 1 + sycl/doc/GetStartedGuide.md | 11 ++++++ sycl/doc/dev/DockerBKMs.md | 72 +++++++++++++++++++++++++++++++++++++ sycl/doc/index.rst | 10 ++++-- 4 files changed, 92 insertions(+), 2 deletions(-) create mode 100644 sycl/doc/dev/DockerBKMs.md diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index 994530608daaa..3e99f18f4e1a7 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -26,6 +26,7 @@ sycl/ReleaseNotes.md @pvchupin @tfzhu sycl/doc/ @pvchupin @bader sycl/doc/extensions/ @intel/dpcpp-specification-reviewers sycl/doc/SPIRV @AlexeySotkin @bashbaug @mbelicki +sycl/doc/dev @bader @vladimirlaz # Sub-groups sycl/include/CL/sycl/detail/spirv.hpp @Pennycook @AlexeySachkov diff --git a/sycl/doc/GetStartedGuide.md b/sycl/doc/GetStartedGuide.md index 932546c90156c..932b997f1e6ae 100644 --- a/sycl/doc/GetStartedGuide.md +++ b/sycl/doc/GetStartedGuide.md @@ -46,6 +46,17 @@ and a wide range of compute accelerators such as GPU and FPGA. * Windows: `Visual Studio` version 15.7 preview 4 or later - [Download](https://visualstudio.microsoft.com/downloads/) +Alternatively you can use Docker image, that has everything you need +pre-installed: + +``` +docker run --name sycl_build -it --entrypoint=/bin/bash -v /local/workspace/dir/:/src ghcr.io/intel/llvm/ubuntu2004_base +``` + +This command will start a terminal session, from which you can proceed with the +instructions below. See [Docker BKMs](dev/DockerBKMs.md) for more info on Docker +commands. + ### Create DPC++ workspace Throughout this document `DPCPP_HOME` denotes the path to the local directory diff --git a/sycl/doc/dev/DockerBKMs.md b/sycl/doc/dev/DockerBKMs.md new file mode 100644 index 0000000000000..db62c17a984c3 --- /dev/null +++ b/sycl/doc/dev/DockerBKMs.md @@ -0,0 +1,72 @@ +# Docker Containers BKMs + +## Docker vs Podman + +Docker and Podman are very similar tools, that allow you to manage and run +container images. Unlike Docker, Podman runs without a daemon, allows you to run +containers without root permissions, but does not let you build a container from +source. The command line syntax is mostly identical for Docker and Podman. +Choose whatever is available on your system. + +## SYCL Containers overview + +The following containers are publicly available for DPC++ compiler development: + +- `ghcr.io/intel/llvm/ubuntu2004_base`: contains basic environment setup for + building DPC++ compiler from source. +- `ghcr.io/intel/llvm/ubuntu2004_intel_drivers`: contains everything from base + container + pre-installed Intel drivers. This image provides two main tags: + `latest`, that uses latest available drivers, and `stable`, that uses + recommended drivers. +- `ghcr.io/intel/llvm/ubuntu2004_nightly`: contains latest successfully built + nightly build of DPC++ compiler, as well as pre-installed Intel drivers. + +## Building a Docker Container from scratch + +Docker containers can be build with the following command: + +``` +docker build -f path/to/devops/containers/file.Dockerfile path/to/devops/ +``` + +The `ubuntu2004_preinstalled.Dockerfile` script expects `llvm_sycl.tar.gz` file +to be present in `devops/` directory. + +Containers other than base provide several configurable arguments, most commonly +used are`base_image` and `base_tag`, that specify base Docker image and its tag. +You can set additional arguments with `--build-arg ARG=value` argument. + +## Running Docker container interactively + +The main application of Docker is containerizing services. But it also allows +you to run containers interactively, so that you can use it as you would use a +terminal or SSH session. The following command allows you to do that: + +``` +docker run --name -it --entrypoint /bin/bash [:] +``` + +This command will download an image, if it does not exist locally. If you've +downloaded an image previously, and you want to update it, use +`docker pull ` command. + +## Passthrough an Intel GPU to container + +Add `--device=/dev/dri` argument to `run` command to passthrough you Intel GPU. + +## Passthrough a directory to container + +Use `-v path/on/host:path/in/container` argument for `run` command to +passthrough a host directory or a file. + +## Managing downloaded Docker image + +List local images: +``` +docker image ls +``` + +Remove local image: +``` +docker image rm +``` diff --git a/sycl/doc/index.rst b/sycl/doc/index.rst index 2341a77869bfa..ecff1e769e9ef 100644 --- a/sycl/doc/index.rst +++ b/sycl/doc/index.rst @@ -2,7 +2,7 @@ Data Parallel C++ Documentation =============================== Using oneAPI DPC++ for Application Development -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +---------------------------------------------- .. toctree:: :maxdepth: 1 @@ -18,7 +18,7 @@ Using oneAPI DPC++ for Application Development EnvironmentVariables Developing oneAPI DPC++ Compiler -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +-------------------------------- .. toctree:: :maxdepth: 1 @@ -41,3 +41,9 @@ Developing oneAPI DPC++ Compiler SYCLInstrumentationUsingXPTI ITTAnnotations +Development BKMs +~~~~~~~~~~~~~~~~ + +.. toctree:: + :maxdepth: 1 + dev/DockerBKMs From 4093e6a32c88223d6091e764a0330c7dc8af1f7e Mon Sep 17 00:00:00 2001 From: Alexander Batashev Date: Wed, 15 Dec 2021 17:59:44 +0300 Subject: [PATCH 2/5] update for current images --- sycl/doc/GetStartedGuide.md | 4 +- sycl/doc/dev/DockerBKMs.md | 104 +++++++++++++++++++++++++++++++----- 2 files changed, 93 insertions(+), 15 deletions(-) diff --git a/sycl/doc/GetStartedGuide.md b/sycl/doc/GetStartedGuide.md index 932b997f1e6ae..28aa0a829db6e 100644 --- a/sycl/doc/GetStartedGuide.md +++ b/sycl/doc/GetStartedGuide.md @@ -46,11 +46,11 @@ and a wide range of compute accelerators such as GPU and FPGA. * Windows: `Visual Studio` version 15.7 preview 4 or later - [Download](https://visualstudio.microsoft.com/downloads/) -Alternatively you can use Docker image, that has everything you need +Alternatively, you can use Docker image, that has everything you need pre-installed: ``` -docker run --name sycl_build -it --entrypoint=/bin/bash -v /local/workspace/dir/:/src ghcr.io/intel/llvm/ubuntu2004_base +docker run --name sycl_build -it -v /local/workspace/dir/:/src ghcr.io/intel/llvm/ubuntu2004_base /bin/bash ``` This command will start a terminal session, from which you can proceed with the diff --git a/sycl/doc/dev/DockerBKMs.md b/sycl/doc/dev/DockerBKMs.md index db62c17a984c3..0d23bbbe084e4 100644 --- a/sycl/doc/dev/DockerBKMs.md +++ b/sycl/doc/dev/DockerBKMs.md @@ -5,7 +5,7 @@ Docker and Podman are very similar tools, that allow you to manage and run container images. Unlike Docker, Podman runs without a daemon, allows you to run containers without root permissions, but does not let you build a container from -source. The command line syntax is mostly identical for Docker and Podman. +the source. The command line syntax is mostly identical for Docker and Podman. Choose whatever is available on your system. ## SYCL Containers overview @@ -14,27 +14,27 @@ The following containers are publicly available for DPC++ compiler development: - `ghcr.io/intel/llvm/ubuntu2004_base`: contains basic environment setup for building DPC++ compiler from source. -- `ghcr.io/intel/llvm/ubuntu2004_intel_drivers`: contains everything from base - container + pre-installed Intel drivers. This image provides two main tags: - `latest`, that uses latest available drivers, and `stable`, that uses - recommended drivers. -- `ghcr.io/intel/llvm/ubuntu2004_nightly`: contains latest successfully built - nightly build of DPC++ compiler, as well as pre-installed Intel drivers. +- `ghcr.io/intel/llvm/ubuntu2004_intel_drivers`: contains everything from the + base container + pre-installed Intel drivers. +- `ghcr.io/intel/llvm/sycl_ubuntu2004_nightly`: contains the latest successfully + built nightly build of DPC++ compiler. The image comes in two flavors: + with pre-installed Intel drivers (`latest`) and without them (`no-drivers`). ## Building a Docker Container from scratch -Docker containers can be build with the following command: +Docker containers can be built with the following command: ``` docker build -f path/to/devops/containers/file.Dockerfile path/to/devops/ ``` -The `ubuntu2004_preinstalled.Dockerfile` script expects `llvm_sycl.tar.gz` file +The `ubuntu2004_preinstalled.Dockerfile` script expects `llvm_sycl.tar.xz` file to be present in `devops/` directory. -Containers other than base provide several configurable arguments, most commonly -used are`base_image` and `base_tag`, that specify base Docker image and its tag. -You can set additional arguments with `--build-arg ARG=value` argument. +Containers other than base provide several configurable arguments, the most +commonly used are `base_image` and `base_tag`, which specify the base Docker +image and its tag. You can set additional arguments with `--build-arg ARG=value` +argument. ## Running Docker container interactively @@ -43,7 +43,7 @@ you to run containers interactively, so that you can use it as you would use a terminal or SSH session. The following command allows you to do that: ``` -docker run --name -it --entrypoint /bin/bash [:] +docker run --name -it [:] /bin/bash ``` This command will download an image, if it does not exist locally. If you've @@ -53,12 +53,46 @@ downloaded an image previously, and you want to update it, use ## Passthrough an Intel GPU to container Add `--device=/dev/dri` argument to `run` command to passthrough you Intel GPU. +Make sure you're a member of `video` group to be able to access GPU. ## Passthrough a directory to container Use `-v path/on/host:path/in/container` argument for `run` command to passthrough a host directory or a file. +## Persisting data with volumes + +Docker container images are read-only. When container is destroyed, all its data +is lost. To persist data when working with containers (i.e. when upgrading +container version) one can use Docker volumes. + +Creating a volume: + +``` +docker volume create +``` + +Listing all volumes: + +``` +docker volume list +``` + +Mounting volume to the container: + +``` +docker run -v :/path/inside/container bash +``` + +Deleting a volume: + +``` +docker volume rm +``` + +See [official documentation](https://docs.docker.com/storage/volumes/) for more +info. + ## Managing downloaded Docker image List local images: @@ -70,3 +104,47 @@ Remove local image: ``` docker image rm ``` + +## Change location of the images on the system + +By default Docker stores images in `/var/lib/docker`, but that can be changed. + +Create a new file called `/etc/docker/daemon.json` and put the following +content: + +``` +{ + "data-root": "/path/to/data/root", + "exec-root": "/path/to/exec/root" +} +``` + +## Running Docker without sudo + +Add your local user to `docker` group to be able to run docker commands without +sudo. + +## Changing Docker user + +By default all processes inside Docker run as root. Some LLVM or Clang tests +expect your user to be anything but root. You can change the user by specifying +`-u ` option. All Docker containers come with user `sycl` +created. + +## Managing disk usage + +See how much space is taken by docker: + +``` +docker system df +``` + +Cleaning unused data: + +``` +docker system prune +``` + +See [official documentation](https://docs.docker.com/engine/reference/commandline/system_prune/) +for more info. + From 1a29acbc026639d2a32d11325ae22f18abc5fe93 Mon Sep 17 00:00:00 2001 From: Alexander Batashev Date: Wed, 15 Dec 2021 18:04:33 +0300 Subject: [PATCH 3/5] Apply suggestions from code review Co-authored-by: vladimirlaz --- sycl/doc/dev/DockerBKMs.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/sycl/doc/dev/DockerBKMs.md b/sycl/doc/dev/DockerBKMs.md index 0d23bbbe084e4..e05379f20478a 100644 --- a/sycl/doc/dev/DockerBKMs.md +++ b/sycl/doc/dev/DockerBKMs.md @@ -39,20 +39,20 @@ argument. ## Running Docker container interactively The main application of Docker is containerizing services. But it also allows -you to run containers interactively, so that you can use it as you would use a +you to run containers interactively, so that you can use them as you would use a terminal or SSH session. The following command allows you to do that: ``` docker run --name -it [:] /bin/bash ``` -This command will download an image, if it does not exist locally. If you've +This command will download an image if it does not exist locally. If you've downloaded an image previously, and you want to update it, use `docker pull ` command. ## Passthrough an Intel GPU to container -Add `--device=/dev/dri` argument to `run` command to passthrough you Intel GPU. +Add `--device=/dev/dri` argument to `run` command to passthrough Intel GPU. Make sure you're a member of `video` group to be able to access GPU. ## Passthrough a directory to container From 20dc963384f80b6e220bc853813e5db6e32e6e03 Mon Sep 17 00:00:00 2001 From: Alexander Batashev Date: Sat, 18 Dec 2021 09:38:10 +0300 Subject: [PATCH 4/5] rearrange things --- sycl/doc/dev/DockerBKMs.md | 126 ++++++++++++++++++++++--------------- 1 file changed, 74 insertions(+), 52 deletions(-) diff --git a/sycl/doc/dev/DockerBKMs.md b/sycl/doc/dev/DockerBKMs.md index e05379f20478a..365d981e22b67 100644 --- a/sycl/doc/dev/DockerBKMs.md +++ b/sycl/doc/dev/DockerBKMs.md @@ -1,12 +1,36 @@ # Docker Containers BKMs -## Docker vs Podman +## Installation + +Follow the [official guide](https://docs.docker.com/engine/install/) for your +OS or distro. + +### Change location of the images on the system + +By default Docker stores images in `/var/lib/docker`, but that can be changed. + +Create a new file called `/etc/docker/daemon.json` and put the following +content: + +``` +{ + "data-root": "/path/to/data/root", + "exec-root": "/path/to/exec/root" +} +``` + +### Running Docker without sudo + +Add your local user to `docker` group to be able to run docker commands without +sudo. + + +### Docker vs Podman Docker and Podman are very similar tools, that allow you to manage and run -container images. Unlike Docker, Podman runs without a daemon, allows you to run -containers without root permissions, but does not let you build a container from -the source. The command line syntax is mostly identical for Docker and Podman. -Choose whatever is available on your system. +container images. Unlike Docker, Podman runs without a daemon and allows you +to run containers without root permissions. The command line syntax is mostly +identical for Docker and Podman. Choose whatever is available on your system. ## SYCL Containers overview @@ -20,22 +44,6 @@ The following containers are publicly available for DPC++ compiler development: built nightly build of DPC++ compiler. The image comes in two flavors: with pre-installed Intel drivers (`latest`) and without them (`no-drivers`). -## Building a Docker Container from scratch - -Docker containers can be built with the following command: - -``` -docker build -f path/to/devops/containers/file.Dockerfile path/to/devops/ -``` - -The `ubuntu2004_preinstalled.Dockerfile` script expects `llvm_sycl.tar.xz` file -to be present in `devops/` directory. - -Containers other than base provide several configurable arguments, the most -commonly used are `base_image` and `base_tag`, which specify the base Docker -image and its tag. You can set additional arguments with `--build-arg ARG=value` -argument. - ## Running Docker container interactively The main application of Docker is containerizing services. But it also allows @@ -50,17 +58,13 @@ This command will download an image if it does not exist locally. If you've downloaded an image previously, and you want to update it, use `docker pull ` command. -## Passthrough an Intel GPU to container - -Add `--device=/dev/dri` argument to `run` command to passthrough Intel GPU. -Make sure you're a member of `video` group to be able to access GPU. - -## Passthrough a directory to container +## Persisting data -Use `-v path/on/host:path/in/container` argument for `run` command to -passthrough a host directory or a file. +This command will download an image, if it does not exist locally. If you've +downloaded an image previously, and you want to update it, use +`docker pull ` command. -## Persisting data with volumes +### Persisting data with volumes Docker container images are read-only. When container is destroyed, all its data is lost. To persist data when working with containers (i.e. when upgrading @@ -93,36 +97,26 @@ docker volume rm See [official documentation](https://docs.docker.com/storage/volumes/) for more info. -## Managing downloaded Docker image +### Passthrough a directory to a container -List local images: -``` -docker image ls -``` +Use `-v path/on/host:path/in/container` argument for `run` command to +passthrough a host directory or a file. -Remove local image: -``` -docker image rm -``` +## GPU passthrough -## Change location of the images on the system +### Intel -By default Docker stores images in `/var/lib/docker`, but that can be changed. +Add `--device=/dev/dri` argument to `run` command to passthrough you Intel GPU. +Make sure you're a member of `video` group to be able to access GPU. -Create a new file called `/etc/docker/daemon.json` and put the following -content: +### AMD -``` -{ - "data-root": "/path/to/data/root", - "exec-root": "/path/to/exec/root" -} -``` +Follow the [official guide](https://rocmdocs.amd.com/en/latest/ROCm_Virtualization_Containers/ROCm-Virtualization-&-Containers.html). -## Running Docker without sudo +### Nvidia -Add your local user to `docker` group to be able to run docker commands without -sudo. +Follow [these](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/user-guide.html) +instructions. ## Changing Docker user @@ -131,6 +125,18 @@ expect your user to be anything but root. You can change the user by specifying `-u ` option. All Docker containers come with user `sycl` created. +## Managing downloaded Docker images + +List local images: +``` +docker image ls +``` + +Remove local image: +``` +docker image rm +``` + ## Managing disk usage See how much space is taken by docker: @@ -148,3 +154,19 @@ docker system prune See [official documentation](https://docs.docker.com/engine/reference/commandline/system_prune/) for more info. +## Building a Docker Container from scratch + +Docker containers can be built with the following command: + +``` +docker build -f path/to/devops/containers/file.Dockerfile path/to/devops/ +``` + +The `ubuntu2004_preinstalled.Dockerfile` script expects `llvm_sycl.tar.xz` file +to be present in `devops/` directory. + +Containers other than base provide several configurable arguments, the most +commonly used are `base_image` and `base_tag`, which specify the base Docker +image and its tag. You can set additional arguments with `--build-arg ARG=value` +argument. + From 806387553bf2fbaf863c658fe304b0a2911a3d58 Mon Sep 17 00:00:00 2001 From: Alexander Batashev Date: Mon, 20 Dec 2021 14:27:27 +0300 Subject: [PATCH 5/5] remove merge conflict leftovers --- sycl/doc/dev/DockerBKMs.md | 4 ---- 1 file changed, 4 deletions(-) diff --git a/sycl/doc/dev/DockerBKMs.md b/sycl/doc/dev/DockerBKMs.md index 365d981e22b67..928c1f82000f4 100644 --- a/sycl/doc/dev/DockerBKMs.md +++ b/sycl/doc/dev/DockerBKMs.md @@ -60,10 +60,6 @@ downloaded an image previously, and you want to update it, use ## Persisting data -This command will download an image, if it does not exist locally. If you've -downloaded an image previously, and you want to update it, use -`docker pull ` command. - ### Persisting data with volumes Docker container images are read-only. When container is destroyed, all its data