From cad0bf5e943ededae09ea60318d70684eebfe953 Mon Sep 17 00:00:00 2001 From: vnijs Date: Mon, 12 Aug 2024 17:40:25 -0700 Subject: [PATCH] updates --- README.md | 10 +- install/rsm-msba-chromeos.md | 2 - install/rsm-msba-linux.md | 2 - install/rsm-msba-macos-arm.md | 301 ++++++++++++++++++++++++++++++++ install/rsm-msba-macos-m1.md | 2 - install/rsm-msba-macos.md | 2 - install/rsm-msba-windows-arm.md | 19 ++ install/rsm-msba-windows.md | 19 ++ rsm-msba-arm/Dockerfile | 1 + rsm-msba-intel/Dockerfile | 1 + 10 files changed, 344 insertions(+), 15 deletions(-) create mode 100644 install/rsm-msba-macos-arm.md diff --git a/README.md b/README.md index 19936e7..2ed3460 100644 --- a/README.md +++ b/README.md @@ -21,14 +21,14 @@ Docker version 20.10.13, build a224086 * For detailed install instructions on Windows (Intel) see [install/rsm-msba-windows.md](install/rsm-msba-windows.md) * For detailed install instructions on Windows (ARM) see [install/rsm-msba-windows-arm.md](install/rsm-msba-windows-arm.md) -* For detailed install instructions on macOS (M1, M2, or M3) see [install/rsm-msba-macos-m1.md](install/rsm-msba-macos-m1.md) +* For detailed install instructions on macOS (M1, M2, M3, etc.) see [install/rsm-msba-macos-arm.md](install/rsm-msba-macos-arm.md) * For detailed install instructions on macOS (Intel) see [install/rsm-msba-macos.md](install/rsm-msba-macos.md) * For detailed install instructions on Linux see [install/rsm-msba-linux.md](install/rsm-msba-linux.md) * For detailed install instructions on ChromeOS see [install/rsm-msba-chromeos.md](install/rsm-msba-chromeos.md) ## rsm-msba-arm and rsm-msba-intel -`rsm-msba-arm` is built for M1, ARM based macOS computers. `rsm-msba-intel` is built for AMD based computers and includes Rstudio Server. To build a new image based on `rsm-jupter-rs` add the following at the top of your Dockerfile +`rsm-msba-arm` is built for M1, M2, etc., ARM based macOS computers. `rsm-msba-intel` is built for AMD based computers and includes Rstudio Server. To build a new image based on `rsm-msba_intel` add the following at the top of your Dockerfile ``` FROM vnijs/rsm-msba-intel:latest @@ -92,8 +92,4 @@ To see the documentation and configuration files for versions prior to 2.0 see < Shiny is registered trademarks of RStudio, Inc. The use of the trademarked terms Shiny through the images hosted on hub.docker.com has been granted by explicit permission of RStudio. Please review RStudio's trademark use policy and address inquiries about further distribution or other questions to permissions@rstudio.com. -Jupyter is distributed under the BSD 3-Clause license (Copyright (c) 2017, Project Jupyter Contributors) - -## Acknowledgements - -Thanks to Ajar Vashisth for helping me get started with Docker and Docker Compose. Thanks also to Raghav Prasad for setting up the > 2.0.0 versions of the docker images to start from Jupyter docker images. +Jupyter is distributed under the BSD 3-Clause license (Copyright (c) 2017, Project Jupyter Contributors) \ No newline at end of file diff --git a/install/rsm-msba-chromeos.md b/install/rsm-msba-chromeos.md index dbea7d3..3f1e14d 100644 --- a/install/rsm-msba-chromeos.md +++ b/install/rsm-msba-chromeos.md @@ -51,8 +51,6 @@ This step will clone and start up a script that will finalize the installation o -After running this command you will be able to start the docker container by typing `launch` from a terminal. - **Step 3**: Check that you can launch Jupyter and Rstudio You will know that the installation was successful if you can start Rstudio and Jupyter Lab. If you press 1 (+ Enter) Jupyter Lab should start up in your default web browser. If you are asked for login credentials, the **username is "jovyan"** and the **password is "jupyter"**. Have your browser remember the username and password so you won't be asked for it again.When you press 2 (+ Enter) in the terminal, Rstudio should start up in a new tab in your web browser. diff --git a/install/rsm-msba-linux.md b/install/rsm-msba-linux.md index d6e5167..3456624 100644 --- a/install/rsm-msba-linux.md +++ b/install/rsm-msba-linux.md @@ -57,8 +57,6 @@ Run the command below to start the launch script from the command line. ~/git/docker/launch-rsm-msba-arm-chromeos.sh -v ~; ``` -After running this command you will be able to start the docker container by typing `launch` from a terminal. - **Step 3**: Check that you can launch Jupyter and Rstudio You will know that the installation was successful if you can start Rstudio and Jupyter Lab. If you press 1 (+ Enter) Jupyter Lab should start up in your default web browser. If you are asked for login credentials, the **username is "jovyan"** and the **password is "jupyter"**. Have your browser remember the username and password so you won't be asked for it again. When you press 2 (+ Enter) in the terminal, Rstudio should start up in a new tab in your web browser. diff --git a/install/rsm-msba-macos-arm.md b/install/rsm-msba-macos-arm.md new file mode 100644 index 0000000..31fe101 --- /dev/null +++ b/install/rsm-msba-macos-arm.md @@ -0,0 +1,301 @@ +# Contents + +- [Installing the RSM-MSBA-ARM computing environment on macOS M1 or M2](#installing-the-rsm-msba-arm-computing-environment-on-macos-M1-or-M2) +- [Updating the RSM-MSBA-ARM computing environment on macOS M1 or M2](#updating-the-rsm-msba-arm-computing-environment-on-macos-M1-or-M2) +- [Using VS Code](#using-vs-code) +- [Connecting to postgresql](#connecting-to-postgresql) +- [Installing Python and R packages locally](#installing-python-and-r-packages-locally) +- [Committing changes to the computing environment](#committing-changes-to-the-computing-environment) +- [Cleanup](#cleanup) +- [Getting help](#getting-help) +- [Trouble shooting](#trouble-shooting) +- [Optional](#optional) + +## Installing the RSM-MSBA-ARM computing environment on macOS M1 or M2 + +Please follow the instructions below to install the rsm-msba-arm computing environment. It has Python, Jupyter Lab, R, Radiant, Rstudio, Postgres, Spark and various required packages pre-installed. The computing environment will be consistent across all students and faculty, easy to update, and also easy to remove if desired (i.e., there will *not* be dozens of pieces of software littered all over your computer). + +**Step 1**: Install docker from the link below and make sure it is running. You will know it is running if you see the icon below at the top-right of your screen. If the containers in the image are moving up and down docker hasn't finished starting up yet. + +![docker](figures/docker-icon.png) + +[download docker for macOS with an M1 or M2 (ARM) chip](https://desktop.docker.com/mac/stable/arm64/Docker.dmg) + +You should change the (maximum) resources docker is allowed to use on your system. We recommend you set this to approximately 50% of the maximum available on your system. + + + +You should also go to the "Advanced" tab and configure the installation of the Command Line Interface (CLI). Set it to "System" as shown in the screenshot below and click on the "Apply & Restart". + + + +Optional: If you are interested, the linked video gives a brief intro to what Docker is: https://www.youtube.com/watch?v=YFl2mCHdv24 + +**Step 2**: Open a terminal and copy-and-paste the code below + +You will need the macOS command line developer tools for the next steps. Follow the prompts until the software is installed. + +```bash +xcode-select --install; +``` + +**Step 3**: Now copy-and-paste the code below + +```bash +git clone https://github.com/radiant-rstats/docker.git ~/git/docker; +cp -p ~/git/docker/launch-rsm-msba-arm.sh ~/Desktop/launch-rsm-msba.command; +~/Desktop/launch-rsm-msba.command; +``` + +This step will clone and start up a script that will finalize the installation of the computing environment. The first time you run this script it will download the latest version of the computing environment which can take some time. Wait for the container to download and follow any prompts. Once the download is complete you should see a menu as in the screenshot below. + + + +The code above also copies the file `launch-rsm-msba-arm.sh` to `launch-rsm-msba.command` on your Desktop. You will be able to double-click this file to start the container again in the future. + +Run the command below to launch the docker container from the command line. + +```bash +~/git/docker/launch-rsm-msba-arm.sh -v ~; +``` + +**Step 4**: Check that you can launch Jupyter + +You will know that the installation was successful if you can start Jupyter Lab. When you press 1 (+ Enter) in the terminal, Jupyter Lab should start up in your default web browser. If you are asked for login credentials, the username is "jovyan" and the password is "jupyter". Have your browser remember the username and password so you won't be asked for it again. + +> Important: Always use q (+ Enter) to shutdown the computing environment + +**Jupyter**: + + + +To finalize the setup, open a terminal in Jupyter lab, press `q` and `Enter` if prompted, and then run the code below in the same terminal: + +```bash +setup; +exit; +``` + +Now open a new terminal in JupyterLab and you should see some icons + + + +## Updating the RSM-MSBA-ARM computing environment on macOS M1 or M2 + +To update the container use the launch script and press 6 (+ Enter). To update the launch script itself, press 7 (+ Enter). + + + +If for some reason you are having trouble updating either the container or the launch script open a terminal and copy-and-paste the code below. These commands will update the docker container, replace the old docker related scripts, and copy the latest version of the launch script to your Desktop. + +```bash +docker pull vnijs/rsm-msba-arm; +rm -rf ~/git/docker; +git clone https://github.com/radiant-rstats/docker.git ~/git/docker; +cp -p ~/git/docker/launch-rsm-msba-arm.sh ~/Desktop/launch-rsm-msba.command; +``` + +## Using VS Code + +Microsoft's open-source integrated development environment (IDE), VS Code or Visual Studio Code, was the most popular development environment according to a [Stack Overflow developer survey](https://survey.stackoverflow.co/2022#section-most-popular-technologies-integrated-development-environment). VS Code is widely used by Google developers and is the [default development environment at Facebook](https://www.zdnet.com/article/facebook-microsofts-visual-studio-code-is-now-our-default-development-platform/). + +VS Code can be installed from the link below and is an excellent, and very popular, editor for Python, R, and many other programming languages. + +https://code.visualstudio.com/download + +Run the code below from a terminal on macOS after installing VS Code to install relevant extensions: + +```bash +cd ~/git/docker/vscode; +./extension-install.sh; +cd -; +``` + +If you get a "code: command not found" error when trying to launch VS Code from a terminal, follow the instructions below to add VS Code to your path: + + + +To learn more about using VS Code to write python code see the links and comments below. + +- Python in VS Code +- VS Code Python Tutorial + +Note that you can use `Shift+Enter` to run the current line in a Python Interactive Window: + +- Executing Python Code in VS Code + +When writing and editing python code you will have access to tools for auto-completion, etc. Your code will also be auto-formatted every time you save it using the "black" formatter. + +- Editing Python in VS Code Python + +VS Code also gives you access to a debugger for your python code. For more information see the link below: + +- Debugging Python in VS Code Python + +You can even open and run Jupyter Notebooks in VS Code + +- Jupyter Notebooks in VS Code + +A major new feature in VS Code is the ability to use AI to help you write code. For more information see the links below: + +- VS Code Copilot +- VS Code AI + +## Connecting to postgresql + +The rsm-msba-arm container comes with postgresql installed. Once the container has been started, you can access postgresql in different ways. The easiest is to use `pgweb`. Start `pgweb` and enter the code below in the "Scheme" tab: + +```bash +postgresql://jovyan:postgres@127.0.0.1:8765/rsm-docker +``` + + + +To access postgresql from Jupyter Lab use the code below: + +```python +## connect to database +from sqlalchemy import create_engine, inspect +engine = create_engine('postgresql://jovyan:postgres@127.0.0.1:8765/rsm-docker') + +## show list of tables +inspector = inspect(engine) +inspector.get_table_names() +``` + +For a more extensive example using Python see: https://github.com/radiant-rstats/docker/blob/master/postgres/postgres-connect.ipynb + +### Trouble shooting + +If you cannot connect to postgresql it is most likely due to an issue with the docker volume that contains the data. The volume can become corrupted if the container is not properly stopped using `q + Enter` in the launch menu. To create a clean volume for postgres (1) stop the running container using `q + Enter`, (2) run the code below in a terminal, and (3) restart the container. If you are still having issues connecting to the postgresql server, please reach out for support through Piazza. + +```bash +docker volume rm pg_data +``` + +## Installing Python and R packages locally + +To install the latest version of R-packages you need, add the lines of code shown below to `~/.Rprofile` or copy-and-paste the lines into the Rstudio console. + +```r +if (Sys.info()["sysname"] == "Linux") { + options(repos = c( + RSPM = "https://packagemanager.posit.co/cran/__linux__/jammy/latest", + CRAN = "https://cloud.r-project.org" + )) +} else { + options(repos = c( + CRAN = "https://cloud.r-project.org" + )) +} +``` + +This will be done for you automatically if you run the `setup` command from a terminal inside the docker container. To install R packages that will persist after restarting the docker container, enter code like the below in Rstudio and follow any prompts. After doing this once, you can use `install.packages("some-other-package")` in the future. + +```r +fs::dir_create(Sys.getenv("R_LIBS_USER"), recurse = TRUE) +install.packages("fortunes", lib = Sys.getenv("R_LIBS_USER")) +``` + +To install Python modules that will **not** persist after restarting the docker container, enter code like the below from the terminal in Jupyter Lab: + +```bash +pip install pyasn1 +``` + +After installing a module you will have to restart any running Python kernels to `import` the module in your code. + +### Using pip to install python packages + +We recommend you use `pip` to install any additional packages you might need. For example, you can use the command below to install a new version of the `pyrsm` package that you will use regularly throughout the Rady MSBA program. Note that adding `--user` is important to ensure the package is still available after you restart the docker container + +```bash +pip install --user --upgrade pyrsm +``` + +### Removing locally installed packages + +To remove locally installed R packages press 8 (and Enter) in the launch menu and follow the prompts. To remove Python modules installed locally using `pip` press 9 (and Enter) in the launch menu + +## Committing changes to the computing environment + +By default re-starting the docker computing environment will remove any changes you made. This allows you to experiment freely, without having to worry about "breaking" things. However, there are times when you might want to keep changes. + +As shown in the previous section, you can install R and Python packages locally rather than in the container. These packages will still be available after a container restart. + +To install binary R packages for Ubuntu Linux you can use the command below. These packages will *not* be installed locally and would normally not be available after a restart. + +```bash +sudo apt update; +sudo apt install r-cran-ada; +``` + +Similarly, some R-packages have requirements that need to be installed in the container (e.g., the `rgdal` package). The following two linux packages would need to be installed from a terminal in the container as follows: + +```bash +sudo apt update; +sudo apt install libgdal-dev libproj-dev; +``` + +After completing the step above you can install the `rgdal` R-package locally using the following from Rstudio: + +`install.packages("rgdal", lib = Sys.getenv("R_LIBS_USER"))` + +To save (or commit) these changes so they *will* be present after a (container) restart type, for example, `c myimage` (+ Enter). This creates a new docker image with your changes and also a new launch script on your Desktop with the name `launch-rsm-msba-myimage.command` that you can use to launch your customized environment in the future. + +If you want to share your customized version of the container with others (e.g., team members) you can push it is to Docker Hub https://hub.docker.com by following the menu dialog after typing, e.g., `c myimage` (+ Enter). To create an account on Docker Hub go to https://hub.docker.com/signup. + +If you want to remove specific images from your computer run the commands below from a (bash) terminal. The first command generates a list of the images you have available. + +`docker image ls;` + +Select the IMAGE ID for the image you want to remove, e.g., `42b88eb6adf8`, and then run the following command with the correct image id: + +`docker rmi 42b88eb6adf8;` + +For additional resources on developing docker images see the links below: + +- +- + +## Cleanup + +To remove any prior Rstudio sessions and locally installed R-packages, press 8 (+ Enter) in the launch menu. To remove locally installed Python modules press 9 (+ Enter) in the launch menu. + +> Note: It is also possible initiate the process of removing locally installed packages and settings from within the container. Open a terminal in Jupyter Lab or Rstudio and type `clean`. Then follow the prompts to indicate what needs to be removed. + +You should always stop the `rsm-msba-arm` docker container using `q` (+ Enter) in the launch menu. If you want a full cleanup and reset of the computational environment on your system, however, execute the following commands from a (bash) terminal to (1) remove prior R(studio) and Python packages, (2) remove all docker images, networks, and (data) volumes, and (3) 'pull' only the docker image you need (e.g., rsm-msba-arm): + +```bash +rm -rf ~/.rstudio; +rm -rf ~/.rsm-msba; +rm -rf ~/.local/share/jupyter +docker system prune --all --volumes --force; +docker pull vnijs/rsm-msba-arm; +``` + +## Getting help + +Please bookmark this page in your browser for easy access in the future. You can also access the documentation page for your OS by typing h (+ Enter) in the launch menu. Note that the launch script can also be started from the command line (i.e., a bash terminal) and has several important arguments: + +* `launch -t 3.0.0` ensures a specific version of the docker container is used. Suppose you used version 3.0.0 for a project. Running the launch script with `-t 3.0.0` from the command line will ensure your code still runs, without modification, years after you last touched it! +* `launch -v ~/rsm-msba` will treat the `~/rsm-msba` directory on the host system (i.e., your macOS computer) as the home directory in the docker container. This can be useful if you want to setup a particular directory that will house multiple projects +* `launch -d ~/project_1` will treat the `project_1` directory on the host system (i.e., your macOS computer) as the project home directory in the docker container. This is an additional level of isolation that can help ensure your work is reproducible in the future. This can be particularly useful in combination with the `-t` option as this will make a copy of the launch script with the appropriate `tag` or `version` already set. Simply double-click the script in the `project_1` directory and you will be back in the development environment you used when you completed the project +* `launch -s` show additional output in the terminal that can be useful to debug any problems +* `launch -h` prints the help shown in the screenshot below + + + +## Trouble shooting + +The only issues we have seen on macOS so far can be addressed by restarting docker and/or restarting your computer + +## Optional + +If you want to make your terminal look nicer and add syntax highlighting, auto-completion, etc. follow the install instructions linked below: + + + + + +To install a more feature-rich terminal for macOS see: {target="_blank"} diff --git a/install/rsm-msba-macos-m1.md b/install/rsm-msba-macos-m1.md index b3986e1..31fe101 100644 --- a/install/rsm-msba-macos-m1.md +++ b/install/rsm-msba-macos-m1.md @@ -59,8 +59,6 @@ Run the command below to launch the docker container from the command line. ~/git/docker/launch-rsm-msba-arm.sh -v ~; ``` -After running this command you will be able to start the docker container by typing `launch` from a terminal. - **Step 4**: Check that you can launch Jupyter You will know that the installation was successful if you can start Jupyter Lab. When you press 1 (+ Enter) in the terminal, Jupyter Lab should start up in your default web browser. If you are asked for login credentials, the username is "jovyan" and the password is "jupyter". Have your browser remember the username and password so you won't be asked for it again. diff --git a/install/rsm-msba-macos.md b/install/rsm-msba-macos.md index c25ae2d..a9f2453 100644 --- a/install/rsm-msba-macos.md +++ b/install/rsm-msba-macos.md @@ -55,8 +55,6 @@ Run the command below to launch the docker container from the command line. ~/git/docker/launch-rsm-msba-intel.sh -v ~; ``` -After running this command you will be able to start the docker container by typing `launch` from a terminal. - **Step 4**: Check that you can launch Jupyter and Rstudio You will know that the installation was successful if you can start Jupyter Lab. When you press 1 (+ Enter) in the terminal, Jupyter Lab should start up in your default web browser. If you are asked for login credentials, the username is "jovyan" and the password is "jupyter". Have your browser remember the username and password so you won't be asked for it again. When you press 2 (+ Enter) in the terminal, Rstudio should start up in a new tab in your web browser. diff --git a/install/rsm-msba-windows-arm.md b/install/rsm-msba-windows-arm.md index 5f9446b..1dfef7d 100644 --- a/install/rsm-msba-windows-arm.md +++ b/install/rsm-msba-windows-arm.md @@ -165,6 +165,25 @@ The created and launched script will finalize the installation of the computing + + **Trouble shooting** If you see `Base dir.: /root` as shown in the image below there was an issue creating a new user at the beginning of Step 4. diff --git a/install/rsm-msba-windows.md b/install/rsm-msba-windows.md index 1075a94..d58071c 100644 --- a/install/rsm-msba-windows.md +++ b/install/rsm-msba-windows.md @@ -171,6 +171,25 @@ The created and launched script will finalize the installation of the computing + + **Trouble shooting** If you see `Base dir.: /root` as shown in the image below there was an issue creating a new user at the beginning of Step 4. diff --git a/rsm-msba-arm/Dockerfile b/rsm-msba-arm/Dockerfile index c01e47c..8038764 100644 --- a/rsm-msba-arm/Dockerfile +++ b/rsm-msba-arm/Dockerfile @@ -32,6 +32,7 @@ RUN apt-get update -qq && apt-get -y --no-install-recommends install \ ant \ ca-certificates-java \ lsof \ + rename \ pipx \ && apt-get clean \ && update-ca-certificates -f; diff --git a/rsm-msba-intel/Dockerfile b/rsm-msba-intel/Dockerfile index 8d0b42c..4d033b5 100644 --- a/rsm-msba-intel/Dockerfile +++ b/rsm-msba-intel/Dockerfile @@ -32,6 +32,7 @@ RUN apt-get update -qq && apt-get -y --no-install-recommends install \ ant \ ca-certificates-java \ lsof \ + rename \ pipx \ && apt-get clean \ && update-ca-certificates -f;