Add persistent volume configuration to jupyter

[wabscale]

I was having issues with the jupyter where any notebooks saved were lost any time the pod kicked over. I've added some very simple configuration to add a persistent volume claim to the jupyter instance (which is disabled by default).

jupyter:
  persistent:
    enable: false
    storageClass: "default"
    size: 1Gi
    path: /home/jovyan/persistent

I've made it so that you can specify where the volume gets mounted, the size, and the storage class. When the storage class name is "default", it will use whatever the default storage class is.

[jacobtomlinson]

Setting permissions to 777 is generally a bad idea. You may find 755 is more appropriate.

However could you provide a little explaination why this is necessary at all?

[wabscale]

The storage class that I am using is longhorn. The volumes they give you are owned by root by default. That init container sets the permissions so that the jovyan user can read and write to the PV.

I'm not aware of any storage classes that let you explicitly set the default ownership. I will add an option so its easy to disable this init container if desired.

[jacobtomlinson]

I would recommend making this just /home/jovyan

[wabscale]

I was trying to explicitly not overwrite the examples directory with the pv. I see your point though, it would be much clearer to the user if the entire home directory is whats saved.

[jacobtomlinson]

The examples are created at runtime. So this should've be a problem.

[wabscale]

When I make the mount path /home/jovyan the examples do not show up.

Checking the Dockerfile from the dask-docker repo shows they are added at build time.

https://github.com/dask/dask-docker/blob/master/notebook/Dockerfile#L47

[jacobtomlinson]

Oh interesting. I must be thinking of the Pangeo helm chart.

[wabscale]

I'm wondering how you want to handle this. I can submit a PR to the docker-helm repo to copy the examples to another directory (say /usr/share/doc/dask-examples). That way they would be independent of the PV, and always available.

[wabscale]

Hey @jacobtomlinson how would you wan tot handle this?

[jacobtomlinson]

I can submit a PR to the docker-helm repo to copy the examples to another directory (say /usr/share/doc/dask-examples).

The trouble with this is discoverability. Really they need to show up in the sidebar.

Perhaps adding them to that location and then adding a line to the prepare.sh script to create a symbolic link would work?

[jacobtomlinson]

This looks like a good change. Just a couple of comments.

[daddydrac]

I have a way to do this on AWS s3

[jacobtomlinson]

That's great @joehoeller! Would you mind sharing your method? Perhaps we could add it to the docs.

[daddydrac]

Yeah let’s connect. I also have another question in LinkedIn that I tagged you in.

…

On Wed, Dec 2, 2020 at 10:57 AM Jacob Tomlinson ***@***.***> wrote: That's great @joehoeller <https://github.com/joehoeller>! Would you mind sharing your method? Perhaps we could add it to the docs. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#78 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABHVQHF44HNJ6XL2QWFRZ2TSSZWY3ANCNFSM4OWFXZPQ> .

[jacobtomlinson]

Yeah I spotted the SO question. I've answered it.

Are you able to share things publicly here? It would be great for the community.

[daddydrac]

Persistent Volumes for Jupyter using RAPIDS and Dask with Amazon s3

The instructions for Dask are not obvious, however it does say to install on both worker and jupyter server nodes:

Inside values.yaml place this in worker and jupyter service:

    env:
      - name: EXTRA_PIP_PACKAGES
        value: "hybridcontents s3fs s3contents --use-feature=2020-resolver --upgrade"

Then, inside the path /dask/templates you will see a file called dask-jupyter-config.yaml, replace that code with everything that is in this configMap gist here: https://gist.github.com/joehoeller/063d055cdaf7d92455ee6d695ead8d0a

You will see these vars:

1. "access_key_id": AWS Access Key
2. "secret_access_key": AWS Secret Key
3. "bucket": The bucket name you created in AWS console or cli
4. "region_name": The region you are making the request from
5. "endpoint_url": This can vary from: https://s3.us-west-2.amazonaws.com, https://s3.us-gov-east-1.amazonaws.com, and for FIPS enabled security (will only accept connection if Jupyter is on https) https://s3-fips.us-gov-east-1.amazonaws.com.

[daddydrac]

@jacobtomlinson did this work for you? I noticed my version I forked internally has a slightly diff file structure.

[daddydrac]

@jacobtomlinson can i close this?