Add docs for deploying QHub to existing EKS cluster

[iameskild]

Fixes | Closes | Resolves #942

Please remove anything marked as optional that you don't need to fill in. Choose one of the keywords preceding to refer to the issue this PR solves, followed by the issue number (e.g Fixes # 666). If there is no issue, remove the line. Remove this note after reading.

Changes:

Add documentation on how to deploy QHub to an existing EKS cluster.

Types of changes

What types of changes does your code introduce?

Put an x in the boxes that apply

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds a feature)
• Breaking change (fix or feature that would cause existing features to not work as expected)
• Documentation Update
• Code style update (formatting, renaming)
• Refactoring (no functional changes, no API changes)
• Build related changes
• Other (please describe):

Testing

Requires testing

• Yes
• No

In case you checked yes, did you write tests?

• Yes
• No

Further comments (optional)

If this is a relatively large or complex change, kick off the discussion by explaining why you chose the solution you did and what alternatives you considered and more.

[phobson]

Hey folks,

I've reviewed the new docs and they clarify a few things, but I'm personally still having trouble deploying to my org's existing server. A few things to keep in mind is that 1) I'm not great with the web and 2) really just getting started with distributed computing. I realize the Qhub docs can't explain everything, so if answering any (or all) of the questions below are outside of the scope of Qhub's docs, I would completely understand.

That said, I'm left unsure about the following things:

1. When I'm configuring and deploying Qhub, should I be would working on my local machine (laptop) or should I be working in a terminal attached to the cluster? This question mainly comes from confusion around the "aws" vs "local" value of the provider? What is "local" here?
2. Cloudflare's role is unclear to me. This is largely because my org and I aren't necessarily looking for a Jupyter Hub interface. We just want to be able to use a YML file to provision/configure our workers in the cluster
3. Beyond the Qhub config file, what is required to be in the Github repository? I suppose the github actions need to be configured as well, but do we have to have a webservice in the repo as well?
4. Lastly, since these docs are so fresh, I've been working from a development installation of Qhub. Any issues with that?

Again, I'm basically starting from scratch here, so please take the questions mostly as things I'm asking myself to figure out and not as burden on the Qhub team.

[iameskild]

Hey @phobson, thanks for the reaching out. I'll see if I can answer some of these questions:

1. The term local is a bit misleading but refers to an "existing" kubernetes cluster (labeled local for historical reasons, updating this label might be worthwhile in the future). So long as you have set the appropriate kubectl context when working with your AWS kubernetes (EKS) cluster and qhub, you should be fine to interact with the cluster from your laptop. This does require that you have kubectl installed. If you have an existing AWS kubernetes cluster you would like to deploy QHub to, then local is what you want.
2. Cloudflare is a DNS service, creating a record so that your qhub domain can be resolved. QHub deploys a Jupyterhub (with many additional features such as dask integration, dash-boarding and more) so I'm not entirely sure what you mean by use the YML to configure/provision our workers. This YML is used to provision users so they can login to a JupyterLab session (authenticated and connected to JupyterHub). Be warned that QHub is moving away from provisioning users in this qhub-config.yaml and replacing it with keycloak.
3. The github repo for your deployment is not necessary, but a recommended way of tracking changes to your QHub "infrastructure". An admin can make changes to the qhub-config.yaml, merge those changes into the repo and whereby those changes will automatically be deployed.
4. If you're deploying from main (ie a "dev" install), then you will be picking up the switch to keycloak (not yet released). I would recommend deploying from a stable version, 0.3.13 (or 0.3.14, just released).

I hope that sheds a little light on some of these questions. We are definitely interested in ways we can improve our documentation, so if there are specific areas that are unclear or misleading, feel free to let me know or open an issue. Thanks again :)

[phobson]

Thanks @iameskild. One last thing that I meant to ask above, but omitted is this: How externally exposed does the cluster need to be for the Cloudfare registration to work? My guess is that properly configuring the kubectl context handles this. But, again, I'm pretty green here.

Thanks for the note about sticking to releases and the incoming keycloak change.

[tylerpotts]

LGTM

[tylerpotts]

@phobson Configuring the kubectl context is more about authenticating so you can interact with the kubernetes cluster via the CLI.

The only ip that needs to be exposed is the qhub-traefik-ingress service, which is the only cluster resource to have an external IP address. This service acts as a proxy for all requests to the cluster.

I'm going to merge this PR which will close it, but that's certainly no indication that you can't ask further questions. If you have any other comments please feel free to open an issue