Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Kubernetes for local dev #118

Merged
merged 85 commits into from
Jul 18, 2023
Merged

Kubernetes for local dev #118

merged 85 commits into from
Jul 18, 2023

Commits on Jun 16, 2023

  1. Get volumes working in k3s

    Using Kubernetes from https://k3s.io/ for local development, we don't
    have a weekly-snapshots-retain-4 storage class, so we need local-path.
    Also, local-path doesn't allow ReadWriteMany -- but we don't need that
    anyway since all of our containers are running in the same pod, and
    therefore on the same node. ReadWriteMany is only required for volumes
    being mounted from different nodes.
    rmunn committed Jun 16, 2023
    Configuration menu
    Copy the full SHA
    74049bf View commit details
    Browse the repository at this point in the history
  2. Remove namespace from secrets for now

    I'll put the namespace back everywhere eventually. But for local dev, we
    don't (yet) need a namespace.
    rmunn committed Jun 16, 2023
    Configuration menu
    Copy the full SHA
    7174aad View commit details
    Browse the repository at this point in the history
  3. Fix "invalid base64" error in secrets.yaml

    The `data:` key tells Kubernetes these are base64-encoded values. If
    they are not encoded, the `stringData` key is correct.
    rmunn committed Jun 16, 2023
    Configuration menu
    Copy the full SHA
    dd62dc6 View commit details
    Browse the repository at this point in the history
  4. Populate Kubernetes dev secrets

    These are the same values found in the .env file checked into the repo,
    so this commit is not exposing any actual secrets, just the ones we use
    during local development.
    rmunn committed Jun 16, 2023
    Configuration menu
    Copy the full SHA
    e37a7b3 View commit details
    Browse the repository at this point in the history
  5. First pass at a Kubernetes setup for local dev

    This creates a container that will hold the Git repo and allow us to
    push to it. Next we'll set up the frontend and backend containers to
    watch that repo.
    rmunn committed Jun 16, 2023
    Configuration menu
    Copy the full SHA
    34a824b View commit details
    Browse the repository at this point in the history
  6. Add "localdev" task to get git container running

    The "localdev" task does everything needed to get a Git container up and
    running, but does not (yet) create a .ssh/config entry for you.
    rmunn committed Jun 16, 2023
    Configuration menu
    Copy the full SHA
    7adca7c View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    d8ec41a View commit details
    Browse the repository at this point in the history
  8. Use Skaffold instead of custom-built solution

    This will enable us to run `skaffold dev` and have a live-reloading
    version of our app running in Kubernetes.
    rmunn committed Jun 16, 2023
    Configuration menu
    Copy the full SHA
    c24372a View commit details
    Browse the repository at this point in the history
  9. Increase memory limits so Hasura isn't OOM killed

    The hasura container was being OOMKilled, so we'll try bumping its
    memory allocation and its hard limit.
    rmunn committed Jun 16, 2023
    Configuration menu
    Copy the full SHA
    e9d475c View commit details
    Browse the repository at this point in the history
  10. Fix CORS error in local k8s dev

    Long-term, we need to specify exactly which clients (e.g., the frontend
    box) are allowed to connect, but AllowAnyOrigin is good enough during
    dev work.
    rmunn committed Jun 16, 2023
    Configuration menu
    Copy the full SHA
    2bb077e View commit details
    Browse the repository at this point in the history
  11. Configuration menu
    Copy the full SHA
    08f5f88 View commit details
    Browse the repository at this point in the history
  12. Restore the k8s "languagedepot" namespace

    All k8s deployment files now use the "languagedepot" namespace. To use
    that namespace by default and not have to specify `-n lanugagedepot` all
    the time, add `namespace: languagedepot` to your ~/.kube/config file in
    the appropriate context stanza, just after the cluster: and user: lines.
    rmunn committed Jun 16, 2023
    Configuration menu
    Copy the full SHA
    351cb12 View commit details
    Browse the repository at this point in the history

Commits on Jun 19, 2023

  1. Fix login process - use Svelte-Kit load properly

    Now the login process will redirect to a server page, forcing a cookie
    reload so that the rest of the app will correctly get the user from the
    load() function in src/routes. This will mean that the user store should
    no longer be stale.
    rmunn committed Jun 19, 2023
    Configuration menu
    Copy the full SHA
    37032f5 View commit details
    Browse the repository at this point in the history
  2. Fix SHA1 hashing during SSR

    The `crypto.subtle` module is a browser-only module; if we're doing
    server-side rendering then we need the NodeJS equivalent.
    rmunn committed Jun 19, 2023
    Configuration menu
    Copy the full SHA
    9d3eb39 View commit details
    Browse the repository at this point in the history
  3. Add k8s deployment setup for docker-desktop

    Docker desktop's embedded Kubernetes doesn't have an ingress controller,
    so part of local k8s dev must include deploying one. It also doesn't
    have local-path storageClass, so we have to use hostpath.
    rmunn committed Jun 19, 2023
    Configuration menu
    Copy the full SHA
    c87d0a3 View commit details
    Browse the repository at this point in the history
  4. Work around issue with recent Node Docker images

    Recent docker images of Node 20 have an issue where node-gyp's
    postinstall step can fail because it's trying to replace a binary while
    it's still running. Moving to Node 18 Docker images, just for now, to
    avoid the problem.
    rmunn committed Jun 19, 2023
    Configuration menu
    Copy the full SHA
    f28baa0 View commit details
    Browse the repository at this point in the history
  5. Node 18 doesn't have webcrypto module

    Can't pre-hash the password if the page was SSR rendered, so as long as
    we're on Node 18, we need to turn off password prehashing.
    rmunn committed Jun 19, 2023
    Configuration menu
    Copy the full SHA
    f83ee84 View commit details
    Browse the repository at this point in the history
  6. Turn off https during local dev

    If there's a tls config block in the ingress deployment YAML, it sets
    HSTS for dev.languagedepot.org even though the TLS is only configured
    for staging.languagedepot.org, and that causes issues. In the future
    we'll use kustomize.yaml to configure this.
    rmunn committed Jun 19, 2023
    Configuration menu
    Copy the full SHA
    64d98e8 View commit details
    Browse the repository at this point in the history
  7. Fix BACKEND_HOST env var

    rmunn committed Jun 19, 2023
    Configuration menu
    Copy the full SHA
    a8d5ad6 View commit details
    Browse the repository at this point in the history

Commits on Jun 20, 2023

  1. Configuration menu
    Copy the full SHA
    b1c9efb View commit details
    Browse the repository at this point in the history

Commits on Jun 21, 2023

  1. More attempts at speeding up skaffold dev

    Port issues here
    rmunn committed Jun 21, 2023
    Configuration menu
    Copy the full SHA
    ee8ebc6 View commit details
    Browse the repository at this point in the history

Commits on Jun 23, 2023

  1. Attempt to split Hasura into separate deployment

    Currently not working: Hasura is saying "inconsistent metadata" and
    claiming that the tables Projects and Users don't exist.
    rmunn committed Jun 23, 2023
    Configuration menu
    Copy the full SHA
    2775ff8 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    e03e891 View commit details
    Browse the repository at this point in the history

Commits on Jun 26, 2023

  1. Configuration menu
    Copy the full SHA
    2187c87 View commit details
    Browse the repository at this point in the history
  2. Have hasura pod wait for backend to be healthy

    Since Hasura needs the backend to be up and healthy before it will
    accept its metadata, we change `wait-db` to `wait-lexbox` in the Hasura
    container (and the lexbox deployment no longer waits for Hasura, but
    instead waits for the database to be up and running).
    rmunn committed Jun 26, 2023
    Configuration menu
    Copy the full SHA
    8d99fb3 View commit details
    Browse the repository at this point in the history
  3. Rename hg-repos-pvc.yaml to just pvc.yaml

    This file will end up containing all PersistentVolumeClaims, so that it
    can be deployed separately from skaffold
    rmunn committed Jun 26, 2023
    Configuration menu
    Copy the full SHA
    d1767ef View commit details
    Browse the repository at this point in the history
  4. Move db PVC into pvc.yaml and create skaffold hook

    This way skaffold won't consider that it deployed the persistent volume
    claims in pvc.yaml and won't clean them up after skaffold dev.
    rmunn committed Jun 26, 2023
    Configuration menu
    Copy the full SHA
    d94f200 View commit details
    Browse the repository at this point in the history
  5. Add namespace to pvc.yaml

    So that if it's run first, it can still deploy into the languagedepot
    namespace. The duplicate namespace.yaml won't cause any issues.
    rmunn committed Jun 26, 2023
    Configuration menu
    Copy the full SHA
    69fa1db View commit details
    Browse the repository at this point in the history
  6. Attempt to get correct ports for local UI dev

    Work in progress
    rmunn committed Jun 26, 2023
    Configuration menu
    Copy the full SHA
    de86efb View commit details
    Browse the repository at this point in the history
  7. Increase memory for dev containers

    Backend and ui containers were getting OOM killed because the build
    process uses more memory than we had allocated. Eventually we'll want
    these values to be dynamically replaced by skaffold dev so that the
    memory allocations for the deployed app stay low.
    rmunn committed Jun 26, 2023
    Configuration menu
    Copy the full SHA
    2564637 View commit details
    Browse the repository at this point in the history
  8. Greatly increase memory request for dotnet watch

    The dotnet watch command is memory-hungry, and if the memory request is
    too low, the container gets OOM killed and then the new dotnet watch has
    to re-restore all the packages, which takes a while. Upping the memory
    limit means that dotnet watch is able to do its job properly.
    rmunn committed Jun 26, 2023
    Configuration menu
    Copy the full SHA
    52574d6 View commit details
    Browse the repository at this point in the history
  9. Avoid merge conflict

    rmunn committed Jun 26, 2023
    Configuration menu
    Copy the full SHA
    89c73c2 View commit details
    Browse the repository at this point in the history

Commits on Jun 27, 2023

  1. Pre-populate Sena-3 repo in k8s deployment

    Eventually we'll want this to be done only on dev and staging.
    rmunn committed Jun 27, 2023
    Configuration menu
    Copy the full SHA
    172c7c2 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    1942832 View commit details
    Browse the repository at this point in the history
  3. Deploy Hasura metadata as part of container image

    Instead of mounting the hasura-metadata volume, we'll just go back to
    inserting it as part of the Docker metadata. There's no reason to add
    the complexity of trying to sync the volume. If the metadata changes,
    we'll just let skaffold rebuild the Docker image.
    rmunn committed Jun 27, 2023
    Configuration menu
    Copy the full SHA
    e146b51 View commit details
    Browse the repository at this point in the history
  4. Remove db deployment from skaffold dev

    Now you will run `task k8s:db-up` (which may become just `task db-up`
    and will eventually be folded into `task infra-up`) to bring up the
    database, so that it will be persistent and will not get torn down when
    you exit skaffold dev. Then once you run `task db-up`, you can run
    `skaffold dev` and it will have the database ready for you.
    rmunn committed Jun 27, 2023
    Configuration menu
    Copy the full SHA
    28bc26c View commit details
    Browse the repository at this point in the history
  5. Hasura no longer needs to wait on backend

    Now that the database is persistent, the only time Hasura will give an
    error is the first time you spin up the whole system, when the database
    is completely empty. And even there, we may be able to avoid that by
    having an initContainer to check that `SELECT * FROM Projects LIMIT 1`
    returns one line rather than zero, which means that the database has
    been populated by the backend's SeedingData.SeedDatabase task.
    rmunn committed Jun 27, 2023
    Configuration menu
    Copy the full SHA
    6c0ea9b View commit details
    Browse the repository at this point in the history

Commits on Jun 28, 2023

  1. Attempt to use Kustomize for dev mode

    This allows us to run `skaffold dev` using docker-desktop, but leave the
    original config in place for staging.
    rmunn committed Jun 28, 2023
    Configuration menu
    Copy the full SHA
    78ada23 View commit details
    Browse the repository at this point in the history
  2. Split UI out into separate pod

    rmunn committed Jun 28, 2023
    Configuration menu
    Copy the full SHA
    67b1c6c View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    90b5550 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    aee8942 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    aa3f735 View commit details
    Browse the repository at this point in the history

Commits on Jun 29, 2023

  1. Update Database Startup procedures

    Updated database startup procedures to handle migration requests with
    added condition on development environment or command line argument.
    Introduced conditional autoExecuteMigrations in AddLexData to only
    AddHostedService when in development environment. Updated
    TestServicesFixtures to build service providers with migrations applied.
    Adjusted DbStartupService to throw exception when migrations are not
    executed, improving error visibility. Furthermore, added logging to
    track migration application duration.
    hahn-kev authored and rmunn committed Jun 29, 2023
    Configuration menu
    Copy the full SHA
    4cf5faa View commit details
    Browse the repository at this point in the history
  2. Add migrate-db task to Taskfile

    hahn-kev authored and rmunn committed Jun 29, 2023
    Configuration menu
    Copy the full SHA
    ced78b8 View commit details
    Browse the repository at this point in the history
  3. Move Hasura container back inside lexbox pod

    The lexbox pod will be running DB migrations on startup, and the Hasura
    container needs to see a current DB version (with all migrations run) at
    the time it starts up. So we put both containers in the same pod and use
    an initContainer to run the migrations.
    rmunn committed Jun 29, 2023
    Configuration menu
    Copy the full SHA
    212ad16 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    60ac444 View commit details
    Browse the repository at this point in the history

Commits on Jun 30, 2023

  1. Speed up Docker builds by caching packages

    Both the backend and frontend builds benefit greatly by caching NuGet or
    NPM packages from build to build, using Docker's built-in caching
    mechanism. You get warnings from pnpm that cross-device symlinks are not
    allowed (becuse the pnpm store is located on the cache volume), but it
    falls back to copying the NPM package files rather than symlinking them,
    which still saves a lot of time compared to downloading them.
    rmunn committed Jun 30, 2023
    Configuration menu
    Copy the full SHA
    0547001 View commit details
    Browse the repository at this point in the history
  2. Attempt to cache nuget packages in k8s dev

    This doesn't work: the custom storageClass does retain the data, but
    then the next PVC is considered a *different* claim and doesn't grab the
    retained data. We'll go for a simpler solution where we just cache the
    packages in the Docker image.
    rmunn committed Jun 30, 2023
    Configuration menu
    Copy the full SHA
    eecf9af View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    f7a88cb View commit details
    Browse the repository at this point in the history
  4. Reorder skaffold.yaml to be slightly more robust

    With the backend and frontend in slots 0 and 1 of the dockerfile
    definitions, it's far less likely to accidentally mess up which
    Dockerfile is associated with which container.
    rmunn committed Jun 30, 2023
    Configuration menu
    Copy the full SHA
    f09a80f View commit details
    Browse the repository at this point in the history
  5. Put backend before frontend in skaffold.yaml

    It makes a little more sense for the backend to be built first and the
    frontend second.
    rmunn committed Jun 30, 2023
    Configuration menu
    Copy the full SHA
    95c1bae View commit details
    Browse the repository at this point in the history

Commits on Jul 3, 2023

  1. Put staging deployment back how it was

    We've decided to run skaffold with `--cleanup=false` so that it won't
    delete the dpeloyments (and persistent volumes!) that it creates. That
    way our task of setting it up is much simpler.
    rmunn committed Jul 3, 2023
    Configuration menu
    Copy the full SHA
    bc031c6 View commit details
    Browse the repository at this point in the history

Commits on Jul 4, 2023

  1. Update skaffold to version 2.6.0

    Released last week, has some bugfixes that might help.
    rmunn committed Jul 4, 2023
    Configuration menu
    Copy the full SHA
    3c70976 View commit details
    Browse the repository at this point in the history
  2. Remove dev.languageforge.org from staging k8s proxy

    Staging config should only know about staging.languagedepot.org; we can
    add the dev.languageforge.org in a patch in dev/kustomize.yaml.
    rmunn committed Jul 4, 2023
    Configuration menu
    Copy the full SHA
    ac02e28 View commit details
    Browse the repository at this point in the history
  3. Node bug is fixed, revert to current Node version

    The Node bug only triggered on certain Linux kernel versions, and the
    latest Docker Desktop release has a version that doesn't trigger the bug
    any more. So we can go back to using the latest version of Node in the
    frontend build.
    rmunn committed Jul 4, 2023
    Configuration menu
    Copy the full SHA
    5737454 View commit details
    Browse the repository at this point in the history
  4. Have Vite listen on all interfaces

    Vite only listens on the loopback interface by default, which is fine
    for local development but doesn't work in a Kubernetes container, where
    the container is assigned its own cluster-local IP address that you're
    supposed to listen to. Telling vite to listen to 0.0.0.0 means "all
    available interfaces" which is what we want for k8s dev.
    rmunn committed Jul 4, 2023
    Configuration menu
    Copy the full SHA
    77e0155 View commit details
    Browse the repository at this point in the history
  5. Use correct port for ingress-lexbox communication

    On dev, the lexbox container is running on port 5158, **not** on port
    80. So the ingress controller couldn't talk to it, which explains all
    the 502 errors I'd been seeing until now.
    rmunn committed Jul 4, 2023
    Configuration menu
    Copy the full SHA
    c0f2ad4 View commit details
    Browse the repository at this point in the history

Commits on Jul 5, 2023

  1. Add skaffold profiles for running locally

    These profiles let you run the frontend, or the frontend *and* the
    backend, locally.
    rmunn committed Jul 5, 2023
    Configuration menu
    Copy the full SHA
    8d9fe0a View commit details
    Browse the repository at this point in the history
  2. Add DOTNET_URLS env to lexbox in dev mode

    Should avoid a problem where ASP.NET Core listens on the wrong interface
    rmunn committed Jul 5, 2023
    Configuration menu
    Copy the full SHA
    9ca7d76 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    bd4e905 View commit details
    Browse the repository at this point in the history
  4. Upgrade NuGet packages in Testing.csproj

    Some of the changes in develop created package conflicts, which are
    solved by upgrading the SIL.Core and SIL.Chorus packages.
    rmunn committed Jul 5, 2023
    Configuration menu
    Copy the full SHA
    9f0a038 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    4f19dfb View commit details
    Browse the repository at this point in the history

Commits on Jul 6, 2023

  1. only proxy from vite in local dev not in a container. Forward some mi…

    …ssing ports for doing dotnet locally.
    hahn-kev committed Jul 6, 2023
    Configuration menu
    Copy the full SHA
    1859901 View commit details
    Browse the repository at this point in the history
  2. fix goesToErrorPage error

    hahn-kev committed Jul 6, 2023
    Configuration menu
    Copy the full SHA
    8784cb5 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    311ff21 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    345cd3c View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    3d51cc4 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    560bacd View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    3a32b67 View commit details
    Browse the repository at this point in the history

Commits on Jul 7, 2023

  1. Minor README fix

    rmunn committed Jul 7, 2023
    Configuration menu
    Copy the full SHA
    e686f3e View commit details
    Browse the repository at this point in the history

Commits on Jul 17, 2023

  1. Merge branch 'develop' into chore/k8s-for-local-dev

    # Conflicts:
    #	deployment/base/lexbox-deployment.yaml
    hahn-kev committed Jul 17, 2023
    Configuration menu
    Copy the full SHA
    0a5f005 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    2cba39e View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    123cd1b View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    4d914cf View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    50f6279 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    7032010 View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    17775dd View commit details
    Browse the repository at this point in the history
  8. Configuration menu
    Copy the full SHA
    1658b5f View commit details
    Browse the repository at this point in the history
  9. Configuration menu
    Copy the full SHA
    0979a42 View commit details
    Browse the repository at this point in the history
  10. fix issue with deployments still trying to reference secrets. Fix mis…

    …sing env variable for hgweb
    hahn-kev committed Jul 17, 2023
    Configuration menu
    Copy the full SHA
    aa2ed76 View commit details
    Browse the repository at this point in the history

Commits on Jul 18, 2023

  1. Configuration menu
    Copy the full SHA
    fcf0d8c View commit details
    Browse the repository at this point in the history
  2. change dotnet port to 5158 instead of 80 across the board. Expose db …

    …via port forwarding in dev
    hahn-kev committed Jul 18, 2023
    Configuration menu
    Copy the full SHA
    56394d4 View commit details
    Browse the repository at this point in the history
  3. Rearrange README a bit, mention extra Windows setup

    Adding lines to /etc/hosts isn't actually needed on Linux, but is
    necessary on Windows, so we'll mention that. And we'll rearrage the
    README to put the extra setup steps immediately after the setup.
    rmunn committed Jul 18, 2023
    Configuration menu
    Copy the full SHA
    b21be5b View commit details
    Browse the repository at this point in the history
  4. Revert "Add "localdev" task to get git container running"

    This reverts commit 7adca7c.
    rmunn committed Jul 18, 2023
    Configuration menu
    Copy the full SHA
    f152551 View commit details
    Browse the repository at this point in the history
  5. Remove localdev-deployment.yaml

    This was an early attempt at getting k8s working for local dev,
    obsoleted by using skaffold.
    rmunn committed Jul 18, 2023
    Configuration menu
    Copy the full SHA
    73901a5 View commit details
    Browse the repository at this point in the history