Release ‐ Deployment Environments

The hub team uses several deployment environments to help with development, testing, and hosting the napari hub.

Release Process

Currently, the releases are made on as needed basis.

The napari hub's release process goes through different stages from development to production. QA is heavily involved in the process and is vital for ensuring a smooth release.

The flow is:

1. Any branches created with a dev- prefix are deployed to a dev environment. Currently, we are utilizing a single dev branch, dev-shared. The process for using this is as follows:
   • Create non-dev branches for all new PRs, e.g. instead of dev-new-feature it would be new-feature
   • Rebase dev-shared to your non-dev branch periodically for testing and/or demo-ing purposes.
     • When rebasing, first also ensure your non-dev branch is up to date with main.
     • Note: this is not a substitute for testing in your local development environment
   • Resolve conflicts as you normally would and coordinate with colleagues as needed
   • If necessary, creating an additional dev branch is still possible, but should not be our default
2. Any branches merged to main (after undergoing code review as a PR) are deployed to the staging environment.
   1. If there are any incoming bug fixes that need to be verified, it’s checked by QA in staging
3. Every 2 weeks, the hub team creates a new release which deploys to the production environment.
   1. The day before deployment, a code freeze is enforced to ensure stability in the next release to prod
   2. It also gives time for QA to do regression testing on staging without new changes coming in
   3. After deployment, QA will also do a regression test on production to verify there are no production specific issues

Cron Jobs

We have one cron job set up in CloudWatch for updating the cached plugin data. This job runs once every 5 minutes in production, once every 1 hour in staging, and once every 24 hours in development environment.

Production

The production environment for the napari hub represents what all public facing users will see when visiting the hub at https://napari-hub.org and https://api.napari-hub.org.

The architecture of the deployment can be found in these diagrams. Overall, the architecture consists of the following components:

• ECS container for the frontend
• AWS lambda for running backend code
• API gateway proxying backend code and caching APIs
• Dynamo for storing plugin data
• CloudWatch event for updating plugin data
• CloudFront for caching frontend assets and serving as public endpoint to the hub
• Terraform for managing infrastructure as code

The tech stack for the backend is very simple: it’s primarily a Python application running as a Flask server that exposes several API endpoints for API gateway. There’s also an AWS client library for interacting with Dynamo and CloudWatch events.

The tech stack for the frontend is a bit more involved, but generally it’s a TypeScript web application built with React.js and Next.js. React.js allows us to build the frontend as a composition of components, making it easier to build UIs on top of each other, and Next.js provides us with utilities such as hosting, fetching data, and rendering components on the server for improved SEO and mobile response time.

Staging

The staging environment for the napari hub represents what quality assurance testers and the hub development team see when visiting the hub at https://staging.napari-hub.org and https://api.staging.napari-hub.org.

The infrastructure and architecture for staging is very similar to production so that testing on staging is consistent with production.

Staging is the environment in which quality assurance testers can ensure that all developed features work correctly. This quality assurance process is tested in various web browsers for all the features in the staging environment. It’s also useful for developers to perform smoke tests and manual checks on the hub in a production-like environment after merging a PR.

If the features do not work as expected, then the quality assurance testers notify the developers of any bugs, and the GitHub issue would have the status of in progress.

By having a quality assurance process in the staging environment, the napari hub can ensure that everything works as expected before officially rolling out the features in the production environment for all public facing users to see.

Development

The development environment for the napari hub represents what all developers see when visiting the hub at https://[dev-branch-name]-frontend.dev.imaging.cziscience.com. The API URL for the backend can be retrieved from the AWS API Gateway console: https://[id].execute-api.us-west-2.amazonaws.com/[dev-branch-name].

The infrastructure and architecture is also the same as production and staging, but we have caching disabled for the CDN and APIs so that we don’t use those resources.

When developers create a branch prefixed by dev- in their IDE, it allows the branch to run the code using GitHub Actions, in which it generates the aforementioned url for the developers to see the actual changes of their features in the development environment. To remove a deployment, developers can delete the branch to cleanup and take down the development environment.

By creating a branch with the aforementioned prefix, it also generates a folder and workspace with the same name in Terraform in the happy-sci-imaging organization, respectively.

The development environment is very useful, in that it allows developers to simulate how the features would function in staging and production environments without the risk of accidentally breaking those environments. As such, the development environment allows the developers to quickly iterate on features until they are ready to be merged in, tested during the quality assurance process. and eventually be ready for all public facing users to see.