Release 2024.1.1

.pre-commit-config.yaml

@@ -51,19 +51,19 @@ repos:
 
   # python
   - repo: https://github.com/psf/black
-    rev: 23.10.1
+    rev: 23.12.1
     hooks:
       - id: black
         args: ["--line-length=88", "--exclude=/src/_nebari/template/"]
 
   - repo: https://github.com/astral-sh/ruff-pre-commit
-    rev: v0.1.4
+    rev: v0.1.9
     hooks:
       - id: ruff
         args: ["--fix"]
 
   - repo: https://github.com/pycqa/isort
-    rev: 5.12.0
+    rev: 5.13.2
     hooks:
       - id: isort
         name: isort
@@ -73,7 +73,7 @@ repos:
 
   # terraform
   - repo: https://github.com/antonbabenko/pre-commit-terraform
-    rev: v1.83.5
+    rev: v1.86.0
     hooks:
       - id: terraform_fmt
         args:

RELEASE.md

@@ -11,6 +11,75 @@ This file is copied to nebari-dev/nebari-docs using a GitHub Action. -->
 
 ## Upcoming Release
 
+## Release 2024.1.1 - January 17, 2024
+
+### Feature changes and enhancements
+
+* Upgrade conda-store to latest version 2024.1.1
+* Add Jhub-Apps
+* Add Jupyterlab-pioneer
+* Minor improvements and bug fixes
+
+### Breaking Changes
+
+> WARNING: jupyterlab-videochat, retrolab, jupyter-tensorboard, jupyterlab-conda-store and jupyter-nvdashboard are no longer supported in Nebari version and will be uninstalled."
+
+### What's Changed
+
+* [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in https://github.com/nebari-dev/nebari/pull/2176
+* Fix logic for dns lookup. by @pt247 in https://github.com/nebari-dev/nebari/pull/2166
+* Integrate JupyterHub App Launcher into Nebari by @aktech in https://github.com/nebari-dev/nebari/pull/2185
+* Pass in permissions boundary to k8s module by @aktech in https://github.com/nebari-dev/nebari/pull/2153
+* Add jupyterlab-pioneer by @aktech in https://github.com/nebari-dev/nebari/pull/2127
+* JHub Apps: Filter conda envs by user by @aktech in https://github.com/nebari-dev/nebari/pull/2187
+* update upgrade command by @dcmcand in https://github.com/nebari-dev/nebari/pull/2198
+* Remove JupyterLab from services list by @aktech in https://github.com/nebari-dev/nebari/pull/2189
+* Adding fields to ignore within keycloak_realm by @costrouc in https://github.com/nebari-dev/nebari/pull/2200
+* Add Nebari menu item configuration. by @marcelovilla in https://github.com/nebari-dev/nebari/pull/2196
+* Disable "Newer update available" popup as default setting by @viniciusdc in https://github.com/nebari-dev/nebari/pull/2192
+* Block usage of pip inside jupyterlab  by @viniciusdc in https://github.com/nebari-dev/nebari/pull/2191
+* Return all environments instead of just those under the user's namespace for jhub-apps by @marcelovilla in https://github.com/nebari-dev/nebari/pull/2206
+* Adding a temporary writable directory for conda-store server /home/conda by @costrouc in https://github.com/nebari-dev/nebari/pull/2209
+* Add demo repositories mechanism to populate user's space by @viniciusdc in https://github.com/nebari-dev/nebari/pull/2207
+* update nebari_workflow_controller and conda_store tags to test rc by @dcmcand in https://github.com/nebari-dev/nebari/pull/2210
+* 2023.12.1 release notes by @dcmcand in https://github.com/nebari-dev/nebari/pull/2211
+* Make it so that jhub-apps default theme doesn't override by @costrouc in https://github.com/nebari-dev/nebari/pull/2213
+* Adding additional theme variables to jupyterhub theme config by @costrouc in https://github.com/nebari-dev/nebari/pull/2215
+* updates Current Release to 2024.1.1 by @dcmcand in https://github.com/nebari-dev/nebari/pull/2227
+
+
+**Full Changelog**: https://github.com/nebari-dev/nebari/compare/2023.12.1...2024.1.1
+
+## Release 2023.12.1 - December 15, 2023
+
+### Feature changes and enhancements
+
+* Upgrade conda-store to latest version 2023.10.1
+* Minor improvements and bug fixes
+
+### Breaking Changes
+
+> WARNING: Prefect, ClearML and kbatch were removed in this release and upgrading to this version will result in all of them being uninstalled.
+
+### What's Changed
+* BUG: fix incorrect config override #2086 by @fangchenli in https://github.com/nebari-dev/nebari/pull/2087
+* ENH: add AWS IAM permissions_boundary option #2078 by @fangchenli in https://github.com/nebari-dev/nebari/pull/2082
+* CI: cleanup local integration workflow by @fangchenli in https://github.com/nebari-dev/nebari/pull/2079
+* ENH: check missing GCP services by @fangchenli in https://github.com/nebari-dev/nebari/pull/2036
+* ENH: use packaging for version parsing, add unit tests by @fangchenli in https://github.com/nebari-dev/nebari/pull/2048
+* ENH: specify required field when retrieving available gcp regions by @fangchenli in https://github.com/nebari-dev/nebari/pull/2033
+* Upgrade conda-store to 2023.10.1 by @iameskild in https://github.com/nebari-dev/nebari/pull/2092
+* Add upgrade command for 2023.11.1 by @iameskild in https://github.com/nebari-dev/nebari/pull/2103
+* CLN: cleanup typing and typing import in init by @fangchenli in https://github.com/nebari-dev/nebari/pull/2107
+* Remove kbatch, prefect and clearml by @iameskild in https://github.com/nebari-dev/nebari/pull/2101
+* Fix integration tests, helm-validate script by @iameskild in https://github.com/nebari-dev/nebari/pull/2102
+* Re-enable AWS tags support by @iameskild in https://github.com/nebari-dev/nebari/pull/2096
+* Update upgrade instructions for 2023.11.1 by @iameskild in https://github.com/nebari-dev/nebari/pull/2112
+* Update nebari-git env pins by by @iameskild in https://github.com/nebari-dev/nebari/pull/2113
+* Update release notes for 2023.11.1 by @iameskild in https://github.com/nebari-dev/nebari/pull/2114
+
+
+**Full Changelog**: https://github.com/nebari-dev/nebari/compare/2023.11.1...2023.12.1
 
 ## Release 2023.11.1 - November 15, 2023
 

src/_nebari/constants.py

@@ -1,4 +1,4 @@
-CURRENT_RELEASE = "2023.11.1"
+CURRENT_RELEASE = "2024.1.1"
 
 # NOTE: Terraform cannot be upgraded further due to Hashicorp licensing changes
 # implemented in August 2023.
@@ -13,9 +13,9 @@
 
 DEFAULT_NEBARI_DASK_VERSION = CURRENT_RELEASE
 DEFAULT_NEBARI_IMAGE_TAG = CURRENT_RELEASE
-DEFAULT_NEBARI_WORKFLOW_CONTROLLER_IMAGE_TAG = "2023.7.2"
+DEFAULT_NEBARI_WORKFLOW_CONTROLLER_IMAGE_TAG = "2024.1.1"
 
-DEFAULT_CONDA_STORE_IMAGE_TAG = "2023.10.1"
+DEFAULT_CONDA_STORE_IMAGE_TAG = "2024.1.1"
 
 LATEST_SUPPORTED_PYTHON_VERSION = "3.10"
 

src/_nebari/stages/infrastructure/template/aws/main.tf

@@ -94,4 +94,5 @@ module "kubernetes" {
 
   endpoint_private_access = var.eks_endpoint_private_access
   public_access_cidrs     = var.eks_public_access_cidrs
+  permissions_boundary    = var.permissions_boundary
 }

src/_nebari/stages/infrastructure/template/local/main.tf

@@ -48,7 +48,7 @@ resource "kind_cluster" "default" {
 
     node {
       role  = "general"
-      image = "kindest/node:v1.21.10"
+      image = "kindest/node:v1.23.13"
     }
   }
 }

src/_nebari/stages/kubernetes_ingress/__init__.py

@@ -75,15 +75,15 @@ def _attempt_dns_lookup(
     ):
         for i in range(num_attempts):
             try:
-                resolved_ip = socket.gethostbyname(domain_name)
-                if resolved_ip == ip:
+                _, _, resolved_ips = socket.gethostbyname_ex(domain_name)
+                if ip in resolved_ips:
                     print(
-                        f"DNS configured domain={domain_name} matches ingress ip={ip}"
+                        f"DNS configured domain={domain_name} matches ingress ips={ip}"
                     )
                     return True
                 else:
                     print(
-                        f"Attempt {i+1} polling DNS domain={domain_name} does not match ip={ip} instead got {resolved_ip}"
+                        f"Attempt {i+1} polling DNS domain={domain_name} does not match ip={ip} instead got {resolved_ips}"
                     )
             except socket.gaierror:
                 print(

src/_nebari/stages/kubernetes_keycloak/template/modules/kubernetes/keycloak-helm/values.yaml

@@ -33,7 +33,7 @@ extraInitContainers: |
           chown 1000:1000 /data/keycloak-metrics-spi-2.5.3.jar &&
           chmod 777 /data/keycloak-metrics-spi-2.5.3.jar
       fi
-    image: busybox:1.31
+    image: busybox:1.36
     name: initialize-spi-metrics-jar
     securityContext:
       runAsUser: 0

src/_nebari/stages/kubernetes_keycloak_configuration/template/main.tf

@@ -24,6 +24,48 @@ resource "keycloak_realm" "main" {
   web_authn_passwordless_policy {
   }
 
+  lifecycle {
+    ignore_changes = [
+      # We want user to have control over attributes we are not managing
+      # If attribute is added above remove it from this list
+      # https://registry.terraform.io/providers/mrparkers/keycloak/latest/docs/resources/realm
+      attributes,
+      registration_allowed,
+      registration_email_as_username,
+      edit_username_allowed,
+      reset_password_allowed,
+      remember_me,
+      verify_email,
+      login_with_email_allowed,
+      login_theme,
+      account_theme,
+      admin_theme,
+      email_theme,
+      sso_session_idle_timeout,
+      sso_session_max_lifespan,
+      sso_session_idle_timeout_remember_me,
+      sso_session_max_lifespan_remember_me,
+      offline_session_idle_timeout,
+      offline_session_max_lifespan,
+      access_token_lifespan,
+      access_token_lifespan_for_implicit_flow,
+      access_code_lifespan,
+      access_code_lifespan_login,
+      access_code_lifespan_user_action,
+      action_token_generated_by_user_lifespan,
+      action_token_generated_by_admin_lifespan,
+      oauth2_device_code_lifespan,
+      oauth2_device_polling_interval,
+      smtp_server,
+      internationalization,
+      security_defenses,
+      password_policy,
+      otp_policy,
+      default_default_client_scopes,
+      default_optional_client_scopes,
+    ]
+  }
+
 }
 
 resource "keycloak_group" "groups" {

src/_nebari/stages/kubernetes_services/__init__.py

@@ -54,12 +54,18 @@ class JupyterHubTheme(schema.Base):
     welcome: str = """Welcome! Learn about Nebari's features and configurations in <a href="https://www.nebari.dev/docs">the documentation</a>. If you have any questions or feedback, reach the team on <a href="https://www.nebari.dev/docs/community#getting-support">Nebari's support forums</a>."""
     logo: str = "https://raw.githubusercontent.com/nebari-dev/nebari-design/main/logo-mark/horizontal/Nebari-Logo-Horizontal-Lockup-White-text.svg"
     primary_color: str = "#4f4173"
+    primary_color_dark: str = "#4f4173"
     secondary_color: str = "#957da6"
+    secondary_color_dark: str = "#957da6"
     accent_color: str = "#32C574"
+    accent_color_dark: str = "#32C574"
     text_color: str = "#111111"
     h1_color: str = "#652e8e"
     h2_color: str = "#652e8e"
     version: str = f"v{__version__}"
+    navbar_color: str = "#1c1d26"
+    navbar_text_color: str = "#f1f1f6"
+    navbar_hover_color: str = "#db96f3"
     display_version: str = "True"  # limitation of theme everything is a str
 
 
@@ -188,10 +194,23 @@ class ArgoWorkflows(schema.Base):
     nebari_workflow_controller: NebariWorkflowController = NebariWorkflowController()
 
 
+class JHubApps(schema.Base):
+    enabled: bool = False
+
+
 class Monitoring(schema.Base):
     enabled: bool = True
 
 
+class JupyterLabPioneer(schema.Base):
+    enabled: bool = False
+    log_format: typing.Optional[str] = None
+
+
+class Telemetry(schema.Base):
+    jupyterlab_pioneer: JupyterLabPioneer = JupyterLabPioneer()
+
+
 class JupyterHub(schema.Base):
     overrides: typing.Dict = {}
 
@@ -208,6 +227,7 @@ class IdleCuller(schema.Base):
 
 class JupyterLab(schema.Base):
     idle_culler: IdleCuller = IdleCuller()
+    initial_repositories: typing.List[typing.Dict[str, str]] = []
 
 
 class InputSchema(schema.Base):
@@ -280,8 +300,10 @@ class InputSchema(schema.Base):
     conda_store: CondaStore = CondaStore()
     argo_workflows: ArgoWorkflows = ArgoWorkflows()
     monitoring: Monitoring = Monitoring()
+    telemetry: Telemetry = Telemetry()
     jupyterhub: JupyterHub = JupyterHub()
     jupyterlab: JupyterLab = JupyterLab()
+    jhub_apps: JHubApps = JHubApps()
 
 
 class OutputSchema(schema.Base):
@@ -329,6 +351,7 @@ class CondaStoreInputVars(schema.Base):
 class JupyterhubInputVars(schema.Base):
     jupyterhub_theme: Dict[str, Any] = Field(alias="jupyterhub-theme")
     jupyterlab_image: ImageNameTag = Field(alias="jupyterlab-image")
+    initial_repositories: str = Field(alias="initial-repositories")
     jupyterhub_overrides: List[str] = Field(alias="jupyterhub-overrides")
     jupyterhub_stared_storage: str = Field(alias="jupyterhub-shared-storage")
     jupyterhub_shared_endpoint: str = Field(None, alias="jupyterhub-shared-endpoint")
@@ -337,6 +360,7 @@ class JupyterhubInputVars(schema.Base):
     jupyterhub_hub_extraEnv: str = Field(alias="jupyterhub-hub-extraEnv")
     idle_culler_settings: Dict[str, Any] = Field(alias="idle-culler-settings")
     argo_workflows_enabled: bool = Field(alias="argo-workflows-enabled")
+    jhub_apps_enabled: bool = Field(alias="jhub-apps-enabled")
 
 
 class DaskGatewayInputVars(schema.Base):
@@ -348,6 +372,13 @@ class MonitoringInputVars(schema.Base):
     monitoring_enabled: bool = Field(alias="monitoring-enabled")
 
 
+class TelemetryInputVars(schema.Base):
+    jupyterlab_pioneer_enabled: bool = Field(alias="jupyterlab-pioneer-enabled")
+    jupyterlab_pioneer_log_format: typing.Optional[str] = Field(
+        alias="jupyterlab-pioneer-log-format"
+    )
+
+
 class ArgoWorkflowsInputVars(schema.Base):
     argo_workflows_enabled: bool = Field(alias="argo-workflows-enabled")
     argo_workflows_overrides: List[str] = Field(alias="argo-workflows-overrides")
@@ -401,6 +432,12 @@ def input_vars(self, stage_outputs: Dict[str, Dict[str, Any]]):
                     "*/*": ["viewer"],
                 },
             },
+            "jhub-apps": {
+                "primary_namespace": "",
+                "role_bindings": {
+                    "*/*": ["viewer"],
+                },
+            },
         }
 
         # Compound any logout URLs from extensions so they are are logged out in succession
@@ -458,6 +495,8 @@ def input_vars(self, stage_outputs: Dict[str, Dict[str, Any]]):
             ),
             idle_culler_settings=self.config.jupyterlab.idle_culler.dict(),
             argo_workflows_enabled=self.config.argo_workflows.enabled,
+            jhub_apps_enabled=self.config.jhub_apps.enabled,
+            initial_repositories=str(self.config.jupyterlab.initial_repositories),
         )
 
         dask_gateway_vars = DaskGatewayInputVars(
@@ -471,6 +510,11 @@ def input_vars(self, stage_outputs: Dict[str, Dict[str, Any]]):
             monitoring_enabled=self.config.monitoring.enabled,
         )
 
+        telemetry_vars = TelemetryInputVars(
+            jupyterlab_pioneer_enabled=self.config.telemetry.jupyterlab_pioneer.enabled,
+            jupyterlab_pioneer_log_format=self.config.telemetry.jupyterlab_pioneer.log_format,
+        )
+
         argo_workflows_vars = ArgoWorkflowsInputVars(
             argo_workflows_enabled=self.config.argo_workflows.enabled,
             argo_workflows_overrides=[json.dumps(self.config.argo_workflows.overrides)],
@@ -486,6 +530,7 @@ def input_vars(self, stage_outputs: Dict[str, Dict[str, Any]]):
             **dask_gateway_vars.dict(by_alias=True),
             **monitoring_vars.dict(by_alias=True),
             **argo_workflows_vars.dict(by_alias=True),
+            **telemetry_vars.dict(by_alias=True),
         }
 
     def check(

src/_nebari/stages/kubernetes_services/template/jupyterhub.tf

@@ -39,6 +39,11 @@ variable "jupyterlab-profiles" {
   description = "JupyterHub profiles to expose to user"
 }
 
+variable "initial-repositories" {
+  description = "Map of folder location and git repo url to clone"
+  type        = string
+}
+
 variable "jupyterhub-hub-extraEnv" {
   description = "Extracted overrides to merge with jupyterhub.hub.extraEnv"
   type        = string
@@ -99,6 +104,8 @@ module "jupyterhub" {
   argo-workflows-enabled                             = var.argo-workflows-enabled
   conda-store-argo-workflows-jupyter-scheduler-token = module.kubernetes-conda-store-server.service-tokens.argo-workflows-jupyter-scheduler
   conda-store-service-name                           = module.kubernetes-conda-store-server.service_name
+  conda-store-jhub-apps-token                        = module.kubernetes-conda-store-server.service-tokens.jhub-apps
+  jhub-apps-enabled                                  = var.jhub-apps-enabled
 
   extra-mounts = {
     "/etc/dask" = {
@@ -127,5 +134,8 @@ module "jupyterhub" {
   jupyterhub-hub-extraEnv        = var.jupyterhub-hub-extraEnv
 
   idle-culler-settings = var.idle-culler-settings
+  initial-repositories = var.initial-repositories
 
+  jupyterlab-pioneer-enabled    = var.jupyterlab-pioneer-enabled
+  jupyterlab-pioneer-log-format = var.jupyterlab-pioneer-log-format
 }