Skip to content

fix(helm): quote custom labels and fix default securityContext - #862

Merged
slin1237 merged 3 commits into
mainfrom
fix/helm-observability
Mar 22, 2026
Merged

slin1237 merged 3 commits into
mainfrom
fix/helm-observability

Conversation

@slin1237

@slin1237 slin1237 commented Mar 22, 2026 •

Copy link
Copy Markdown
Member

Description

Problem

Two issues discovered during observability testing of the Helm chart:

  1. Custom labels on ServiceMonitor and Grafana dashboard ConfigMap fail with --set: The toYaml helper renders numeric-like values without quoting (e.g. grafana_dashboard: 1 instead of grafana_dashboard: "1"). Kubernetes requires label values to be strings, so this causes unable to decode: json: cannot unmarshal number into Go struct field ObjectMeta.metadata.labels of type string.

  2. Default securityContext.runAsNonRoot: true is incompatible with the SMG image: The lightseekorg/smg image runs as root, causing CreateContainerConfigError on every fresh install with default values.

Solution

  • Replace toYaml label rendering with range + quote to ensure all custom label values are always emitted as strings, regardless of whether they look numeric
  • Change the default runAsNonRoot from true to false to match the image's runtime user

Changes

  • deploy/helm/smg/templates/grafana-dashboard.yaml: replace toYaml with range+quote for custom labels
  • deploy/helm/smg/templates/servicemonitor.yaml: same label quoting fix
  • deploy/helm/smg/values.yaml: set runAsNonRoot default to false

Test Plan

Full observability test suite run against helm-test-observability namespace on the dev cluster:

# Test Result
2 Deploy with all observability features PASS
3 Pod running (1/1 Ready) PASS
4 ServiceMonitor created with correct interval (15s) and labels (release: prometheus) PASS
5 Grafana dashboard ConfigMap created with grafana_dashboard: "1" label PASS
6 JSON logging (structured JSON log lines with timestamp, level, message) PASS
7 Service has both http (80) and metrics (29000) ports PASS
8 Metrics endpoint responds with Prometheus metrics at /metrics PASS
9 CLI args contain --log-json, --log-level debug, --prometheus-port 29000 PASS
10 Tracing dry-run renders --enable-trace and --otlp-traces-endpoint PASS

Namespace cleaned up after testing (confirmed NotFound).

Checklist
  • Helm chart updated
  • Tested on live cluster (helm-test-observability namespace)
  • Namespace cleaned up after testing
  • (Optional) Please join us on Slack #sig-smg to discuss, review, and merge PRs

Summary by CodeRabbit

  • New Features

    • Added prefix_hash as a new router load-balancing policy option.
  • Chores

    • Updated application version to 1.3.3.
    • Updated configuration flag names for improved consistency.
    • Modified cache eviction interval to 120 seconds.

What changed:
- deploy/helm/smg/templates/_helpers.tpl:
  - --eviction-interval-secs → --eviction-interval (field: eviction_interval, main.rs:169)
  - --postgres-pool-max → --postgres-pool-max-size (field: postgres_pool_max_size, main.rs:529)
  - --redis-pool-max → --redis-pool-max-size (field: redis_pool_max_size, main.rs:538)
  - --oracle-connect-descriptor → --oracle-dsn (field: oracle_dsn, main.rs:491)
  - --oracle-username → --oracle-user (field: oracle_user, main.rs:495)
- deploy/helm/smg/Chart.yaml:
  - appVersion: "0.7.0" → "1.3.3" (latest release tag v1.3.3)
- deploy/helm/smg/values.schema.json:
  - Added missing policy enum values: consistent_hashing, prefix_hash, bucket
    (all present in PolicyConfig enum in config/types.rs:247-320)
- deploy/helm/smg/values.yaml:
  - evictionIntervalSecs: 60 → 120 (matches default_value_t = 120, main.rs:168)
  - Updated policy comment to list all 8 valid values

Why:
The Helm chart generated incorrect CLI flags that would cause the SMG
binary to reject them at startup. Five flag names did not match the
actual clap field names in CliArgs, the eviction interval default was
wrong, the appVersion was stale, and three routing policies were
missing from the schema validation.

How:
Cross-referenced every --flag-name in _helpers.tpl against the CliArgs
struct fields in model_gateway/src/main.rs (clap converts field
snake_case to --kebab-case). Verified all default values in values.yaml
against default_value_t annotations. Checked PolicyConfig enum variants
in config/types.rs against values.schema.json. Validated with helm lint
and helm template rendering for all backend types (postgres, redis,
oracle).

Signed-off-by: Simo Lin <linsimo.mark@gmail.com>
The parse_policy function in main.rs (lines 795-826) lacks match arms
for consistent_hashing and bucket, causing them to silently fall back
to round_robin. Until the application code is fixed, excluding these
from the Helm schema prevents users from selecting non-functional
policies. prefix_hash is retained as it is correctly handled.

Addresses review feedback from Gemini Code Assist and CodeRabbit.

Signed-off-by: Simo Lin <linsimo.mark@gmail.com>
Custom labels on ServiceMonitor and Grafana dashboard ConfigMap were
rendered via toYaml, which does not quote numeric-like values (e.g.
grafana_dashboard: 1). Kubernetes rejects non-string label values,
causing installation failures when users pass --set with numeric labels.

Changed label rendering to use range with explicit quoting so that
values like "1" are always emitted as strings.

Also changed the default securityContext.runAsNonRoot from true to
false because the lightseekorg/smg image runs as root, which caused
CreateContainerConfigError on every fresh install.

What changed:
- deploy/helm/smg/templates/grafana-dashboard.yaml: replace toYaml
  with range+quote for custom labels
- deploy/helm/smg/templates/servicemonitor.yaml: same label quoting fix
- deploy/helm/smg/values.yaml: set runAsNonRoot default to false

Signed-off-by: Simon Olin <simon@lightseek.org>
Signed-off-by: Simo Lin <linsimo.mark@gmail.com>
@slin1237
slin1237 requested a review from CatherineSue as a code owner March 22, 2026 00:05
@coderabbitai

coderabbitai Bot commented Mar 22, 2026 •

Copy link
Copy Markdown
📝 Walkthrough

Walkthrough

Updated Helm chart version to 1.3.3, renamed router and history CLI flags to new option names, added prefix_hash routing policy option, adjusted router cache eviction interval from 60 to 120 seconds, disabled non-root security context, and refactored label rendering templates for consistency.

Changes

Cohort / File(s) Summary
Version Update
deploy/helm/smg/Chart.yaml
Bumped Helm chart appVersion from 0.7.0 to 1.3.3.
Router Configuration
deploy/helm/smg/templates/_helpers.tpl, deploy/helm/smg/values.schema.json, deploy/helm/smg/values.yaml
Renamed router/history CLI flags: --eviction-interval-secs → --eviction-interval, --postgres-pool-max → --postgres-pool-max-size, --redis-pool-max → --redis-pool-max-size, --oracle-connect-descriptor → --oracle-dsn, --oracle-username → --oracle-user. Added prefix_hash to router.policy enum. Increased router.evictionIntervalSecs from 60 to 120. Changed router.securityContext.runAsNonRoot from true to false.
Label Rendering Templates
deploy/helm/smg/templates/grafana-dashboard.yaml, deploy/helm/smg/templates/servicemonitor.yaml
Replaced with + toYaml label rendering with explicit key/value range iteration, changing YAML output formatting to emit labels as key: "value" with quoted values.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

Possibly related PRs

  • PR #699: Introduces the original Helm chart templates and configuration structure being updated in this PR.
  • PR #859: Performs identical code-level changes to Chart.yaml appVersion, router CLI flag names, and router policy enum values.
  • PR #857: Coordinates related version bumps across the codebase to align with the 1.3.3 release.

Suggested reviewers

  • CatherineSue
  • key4ng
  • XinyueZhang369

Poem

🐰 Hop along to version one-point-three!
Flags renamed for clarity,
Labels rendered with care,
New routing paths to declare—
Our Helm chart hops merrily! 🌟

🚥 Pre-merge checks | ✅ 3
✅ Passed checks (3 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately summarizes the main changes: quoting custom labels in Helm templates and fixing the securityContext.runAsNonRoot default, which are the core bug fixes addressed in the PR.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/helm-observability

Comment @coderabbitai help to get the list of available commands and usage tips.

@gemini-code-assist

Copy link
Copy Markdown
Contributor

Summary of Changes

Hello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request enhances the robustness and compatibility of the Helm chart by addressing two critical configuration issues. It ensures that custom labels are correctly formatted for Kubernetes, preventing deployment failures, and adjusts the default security context to align with the application's runtime requirements, facilitating smoother deployments.

Highlights

  • Custom Label Quoting: Resolved an issue where custom labels on ServiceMonitor and Grafana dashboard ConfigMaps failed due to unquoted numeric-like values, causing Kubernetes to reject them. The fix ensures all custom label values are explicitly quoted.
  • Security Context Default: Corrected the default securityContext.runAsNonRoot setting from true to false in the Helm chart, addressing CreateContainerConfigError when deploying the SMG image which runs as root.
Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@deploy/helm/smg/Chart.yaml`:
- Around line 5-6: Bump the Chart.yaml chart version to reflect the appVersion
change: update the version field (currently "0.1.0") to a higher SemVer (e.g.,
"0.1.1" or appropriate minor/patch bump) so users are signaled about the
breaking/default value changes; modify the version key in Chart.yaml adjacent to
the existing appVersion key to a new SemVer value.

In `@deploy/helm/smg/values.yaml`:
- Around line 155-158: Add an inline comment next to the securityContext
configuration explaining why runAsNonRoot is set to false: document that the
container image "lightseekorg/smg" currently requires root to start (it triggers
CreateContainerConfigError when enforced), note any known limitations or plans
to upstream a non-root build, and reference any tickets or TODOs for converting
the image to non-root in the future; update the comment near the
securityContext/runAsNonRoot setting so future maintainers understand the
constraint and mitigation plan.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: 21ec19a2-a29b-4b93-a4ef-e9fa2266a0cc

📥 Commits

Reviewing files that changed from the base of the PR and between 2ef54dd and a7701c4.

📒 Files selected for processing (6)
  • deploy/helm/smg/Chart.yaml
  • deploy/helm/smg/templates/_helpers.tpl
  • deploy/helm/smg/templates/grafana-dashboard.yaml
  • deploy/helm/smg/templates/servicemonitor.yaml
  • deploy/helm/smg/values.schema.json
  • deploy/helm/smg/values.yaml

Comment on lines 5 to +6
version: 0.1.0
appVersion: "0.7.0"
appVersion: "1.3.3"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick | 🔵 Trivial

Consider bumping chart version alongside appVersion.

The appVersion bump to 1.3.3 reflects CLI flag renames (breaking changes for users with custom values). While version: 0.1.0 can remain if chart structure is unchanged, SemVer best practice for Helm charts is to bump the chart version when default values or template behavior changes, as this signals to users that an upgrade may require values file updates.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@deploy/helm/smg/Chart.yaml` around lines 5 - 6, Bump the Chart.yaml chart
version to reflect the appVersion change: update the version field (currently
"0.1.0") to a higher SemVer (e.g., "0.1.1" or appropriate minor/patch bump) so
users are signaled about the breaking/default value changes; modify the version
key in Chart.yaml adjacent to the existing appVersion key to a new SemVer value.

Comment on lines 155 to 158
securityContext:
readOnlyRootFilesystem: true
runAsNonRoot: true
runAsNonRoot: false
allowPrivilegeEscalation: false

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick | 🔵 Trivial

Document the rationale for runAsNonRoot: false.

The change fixes the CreateContainerConfigError since lightseekorg/smg runs as root. However, this weakens the default security posture. Consider adding a comment explaining why root is required, so future maintainers understand the constraint:

   # -- Security context for router container
+  # NOTE: runAsNonRoot is false because the lightseekorg/smg image currently
+  # runs as root. Set to true if using a non-root image variant.
   securityContext:
     readOnlyRootFilesystem: true
     runAsNonRoot: false
     allowPrivilegeEscalation: false
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@deploy/helm/smg/values.yaml` around lines 155 - 158, Add an inline comment
next to the securityContext configuration explaining why runAsNonRoot is set to
false: document that the container image "lightseekorg/smg" currently requires
root to start (it triggers CreateContainerConfigError when enforced), note any
known limitations or plans to upstream a non-root build, and reference any
tickets or TODOs for converting the image to non-root in the future; update the
comment near the securityContext/runAsNonRoot setting so future maintainers
understand the constraint and mitigation plan.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates the SMG Helm chart, incrementing the application version to 1.3.3. It renames several command-line arguments for the router, including --eviction-interval-secs to --eviction-interval and various database pool size and Oracle connection parameters. The templating for Grafana dashboard and ServiceMonitor labels has been refined, and a new prefix_hash routing policy has been introduced. Additionally, the default evictionIntervalSecs has been increased to 120, and the router container's runAsNonRoot security context has been set to false.

slin1237 added a commit that referenced this pull request Mar 22, 2026
The chart's default securityContext sets readOnlyRootFilesystem: true,
which prevents the container from writing to any path in the image
filesystem. Python and other runtime components may need to write
temporary files to /tmp.

This was discovered during Helm chart testing against a live cluster
where the pod would fail to start or crash when the process attempted
to write temp files. The workaround branches (fix/helm-edge-cases,
fix/helm-minimal-deploy, fix/helm-observability) included this fix
alongside a runAsNonRoot: false change. Now that the Docker image runs
as non-root (previous commit), runAsNonRoot: true is correct and this
/tmp volume is the only remaining piece needed.

What changed:
- deploy/helm/smg/templates/deployment-router.yaml: added emptyDir
  volume "tmp" mounted at /tmp in the router container

Why:
- readOnlyRootFilesystem: true is a security best practice but requires
  writable mounts for directories that need runtime writes
- /tmp is the standard location for temporary files in Python processes

Refs: #860, #861, #862
Signed-off-by: Simo Lin <linsimo.mark@gmail.com>
@slin1237
slin1237 merged commit a94176c into main Mar 22, 2026
27 checks passed
@slin1237
slin1237 deleted the fix/helm-observability branch March 22, 2026 00:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant