diff --git a/.agents/skills/debug-openshell-cluster/SKILL.md b/.agents/skills/debug-openshell-cluster/SKILL.md index 3fc53df904..4617d0d75f 100644 --- a/.agents/skills/debug-openshell-cluster/SKILL.md +++ b/.agents/skills/debug-openshell-cluster/SKILL.md @@ -269,14 +269,13 @@ When `userns` is configured (e.g. `userns = "auto"` or `userns = "keep-id"`): helm -n openshell status openshell helm -n openshell get values openshell kubectl -n openshell get deployment,statefulset,pod,svc,pvc -kubectl -n openshell logs deployment/openshell -c openshell-gateway --tail=200 -kubectl -n openshell logs statefulset/openshell -c openshell-gateway --tail=200 -kubectl -n openshell rollout status deployment/openshell -kubectl -n openshell rollout status statefulset/openshell +GATEWAY_DEPLOYMENT="$(kubectl -n openshell get deployment openshell >/dev/null 2>&1 && echo deployment/openshell || echo statefulset/openshell)" +kubectl -n openshell logs "${GATEWAY_DEPLOYMENT}" -c openshell-gateway --tail=200 +kubectl -n openshell rollout status "${GATEWAY_DEPLOYMENT}" ``` -Use the log and rollout commands for the workload kind that exists in the -release. Look for failed installs, unexpected values, missing namespace, wrong +Use the log and rollout commands for the gateway resource kind that exists in +the release. Look for failed installs, unexpected values, missing namespace, wrong image tag, TLS settings that do not match the registered endpoint, and scheduling failures. @@ -302,6 +301,54 @@ kubectl -n openshell get deployment,service,pod -l app.kubernetes.io/name=opensh kubectl -n openshell logs deployment/openshell-e2e-postgres --tail=200 ``` +Multi-replica gateways serialize cross-object sandbox and provider mutations +with a PostgreSQL advisory lock. If those RPCs stall while ordinary reads and +health checks remain responsive, inspect long-running database sessions and +advisory-lock waiters. Do not print the database URI or Secret contents into +logs: + +```sql +SELECT pid, granted, waitstart +FROM pg_locks +WHERE locktype = 'advisory'; +``` + +For multi-replica gateway installs, supervisor and client session traffic may +be served by a non-owner gateway replica and relayed to the current supervisor +owner over the internal `PeerRelay` RPC. Check the headless peer Service, +projected peer ServiceAccount token volume, and TokenReview RBAC: + +```bash +kubectl -n openshell get svc openshell-peer -o wide +kubectl -n openshell get endpoints openshell-peer +kubectl -n openshell get pod -l app.kubernetes.io/instance=openshell \ + -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{.spec.volumes[?(@.name=="gateway-peer-token")]}{"\n"}{.spec.volumes[?(@.name=="peer-client-tls")]}{"\n"}{.spec.containers[0].env[?(@.name=="OPENSHELL_PEER_SERVICE_ACCOUNT_TOKEN_FILE")]}{"\n"}{.spec.containers[0].env[?(@.name=="OPENSHELL_PEER_ENDPOINT")]}{"\n"}{.spec.containers[0].env[?(@.name=="OPENSHELL_PEER_TLS_SERVER_NAME")]}{"\n"}{end}' +kubectl auth can-i create tokenreviews.authentication.k8s.io \ + --as=system:serviceaccount:openshell:openshell +kubectl auth can-i get pods -n openshell \ + --as=system:serviceaccount:openshell:openshell +kubectl -n openshell logs "${GATEWAY_DEPLOYMENT}" --tail=200 | grep -E 'gateway peer|PeerRelay|supervisor owner|owner relay' +``` + +Expected gateway startup logs include +`gateway peer ServiceAccount TokenReview authentication enabled`. If peer relay +calls fail with `Unauthenticated`, verify the `gateway-peer-token` projected +volume has audience `openshell-gateway-peer` and that the receiving gateway can +create TokenReviews. If they fail with `PermissionDenied`, verify the gateway +ServiceAccount name, release namespace, pod UID, and Helm selector labels match +the live gateway pods. Deployment-backed gateway pods should also publish +`OPENSHELL_PEER_ENDPOINT` from their pod IP. The +`OPENSHELL_PEER_SERVICE_ACCOUNT_TOKEN_FILE` name follows the existing +token-file convention used by `OPENSHELL_SANDBOX_TOKEN_FILE` and +`OPENSHELL_K8S_SA_TOKEN_FILE`. + +For TLS-enabled gateways, peer clients verify the stable gateway Service DNS +name and load the chart CA plus client identity from +`OPENSHELL_PEER_TLS_CA_FILE`, `OPENSHELL_PEER_TLS_CERT_FILE`, and +`OPENSHELL_PEER_TLS_KEY_FILE`. If peer calls fail during TLS negotiation, verify +the `peer-client-tls` volume exists, those files are readable, and the server +certificate includes the name in `OPENSHELL_PEER_TLS_SERVER_NAME`. + Check required Helm deployment secrets: ```bash @@ -400,8 +447,8 @@ label, supervisor env vars `OPENSHELL_K8S_SA_TOKEN_FILE` and Check the image references currently used by the gateway deployment: ```bash -kubectl -n openshell get deployment openshell -o jsonpath="{.spec.template.spec.containers[*].image}{\"\n\"}{.spec.template.spec.containers[*].env[?(@.name==\"OPENSHELL_SUPERVISOR_IMAGE\")].value}{\"\n\"}" -kubectl -n openshell get statefulset openshell -o jsonpath="{.spec.template.spec.containers[*].image}{\"\n\"}{.spec.template.spec.containers[*].env[?(@.name==\"OPENSHELL_SUPERVISOR_IMAGE\")].value}{\"\n\"}" +GATEWAY_DEPLOYMENT="$(kubectl -n openshell get deployment openshell >/dev/null 2>&1 && echo deployment/openshell || echo statefulset/openshell)" +kubectl -n openshell get "${GATEWAY_DEPLOYMENT}" -o jsonpath="{.spec.template.spec.containers[*].image}{\"\n\"}{.spec.template.spec.containers[*].env[?(@.name==\"OPENSHELL_SUPERVISOR_IMAGE\")].value}{\"\n\"}" helm -n openshell get values openshell | grep -E 'repository|tag|supervisorImage|workload' ``` @@ -445,8 +492,8 @@ If the gateway is healthy but sandbox creation fails: ```bash kubectl -n openshell get pods kubectl -n openshell get events --sort-by=.lastTimestamp | tail -n 50 -kubectl -n openshell logs deployment/openshell -c openshell-gateway --tail=200 -kubectl -n openshell logs statefulset/openshell -c openshell-gateway --tail=200 +GATEWAY_DEPLOYMENT="$(kubectl -n openshell get deployment openshell >/dev/null 2>&1 && echo deployment/openshell || echo statefulset/openshell)" +kubectl -n openshell logs "${GATEWAY_DEPLOYMENT}" -c openshell-gateway --tail=200 ``` Check the configured sandbox namespace: diff --git a/.agents/skills/helm-dev-environment/SKILL.md b/.agents/skills/helm-dev-environment/SKILL.md index 2dad568c79..f094039121 100644 --- a/.agents/skills/helm-dev-environment/SKILL.md +++ b/.agents/skills/helm-dev-environment/SKILL.md @@ -83,10 +83,31 @@ install. Envoy Gateway opt-in; see the Optional Add-ons section below. The gateway Service uses ClusterIP. Access is via Envoy Gateway (port `8080`) or `kubectl port-forward`. -**HA test deploy** (two gateway replicas + external PostgreSQL Secret): uncomment -`#- ci/values-high-availability.yaml` in `deploy/helm/openshell/skaffold.yaml`, -create the Secret named `openshell-ha-pg` with a `uri` key, then run -`mise run helm:skaffold:run` or `mise run helm:skaffold:dev`. +The Skaffold profile for HA reverse-proxy development is available from +`deploy/helm/openshell/`: + +```bash +# Two gateway replicas + external PostgreSQL Secret + Envoy Gateway + Gateway API route. +KUBECONFIG=../../../kubeconfig skaffold run -p high-availability +``` + +The `high-availability` profile expects a Secret named `openshell-ha-pg` in the `openshell` +namespace with a `uri` key. For local manual testing, either create your own +PostgreSQL Secret or use the e2e PostgreSQL fixture manifest in +`e2e/kubernetes/postgres-fixture.yaml`. + +For the `high-availability` profile, return to the repository root and apply the +GatewayClass and BackendTrafficPolicy manifest after Skaffold has installed +Envoy Gateway: + +```bash +KUBECONFIG=kubeconfig mise run helm:gateway:apply +``` + +The BackendTrafficPolicy disables Envoy request and stream-duration timeouts for +OpenShell's `GRPCRoute`. Keep that policy in `deploy/kube/manifests/envoy-gateway-openshell.yaml`, +not in the Helm chart; it is required for long-lived gRPC create/watch/exec/relay +streams during gateway rollouts and scale events. ### TLS behaviour @@ -163,23 +184,85 @@ but will point to a deleted cluster — safe to ignore or clean up manually. ## Optional Add-ons -Each add-on requires uncommenting the corresponding `valuesFiles` entry in -`deploy/helm/openshell/skaffold.yaml` before running `helm:skaffold:dev` or `helm:skaffold:run`. +Some add-ons can be enabled by uncommenting values in `skaffold.yaml`, but prefer +the dedicated Skaffold profiles when they exist. Profiles avoid leaving local +manual edits in the worktree. ### Envoy Gateway (Gateway API / GRPCRoute) -Envoy Gateway is already installed by Skaffold (the `envoy-gateway` Helm release in -`skaffold.yaml`). To activate routing: +Use the `high-availability` Skaffold profile for HA reverse-proxy testing. The +profile intentionally includes Envoy Gateway so multi-replica behavior is +exercised through the same Gateway API path used by reverse-proxy deployments: -1. Uncomment `#- values-gateway.yaml` in `skaffold.yaml` -2. Redeploy: `mise run helm:skaffold:run` -3. Apply the GatewayClass: `mise run helm:gateway:apply` -4. Access: `http://127.0.0.1:8080` +```bash +cd deploy/helm/openshell +KUBECONFIG=../../../kubeconfig skaffold run -p high-availability +cd ../../.. +KUBECONFIG=kubeconfig mise run helm:gateway:apply +``` + +`values-gateway.yaml` creates a `Gateway` (listener on port 80, class `eg`) and +`GRPCRoute` in the `openshell` namespace. The `high-availability` profile +installs the Envoy Gateway Helm chart and layers both +`values-high-availability.yaml` and `values-gateway.yaml` onto the OpenShell +release. + +`deploy/kube/manifests/envoy-gateway-openshell.yaml` creates: + +- `GatewayClass/eg` +- `BackendTrafficPolicy/openshell-grpc-timeouts` + +The Envoy Gateway proxy Service is usually exposed through the k3d load balancer +at `http://127.0.0.1:8080`. If the cluster was created with a different +`HELM_K3S_LB_HOST_PORT`, use that host port instead. + +For manual tests against an existing cluster, prefer forwarding the Envoy proxy +Service rather than `svc/openshell`. That keeps client traffic on the same path +as a real reverse proxy while gateway pods rotate behind it: + +```bash +KUBECONFIG=kubeconfig kubectl get svc -A \ + -l gateway.envoyproxy.io/owning-gateway-name=openshell +KUBECONFIG=kubeconfig kubectl -n port-forward \ + svc/ 8080:80 +openshell gateway add http://127.0.0.1:8080 --name openshell --local +``` + +When running e2e tests manually through Envoy, register gateway metadata (as +above) instead of relying only on `OPENSHELL_GATEWAY_ENDPOINT`; some tests call +`openshell gateway info` and expect metadata for the active gateway. + +### Kubernetes E2E Notes + +Use `mise run e2e:kubernetes` for the standard Helm-backed Kubernetes suite. +The kube e2e wrapper creates only one port-forward, to `svc/openshell`; it no +longer forwards the unauthenticated health listener or runs a `/readyz` e2e +target. `/readyz` remains covered by server unit/integration tests. + +Use `mise run e2e:kubernetes:ha-rebalancing` for full-suite HA coverage. The +task creates an external PostgreSQL fixture, installs Envoy Gateway, applies +`deploy/kube/manifests/envoy-gateway-openshell.yaml`, enables the chart +`GRPCRoute`, and runs the full Kubernetes e2e suite, including +`kubernetes_ha_rebalancing`. That coverage validates sandbox create/watch and +exec through the Envoy proxy while gateway replicas scale up, scale down, and +rotate. It also keeps a long-running sandbox alive and runs upload/download +operations while gateway pods roll, so file sync exercises the same relay retry +path as interactive sessions. + +If you reuse an existing Skaffold cluster for the full kube suite, make sure the +chart has `server.hostGatewayIP` set so sandbox pods can resolve +`host.openshell.internal` back to the test host. The e2e wrapper detects this on +chart installs; manual reuse may require: + +```bash +HOST_GATEWAY_IP="${OPENSHELL_E2E_HOST_GATEWAY_IP:?set host gateway IP}" +KUBECONFIG=kubeconfig helm upgrade openshell deploy/helm/openshell \ + --namespace openshell --reuse-values \ + --set "server.hostGatewayIP=${HOST_GATEWAY_IP}" \ + --wait --timeout 5m +``` -`values-gateway.yaml` creates a `Gateway` (listener on port 80, class `eg`) and a -`GRPCRoute` in the `openshell` namespace. Envoy Gateway provisions a LoadBalancer -service for the proxy; klipper-lb binds it to hostPort 80, reachable via the -`8080:80` load balancer port mapping. +Use the IP that pods in that cluster use to reach listeners on the test host. ### Keycloak OIDC @@ -278,6 +361,6 @@ for dependencies still declared in `Chart.yaml`. | `deploy/helm/openshell/ci/values-spire.yaml` | SPIFFE/SPIRE provider token grant overlay | | `deploy/helm/openshell/ci/values-spire-stack.yaml` | SPIRE hardened chart values for local dev | | `deploy/helm/openshell/ci/values-tls-disabled.yaml` | Lint-only: TLS + auth disabled (reverse-proxy edge termination) | -| `deploy/kube/manifests/envoy-gateway-openshell.yaml` | GatewayClass for Envoy Gateway (`mise run helm:gateway:apply`) | +| `deploy/kube/manifests/envoy-gateway-openshell.yaml` | GatewayClass and BackendTrafficPolicy for Envoy Gateway (`mise run helm:gateway:apply`) | | `tasks/scripts/helm-k3s-local.sh` | k3d cluster create/delete/start/stop/status | | `tasks/scripts/keycloak-k8s-setup.sh` | Keycloak deploy + realm import | diff --git a/.github/workflows/branch-e2e.yml b/.github/workflows/branch-e2e.yml index 2b4d9d5d46..72733720d2 100644 --- a/.github/workflows/branch-e2e.yml +++ b/.github/workflows/branch-e2e.yml @@ -237,6 +237,8 @@ jobs: job-name: Kubernetes HA E2E (Rust smoke) extra-helm-values: deploy/helm/openshell/ci/values-high-availability.yaml external-postgres-secret: openshell-ha-pg + kubernetes-features: e2e,e2e-host-gateway,e2e-kubernetes,e2e-kubernetes-ha + use-envoy-gateway: true cli-artifact-prefix: rust-binary-cli kubernetes-credential-drivers-e2e: diff --git a/.github/workflows/e2e-kubernetes-test.yml b/.github/workflows/e2e-kubernetes-test.yml index 3e36570144..2760912075 100644 --- a/.github/workflows/e2e-kubernetes-test.yml +++ b/.github/workflows/e2e-kubernetes-test.yml @@ -37,6 +37,21 @@ on: required: false type: string default: "v0.5.0" + test-name: + description: "Rust e2e test target to run (sets OPENSHELL_E2E_KUBE_TEST)" + required: false + type: string + default: "" + kubernetes-features: + description: "Cargo feature list for the Kubernetes e2e crate" + required: false + type: string + default: "" + use-envoy-gateway: + description: "Install Envoy Gateway and run the e2e command through the chart GRPCRoute" + required: false + type: boolean + default: false e2e-task: description: "mise task to run for the Kubernetes e2e job" required: false @@ -144,6 +159,9 @@ jobs: OPENSHELL_E2E_KUBE_CONTEXT: kind-${{ env.KIND_CLUSTER_NAME }} OPENSHELL_E2E_KUBE_EXTRA_VALUES: ${{ inputs.extra-helm-values }} OPENSHELL_E2E_KUBE_EXTERNAL_POSTGRES_SECRET: ${{ inputs.external-postgres-secret }} + OPENSHELL_E2E_KUBE_TEST: ${{ inputs.test-name }} + OPENSHELL_E2E_KUBE_FEATURES: ${{ inputs.kubernetes-features }} + OPENSHELL_E2E_KUBE_USE_ENVOY: ${{ inputs.use-envoy-gateway }} IMAGE_TAG: ${{ inputs.image-tag }} OPENSHELL_REGISTRY: ghcr.io/nvidia/openshell E2E_TASK: ${{ inputs.e2e-task }} diff --git a/architecture/gateway.md b/architecture/gateway.md index 32bca6a1f6..044a9bf8fa 100644 --- a/architecture/gateway.md +++ b/architecture/gateway.md @@ -249,6 +249,58 @@ authenticated sandbox ID with any sandbox ID or name resolved from the request. Supervisor control and relay streams require a matching sandbox principal before the gateway registers the session or bridges relay bytes. +## HA Supervisor Ownership + +In multi-replica Kubernetes deployments, every gateway pod can accept client +RPCs, but a sandbox supervisor maintains one active stream to one gateway +replica at a time. The connected replica publishes a short-lived supervisor +owner record in the shared Postgres object store with its replica id, peer DNS +endpoint, supervisor instance id, and connection epoch. Ownership does not move +because another gateway receives a client request. It changes only when the +supervisor opens a new control stream, usually after the previous owner pod is +terminated or the stream breaks. A reconnect from the same supervisor instance +with a newer epoch can supersede the previous owner before the TTL expires, and +heartbeats from the active connection renew that current owner record. +Cleanup from an older connection checks the shared owner record before and +after changing sandbox readiness. It cannot demote a sandbox after a newer +replica has published replacement ownership. + +Session-bound operations such as exec, TCP forwarding, file sync, and sandbox +service routing first check the local session registry. If the supervisor is +owned by another gateway replica, the serving gateway opens an internal +`PeerRelay` stream to that owner and asks it to open the supervisor relay. This +keeps client traffic working when a Kubernetes Service routes the client to a +non-owner gateway pod. If a peer owner is stale or unreachable during a rollout, +the serving gateway retries ownership lookup until the normal relay wait +deadline. Each retry re-reads the owner record, so a supervisor reconnect or +heartbeat can surface a new owner; if no fresh reachable owner appears before +the deadline, the client operation fails rather than electing an owner itself. + +File upload and download use tar-over-SSH through the same relay path. A gateway +pod termination drops the active SSH proxy byte stream, so the CLI retries the +whole sync operation with a fresh SSH session instead of attempting mid-stream +resume. + +Gateway peer RPCs authenticate with Kubernetes ServiceAccount identity rather +than a shared secret. Helm mounts a projected, pod-bound token with audience +`openshell-gateway-peer`; the receiving gateway validates it through +TokenReview, checks the live pod UID and chart selector labels, and authorizes +only the internal peer relay method. When gateway TLS is enabled, peer clients +also trust the chart CA, present the chart-generated client certificate for +mTLS, and verify the stable gateway Service DNS name even when connecting to a +Deployment pod IP. + +`WatchSandbox` uses the local update bus for same-replica writes. One shared +poller per gateway observes resource-version changes made by other replicas and +feeds that bus for all local watchers, avoiding a database poll per client +stream. + +Mutations whose invariants span sandbox, provider-profile, policy, or provider +records take a process-local mutex and a shared PostgreSQL advisory lock. The +database session remains dedicated to the request and closes when the guard is +dropped, which releases the lock on normal completion, cancellation, or error. +SQLite deployments use only the local mutex because they are single-replica. + ## API Surface The gateway API is organized around platform objects and operational streams: @@ -443,7 +495,7 @@ migrations backfill existing rows with version 1. Provider profile imports, updates, and deletes hold the sandbox synchronization guard while checking attached-sandbox dynamic token grant ambiguity or in-use state and writing the profile record. Sandbox creation with initial providers and -sandbox provider attach/detach use the same guard, so one gateway process cannot +sandbox provider attach/detach use the same guard, so gateway replicas cannot interleave a profile mutation with a sandbox provider-set mutation that would leave an ambiguous final dynamic-token state or a deleted custom profile that is still referenced by a sandbox. diff --git a/crates/openshell-cli/src/ssh.rs b/crates/openshell-cli/src/ssh.rs index 4768dc27f2..ba09f4f0b3 100644 --- a/crates/openshell-cli/src/ssh.rs +++ b/crates/openshell-cli/src/ssh.rs @@ -4,7 +4,7 @@ //! SSH connection and proxy utilities. use crate::tls::{TlsOptions, grpc_client}; -use miette::{IntoDiagnostic, Result, WrapErr}; +use miette::{IntoDiagnostic, Report, Result, WrapErr}; #[cfg(unix)] use nix::sys::signal::{SaFlags, SigAction, SigHandler, SigSet, Signal, sigaction}; use openshell_core::forward::{ @@ -40,6 +40,9 @@ const FORWARD_LISTENER_PROBE_INTERVAL: Duration = Duration::from_millis(50); /// grace period. const FORWARD_LISTENER_CONNECT_TIMEOUT: Duration = Duration::from_millis(200); +const SYNC_RETRY_ATTEMPTS: usize = 4; +const SYNC_RETRY_DELAY: Duration = Duration::from_secs(2); + #[derive(Clone, Copy, Debug)] pub enum Editor { Vscode, @@ -603,6 +606,7 @@ pub async fn sandbox_exec( } /// What to pack into the tar archive streamed to the sandbox. +#[derive(Clone)] enum UploadSource { /// A single local file or directory. `tar_name` controls the entry name /// inside the archive (e.g. the target basename for file-to-file uploads). @@ -980,18 +984,15 @@ pub async fn sandbox_sync_up_files( if files.is_empty() { return Ok(()); } - ssh_tar_upload( - server, - name, - dest, - UploadSource::FileList { - base_dir: base_dir.to_path_buf(), - files: files.to_vec(), - archive_prefix: file_list_archive_prefix(local_path), - }, - tls, - workspace, - ) + let source = UploadSource::FileList { + base_dir: base_dir.to_path_buf(), + files: files.to_vec(), + archive_prefix: file_list_archive_prefix(local_path), + }; + retry_sandbox_sync("upload", || { + let source = source.clone(); + async move { ssh_tar_upload(server, name, dest, source, tls, workspace).await } + }) .await } @@ -1026,17 +1027,16 @@ pub async fn sandbox_sync_up( { let (parent, target_name) = split_sandbox_path(path); if parent != "/" { - return ssh_tar_upload( - server, - name, - Some(parent), - UploadSource::SinglePath { - local_path: local_path.to_path_buf(), - tar_name: target_name.into(), - }, - tls, - workspace, - ) + let source = UploadSource::SinglePath { + local_path: local_path.to_path_buf(), + tar_name: target_name.into(), + }; + return retry_sandbox_sync("upload", || { + let source = source.clone(); + async move { + ssh_tar_upload(server, name, Some(parent), source, tls, workspace).await + } + }) .await; } } @@ -1054,17 +1054,14 @@ pub async fn sandbox_sync_up( directory_upload_prefix(local_path) }; - ssh_tar_upload( - server, - name, - sandbox_path, - UploadSource::SinglePath { - local_path: local_path.to_path_buf(), - tar_name, - }, - tls, - workspace, - ) + let source = UploadSource::SinglePath { + local_path: local_path.to_path_buf(), + tar_name, + }; + retry_sandbox_sync("upload", || { + let source = source.clone(); + async move { ssh_tar_upload(server, name, sandbox_path, source, tls, workspace).await } + }) .await } @@ -1203,6 +1200,20 @@ pub async fn sandbox_sync_down( dest: &str, tls: &TlsOptions, workspace: &str, +) -> Result<()> { + retry_sandbox_sync("download", || async { + sandbox_sync_down_once(server, name, sandbox_path, dest, tls, workspace).await + }) + .await +} + +async fn sandbox_sync_down_once( + server: &str, + name: &str, + sandbox_path: &str, + dest: &str, + tls: &TlsOptions, + workspace: &str, ) -> Result<()> { let session = ssh_session_config(server, name, tls, workspace).await?; let sandbox_path = resolve_sandbox_source_path(&session, sandbox_path).await?; @@ -1216,6 +1227,54 @@ pub async fn sandbox_sync_down( } } +async fn retry_sandbox_sync(operation: &str, mut run: F) -> Result<()> +where + F: FnMut() -> Fut, + Fut: Future>, +{ + let mut attempt = 1; + loop { + match run().await { + Ok(()) => return Ok(()), + Err(err) if attempt < SYNC_RETRY_ATTEMPTS && sync_error_is_retryable(&err) => { + tracing::warn!( + operation, + attempt, + max_attempts = SYNC_RETRY_ATTEMPTS, + error = %err, + "sandbox sync operation failed; retrying" + ); + tokio::time::sleep(SYNC_RETRY_DELAY).await; + attempt += 1; + } + Err(err) => return Err(err), + } + } +} + +fn sync_error_is_retryable(err: &Report) -> bool { + let message = format!("{err:?}").to_ascii_lowercase(); + [ + "broken pipe", + "connection", + "early eof", + "http2", + "h2 protocol", + "reset before headers", + "service is currently unavailable", + "transport error", + "unexpected eof", + "unavailable", + "upstream connect error", + "ssh probe exited with status exit status: 255", + "ssh tar create exited", + "ssh tar extract exited", + "failed to extract tar archive from sandbox", + ] + .iter() + .any(|needle| message.contains(needle)) +} + /// Stream a tar archive from the sandbox and extract it into a fresh /// destination directory. The source is always wrapped on the sandbox side so /// the host can pick a basename when needed. @@ -1729,6 +1788,28 @@ mod tests { assert_eq!(output.matches("Host openshell-demo").count(), 1); } + #[test] + fn sync_error_retry_filter_accepts_transport_failures() { + let err = miette::miette!("transport error: connection reset by peer"); + assert!(sync_error_is_retryable(&err)); + } + + #[test] + fn sync_error_retry_filter_accepts_transient_ssh_probe_failures() { + let err = Err::<(), _>(miette::miette!( + "ssh probe exited with status exit status: 255" + )) + .wrap_err("failed to resolve sandbox source path '/sandbox/ha-sync/ha-sync-upload'") + .unwrap_err(); + assert!(sync_error_is_retryable(&err)); + } + + #[test] + fn sync_error_retry_filter_rejects_validation_failures() { + let err = miette::miette!("sandbox source path '/etc/passwd' resolves outside /sandbox"); + assert!(!sync_error_is_retryable(&err)); + } + #[test] #[allow(unsafe_code)] // Test-only: env vars require unsafe in Rust 2024. fn install_ssh_config_adds_include_once_and_updates_managed_file() { diff --git a/crates/openshell-cli/tests/ensure_providers_integration.rs b/crates/openshell-cli/tests/ensure_providers_integration.rs index 8192989375..f4736bdb33 100644 --- a/crates/openshell-cli/tests/ensure_providers_integration.rs +++ b/crates/openshell-cli/tests/ensure_providers_integration.rs @@ -630,6 +630,17 @@ impl OpenShell for TestOpenShell { Err(Status::unimplemented("not implemented in test")) } + type PeerRelayStream = tokio_stream::wrappers::ReceiverStream< + Result, + >; + + async fn peer_relay( + &self, + _request: tonic::Request>, + ) -> Result, Status> { + Err(Status::unimplemented("not implemented in test")) + } + type ForwardTcpStream = tokio_stream::wrappers::ReceiverStream< Result, >; diff --git a/crates/openshell-cli/tests/mtls_integration.rs b/crates/openshell-cli/tests/mtls_integration.rs index 91d520d1af..872e59110f 100644 --- a/crates/openshell-cli/tests/mtls_integration.rs +++ b/crates/openshell-cli/tests/mtls_integration.rs @@ -509,6 +509,17 @@ impl OpenShell for TestOpenShell { Err(Status::unimplemented("not implemented in test")) } + type PeerRelayStream = tokio_stream::wrappers::ReceiverStream< + Result, + >; + + async fn peer_relay( + &self, + _request: tonic::Request>, + ) -> Result, Status> { + Err(Status::unimplemented("not implemented in test")) + } + type ForwardTcpStream = tokio_stream::wrappers::ReceiverStream< Result, >; diff --git a/crates/openshell-cli/tests/provider_commands_integration.rs b/crates/openshell-cli/tests/provider_commands_integration.rs index c349bbdbb4..8e37a3429c 100644 --- a/crates/openshell-cli/tests/provider_commands_integration.rs +++ b/crates/openshell-cli/tests/provider_commands_integration.rs @@ -1032,6 +1032,17 @@ impl OpenShell for TestOpenShell { Err(Status::unimplemented("not implemented in test")) } + type PeerRelayStream = tokio_stream::wrappers::ReceiverStream< + Result, + >; + + async fn peer_relay( + &self, + _request: tonic::Request>, + ) -> Result, Status> { + Err(Status::unimplemented("not implemented in test")) + } + type ForwardTcpStream = tokio_stream::wrappers::ReceiverStream< Result, >; diff --git a/crates/openshell-cli/tests/sandbox_create_lifecycle_integration.rs b/crates/openshell-cli/tests/sandbox_create_lifecycle_integration.rs index bcc07619ee..2d7e75b55d 100644 --- a/crates/openshell-cli/tests/sandbox_create_lifecycle_integration.rs +++ b/crates/openshell-cli/tests/sandbox_create_lifecycle_integration.rs @@ -703,6 +703,17 @@ impl OpenShell for TestOpenShell { Err(Status::unimplemented("not implemented in test")) } + type PeerRelayStream = tokio_stream::wrappers::ReceiverStream< + Result, + >; + + async fn peer_relay( + &self, + _request: tonic::Request>, + ) -> Result, Status> { + Err(Status::unimplemented("not implemented in test")) + } + type ForwardTcpStream = tokio_stream::wrappers::ReceiverStream< Result, >; diff --git a/crates/openshell-cli/tests/sandbox_name_fallback_integration.rs b/crates/openshell-cli/tests/sandbox_name_fallback_integration.rs index 7e2cf74f50..409a97eceb 100644 --- a/crates/openshell-cli/tests/sandbox_name_fallback_integration.rs +++ b/crates/openshell-cli/tests/sandbox_name_fallback_integration.rs @@ -597,6 +597,17 @@ impl OpenShell for TestOpenShell { Err(Status::unimplemented("not implemented in test")) } + type PeerRelayStream = tokio_stream::wrappers::ReceiverStream< + Result, + >; + + async fn peer_relay( + &self, + _request: tonic::Request>, + ) -> Result, Status> { + Err(Status::unimplemented("not implemented in test")) + } + type ForwardTcpStream = tokio_stream::wrappers::ReceiverStream< Result, >; diff --git a/crates/openshell-sdk/tests/client_mock.rs b/crates/openshell-sdk/tests/client_mock.rs index 4b06b9c1e5..42b081415f 100644 --- a/crates/openshell-sdk/tests/client_mock.rs +++ b/crates/openshell-sdk/tests/client_mock.rs @@ -377,6 +377,16 @@ impl OpenShell for TestOpenShell { Err(Status::unimplemented("unused")) } + type PeerRelayStream = + tokio_stream::wrappers::ReceiverStream>; + + async fn peer_relay( + &self, + _: tonic::Request>, + ) -> Result, Status> { + Err(Status::unimplemented("unused")) + } + type ExecSandboxInteractiveStream = tokio_stream::wrappers::ReceiverStream>; diff --git a/crates/openshell-server/src/auth/descriptor_authz.rs b/crates/openshell-server/src/auth/descriptor_authz.rs index dbb9fa7ca2..1b060bcf8d 100644 --- a/crates/openshell-server/src/auth/descriptor_authz.rs +++ b/crates/openshell-server/src/auth/descriptor_authz.rs @@ -123,6 +123,7 @@ impl DescriptorAuthTable { "sandbox" => AuthMode::Sandbox, "bearer" => AuthMode::Bearer, "dual" => AuthMode::Dual, + "peer" => AuthMode::Peer, other => { return Err(format!("method {path}: unknown auth_mode '{other}'")); } diff --git a/crates/openshell-server/src/auth/guard.rs b/crates/openshell-server/src/auth/guard.rs index edcd6bc013..a3d55eac71 100644 --- a/crates/openshell-server/src/auth/guard.rs +++ b/crates/openshell-server/src/auth/guard.rs @@ -30,6 +30,9 @@ use tracing::info; pub fn ensure_sandbox_scope(principal: &Principal, claimed_sandbox_id: &str) -> Result<(), Status> { match principal { Principal::User(_) => Ok(()), + Principal::Peer(_) => Err(Status::permission_denied( + "gateway peer principals may not call sandbox-scoped methods", + )), Principal::Sandbox(p) => { if p.sandbox_id == claimed_sandbox_id { Ok(()) @@ -84,7 +87,7 @@ pub fn ensure_sandbox_principal_scope( ensure_sandbox_scope(principal, claimed_sandbox_id)?; Ok(p.clone()) } - Principal::User(_) => Err(Status::permission_denied( + Principal::User(_) | Principal::Peer(_) => Err(Status::permission_denied( "supervisor RPCs require a sandbox principal", )), Principal::Anonymous => Err(Status::unauthenticated( diff --git a/crates/openshell-server/src/auth/method_authz.rs b/crates/openshell-server/src/auth/method_authz.rs index 71eb7acac6..9b9862878a 100644 --- a/crates/openshell-server/src/auth/method_authz.rs +++ b/crates/openshell-server/src/auth/method_authz.rs @@ -22,6 +22,8 @@ pub enum AuthMode { /// Either sandbox principal or Bearer; scope and role apply on /// the Bearer path only. Dual, + /// Only callable by a gateway peer principal. + Peer, } /// Coarse role mapping. Maps to the configured `admin_role` / @@ -86,11 +88,17 @@ pub fn is_sandbox_callable(method: &str) -> bool { #[must_use] pub fn is_user_callable(method: &str) -> bool { match lookup(method).map(|m| m.auth_mode) { - Some(AuthMode::Sandbox | AuthMode::Unauthenticated) => false, + Some(AuthMode::Sandbox | AuthMode::Unauthenticated | AuthMode::Peer) => false, Some(AuthMode::Bearer | AuthMode::Dual) | None => true, } } +/// `true` if the method is callable by a gateway peer. +#[must_use] +pub fn is_peer_callable(method: &str) -> bool { + matches!(lookup(method).map(|m| m.auth_mode), Some(AuthMode::Peer)) +} + #[cfg(test)] mod tests { use super::*; @@ -139,6 +147,8 @@ mod tests { "/openshell.v1.OpenShell/ConnectSupervisor" )); assert!(!is_user_callable("/openshell.v1.OpenShell/RelayStream")); + assert!(!is_user_callable("/openshell.v1.OpenShell/PeerRelay")); + assert!(is_peer_callable("/openshell.v1.OpenShell/PeerRelay")); assert!(!is_user_callable( "/openshell.inference.v1.Inference/GetInferenceBundle" )); diff --git a/crates/openshell-server/src/auth/mod.rs b/crates/openshell-server/src/auth/mod.rs index bedbebe015..f8b77296b1 100644 --- a/crates/openshell-server/src/auth/mod.rs +++ b/crates/openshell-server/src/auth/mod.rs @@ -18,6 +18,7 @@ pub mod identity; pub mod k8s_sa; pub mod method_authz; pub mod oidc; +pub mod peer; pub mod principal; pub mod sandbox_jwt; pub mod sandbox_methods; diff --git a/crates/openshell-server/src/auth/peer.rs b/crates/openshell-server/src/auth/peer.rs new file mode 100644 index 0000000000..b92c77bd31 --- /dev/null +++ b/crates/openshell-server/src/auth/peer.rs @@ -0,0 +1,597 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! Gateway peer authentication for internal replica-to-replica RPCs. +//! +//! Peer calls use Kubernetes projected `ServiceAccount` tokens, not an +//! OpenShell-managed shared secret. The caller presents its pod-bound gateway +//! `ServiceAccount` token with the peer audience; the receiver validates it with +//! the apiserver `TokenReview` API, checks the live pod UID and required labels, +//! and only then produces a [`Principal::Peer`]. + +use super::authenticator::Authenticator; +use super::principal::{PeerPrincipal, Principal}; +use async_trait::async_trait; +use k8s_openapi::api::{ + authentication::v1::{TokenReview, TokenReviewSpec, TokenReviewStatus, UserInfo}, + core::v1::Pod, +}; +use k8s_openapi::apimachinery::pkg::apis::meta::v1::ObjectMeta; +use kube::api::{Api, PostParams}; +use std::path::PathBuf; +use std::sync::Arc; +use tonic::Status; +use tracing::{debug, info, warn}; + +/// gRPC path for internal gateway relay forwarding. +pub const PEER_RELAY_PATH: &str = "/openshell.v1.OpenShell/PeerRelay"; +/// Audience used for gateway-to-gateway projected `ServiceAccount` tokens. +pub const DEFAULT_PEER_TOKEN_AUDIENCE: &str = "openshell-gateway-peer"; +/// Environment variable overriding the expected peer token audience. +pub const PEER_TOKEN_AUDIENCE_ENV: &str = "OPENSHELL_PEER_TOKEN_AUDIENCE"; +/// Environment variable carrying the projected peer `ServiceAccount` token path. +/// Uses the `*_TOKEN_FILE` convention from sandbox token env vars. +pub const PEER_SA_TOKEN_FILE_ENV: &str = "OPENSHELL_PEER_SERVICE_ACCOUNT_TOKEN_FILE"; +/// Default mount path for the projected peer `ServiceAccount` token. +pub const DEFAULT_PEER_SA_TOKEN_FILE: &str = "/var/run/secrets/openshell-peer/token"; +/// Environment variable with comma-separated `key=value` pod labels required +/// on authenticated gateway peer pods. +pub const PEER_REQUIRED_POD_LABELS_ENV: &str = "OPENSHELL_PEER_POD_LABELS"; +const POD_NAME_EXTRA: &str = "authentication.kubernetes.io/pod-name"; +const POD_UID_EXTRA: &str = "authentication.kubernetes.io/pod-uid"; + +#[derive(Debug, Clone)] +pub struct ResolvedGatewayPeerIdentity { + pub pod_name: String, + pub pod_uid: String, +} + +#[async_trait] +pub trait GatewayPeerIdentityResolver: Send + Sync + 'static { + async fn resolve(&self, token: &str) -> Result, Status>; +} + +#[derive(Debug)] +struct PeerTokenReviewIdentity { + pod_name: String, + pod_uid: String, +} + +pub struct PeerServiceAccountAuthenticator { + resolver: Arc, +} + +impl std::fmt::Debug for PeerServiceAccountAuthenticator { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.debug_struct("PeerServiceAccountAuthenticator") + .finish_non_exhaustive() + } +} + +impl PeerServiceAccountAuthenticator { + pub fn new(resolver: Arc) -> Self { + Self { resolver } + } +} + +#[async_trait] +impl Authenticator for PeerServiceAccountAuthenticator { + async fn authenticate( + &self, + headers: &http::HeaderMap, + path: &str, + ) -> Result, Status> { + if path != PEER_RELAY_PATH { + return Ok(None); + } + + let Some(token) = headers + .get("authorization") + .and_then(|v| v.to_str().ok()) + .and_then(|v| v.strip_prefix("Bearer ")) + else { + return Ok(None); + }; + + let Some(resolved) = self.resolver.resolve(token).await? else { + debug!("K8s gateway peer token did not authenticate; falling through"); + return Ok(None); + }; + + if let Some(claimed_replica) = headers + .get("x-openshell-peer-replica") + .and_then(|v| v.to_str().ok()) + .filter(|v| !v.is_empty()) + && claimed_replica != resolved.pod_name.as_str() + { + warn!( + claimed_replica, + pod_name = %resolved.pod_name, + "gateway peer replica header does not match authenticated pod" + ); + return Err(Status::permission_denied( + "gateway peer replica does not match authenticated pod", + )); + } + + Ok(Some(Principal::Peer(PeerPrincipal { + replica_id: resolved.pod_name, + pod_uid: resolved.pod_uid, + }))) + } +} + +/// Resolver backed by Kubernetes `TokenReview` and a live Pod lookup. +pub struct LiveGatewayPeerResolver { + token_reviews_api: Api, + pods_api: Api, + expected_audience: String, + namespace: String, + expected_service_account: String, + required_pod_labels: Vec<(String, String)>, +} + +impl LiveGatewayPeerResolver { + pub fn new( + client: kube::Client, + namespace: &str, + expected_audience: String, + expected_service_account: String, + required_pod_labels: Vec<(String, String)>, + ) -> Self { + let token_reviews_api: Api = Api::all(client.clone()); + let pods_api: Api = Api::namespaced(client, namespace); + Self { + token_reviews_api, + pods_api, + expected_audience, + namespace: namespace.to_string(), + expected_service_account, + required_pod_labels, + } + } +} + +#[async_trait] +impl GatewayPeerIdentityResolver for LiveGatewayPeerResolver { + async fn resolve(&self, token: &str) -> Result, Status> { + let review = TokenReview { + metadata: ObjectMeta::default(), + spec: TokenReviewSpec { + audiences: Some(vec![self.expected_audience.clone()]), + token: Some(token.to_string()), + }, + status: None, + }; + + let review = self + .token_reviews_api + .create(&PostParams::default(), &review) + .await + .map_err(|err| { + warn!(error = %err, "K8s TokenReview failed for gateway peer"); + Status::internal(format!("peer tokenreview failed: {err}")) + })?; + let status = review + .status + .ok_or_else(|| Status::internal("TokenReview response missing status"))?; + let Some(identity) = peer_token_review_identity( + &status, + &self.expected_audience, + &self.namespace, + &self.expected_service_account, + )? + else { + return Ok(None); + }; + + let pod = self + .pods_api + .get_opt(&identity.pod_name) + .await + .map_err(|err| { + warn!( + pod = %identity.pod_name, + error = %err, + "failed to fetch gateway peer pod" + ); + Status::internal(format!("gateway peer pod GET failed: {err}")) + })?; + let Some(pod) = pod else { + warn!( + pod = %identity.pod_name, + "gateway peer pod referenced by SA token not found" + ); + return Err(Status::not_found("gateway peer pod not found")); + }; + + let actual_uid = pod.metadata.uid.as_deref().unwrap_or_default(); + if actual_uid != identity.pod_uid { + warn!( + pod = %identity.pod_name, + claimed_uid = %identity.pod_uid, + actual_uid, + "gateway peer SA token pod UID does not match live pod" + ); + return Err(Status::permission_denied( + "gateway peer SA token pod UID mismatch", + )); + } + + let actual_service_account = pod + .spec + .as_ref() + .and_then(|spec| spec.service_account_name.as_deref()) + .unwrap_or("default"); + if actual_service_account != self.expected_service_account { + warn!( + pod = %identity.pod_name, + service_account = %actual_service_account, + expected_service_account = %self.expected_service_account, + "gateway peer pod service account does not match TokenReview principal" + ); + return Err(Status::permission_denied( + "gateway peer pod service account mismatch", + )); + } + + validate_required_pod_labels(&pod, &self.required_pod_labels)?; + + info!( + pod_name = %identity.pod_name, + pod_uid = %identity.pod_uid, + service_account = %self.expected_service_account, + "validated gateway peer ServiceAccount token via TokenReview" + ); + + Ok(Some(ResolvedGatewayPeerIdentity { + pod_name: identity.pod_name, + pod_uid: identity.pod_uid, + })) + } +} + +#[allow(clippy::result_large_err)] +fn peer_token_review_identity( + status: &TokenReviewStatus, + expected_audience: &str, + namespace: &str, + expected_service_account: &str, +) -> Result, Status> { + if status.authenticated != Some(true) { + debug!( + error = status.error.as_deref().unwrap_or_default(), + "K8s TokenReview did not authenticate gateway peer token" + ); + return Ok(None); + } + + let audiences = status.audiences.as_deref().unwrap_or_default(); + if !audiences.iter().any(|aud| aud == expected_audience) { + warn!( + expected_audience, + audiences = ?audiences, + "K8s TokenReview authenticated gateway peer token without expected audience" + ); + return Err(Status::unauthenticated( + "gateway peer token audience not accepted", + )); + } + + let user = status + .user + .as_ref() + .ok_or_else(|| Status::permission_denied("TokenReview response missing user info"))?; + let username = user + .username + .as_deref() + .ok_or_else(|| Status::permission_denied("TokenReview response missing username"))?; + let expected_username = format!("system:serviceaccount:{namespace}:{expected_service_account}"); + if username != expected_username { + warn!( + username, + namespace, + service_account = %expected_service_account, + "K8s TokenReview principal is not the configured gateway service account" + ); + return Err(Status::permission_denied( + "gateway peer token is not from the configured service account", + )); + } + + let pod_name = user_extra_one(user, POD_NAME_EXTRA)?; + let pod_uid = user_extra_one(user, POD_UID_EXTRA)?; + Ok(Some(PeerTokenReviewIdentity { pod_name, pod_uid })) +} + +#[allow(clippy::result_large_err)] +fn user_extra_one(user: &UserInfo, key: &str) -> Result { + let Some(values) = user.extra.as_ref().and_then(|extra| extra.get(key)) else { + return Err(Status::permission_denied( + "gateway peer token is not pod-bound", + )); + }; + if values.len() != 1 || values[0].is_empty() { + return Err(Status::permission_denied( + "gateway peer token has invalid pod binding", + )); + } + Ok(values[0].clone()) +} + +#[allow(clippy::result_large_err)] +fn validate_required_pod_labels( + pod: &Pod, + required_labels: &[(String, String)], +) -> Result<(), Status> { + let labels = pod.metadata.labels.as_ref(); + for (key, expected) in required_labels { + let actual = labels + .and_then(|labels| labels.get(key)) + .map(String::as_str) + .unwrap_or_default(); + if actual != expected { + warn!( + pod = %pod.metadata.name.as_deref().unwrap_or_default(), + label = %key, + expected, + actual, + "gateway peer pod missing required label" + ); + return Err(Status::permission_denied( + "gateway peer pod labels do not match", + )); + } + } + Ok(()) +} + +pub fn peer_token_audience_from_env() -> String { + std::env::var(PEER_TOKEN_AUDIENCE_ENV) + .ok() + .filter(|value| !value.trim().is_empty()) + .unwrap_or_else(|| DEFAULT_PEER_TOKEN_AUDIENCE.to_string()) +} + +pub fn peer_service_account_token_file_from_env() -> Option { + if let Ok(path) = std::env::var(PEER_SA_TOKEN_FILE_ENV) + && !path.trim().is_empty() + { + return Some(PathBuf::from(path.trim())); + } + + let default_path = PathBuf::from(DEFAULT_PEER_SA_TOKEN_FILE); + default_path.exists().then_some(default_path) +} + +pub fn load_peer_service_account_token_from_env() -> Result, String> { + let Some(path) = peer_service_account_token_file_from_env() else { + return Ok(None); + }; + + let contents = std::fs::read_to_string(&path) + .map_err(|err| format!("failed to read {}: {err}", path.display()))?; + let token = contents.trim(); + if token.is_empty() { + return Err(format!( + "peer ServiceAccount token file {} is empty", + path.display() + )); + } + + Ok(Some(token.to_string())) +} + +pub fn required_pod_labels_from_env() -> Result, String> { + let raw = std::env::var(PEER_REQUIRED_POD_LABELS_ENV).unwrap_or_default(); + parse_required_pod_labels(&raw) +} + +fn parse_required_pod_labels(raw: &str) -> Result, String> { + let mut labels = Vec::new(); + for entry in raw + .split(',') + .map(str::trim) + .filter(|entry| !entry.is_empty()) + { + let Some((key, value)) = entry.split_once('=') else { + return Err(format!( + "{PEER_REQUIRED_POD_LABELS_ENV} entry {entry:?} must be key=value" + )); + }; + let key = key.trim(); + let value = value.trim(); + if key.is_empty() || value.is_empty() { + return Err(format!( + "{PEER_REQUIRED_POD_LABELS_ENV} entry {entry:?} must have non-empty key and value" + )); + } + labels.push((key.to_string(), value.to_string())); + } + Ok(labels) +} + +#[cfg(test)] +pub mod test_support { + use super::*; + use std::sync::Mutex; + + pub struct FakeGatewayPeerResolver { + pub outcome: Result, Status>, + pub seen_tokens: Mutex>, + } + + impl FakeGatewayPeerResolver { + pub fn returning(outcome: Result, Status>) -> Self { + Self { + outcome, + seen_tokens: Mutex::new(Vec::new()), + } + } + } + + #[async_trait] + impl GatewayPeerIdentityResolver for FakeGatewayPeerResolver { + async fn resolve( + &self, + token: &str, + ) -> Result, Status> { + self.seen_tokens.lock().unwrap().push(token.to_string()); + match &self.outcome { + Ok(opt) => Ok(opt.clone()), + Err(status) => Err(Status::new(status.code(), status.message())), + } + } + } +} + +#[cfg(test)] +mod tests { + use super::test_support::FakeGatewayPeerResolver; + use super::*; + use std::collections::BTreeMap; + + fn bearer_headers(token: &str) -> http::HeaderMap { + let mut headers = http::HeaderMap::new(); + headers.insert( + "authorization", + http::HeaderValue::from_str(&format!("Bearer {token}")).unwrap(), + ); + headers + } + + fn token_review_status( + authenticated: bool, + audiences: Vec<&str>, + username: &str, + extra: Vec<(&str, &str)>, + ) -> TokenReviewStatus { + TokenReviewStatus { + authenticated: Some(authenticated), + audiences: Some(audiences.into_iter().map(str::to_string).collect()), + error: None, + user: Some(UserInfo { + username: Some(username.to_string()), + uid: Some("sa-uid".to_string()), + groups: Some(vec![ + "system:serviceaccounts".to_string(), + "system:serviceaccounts:openshell".to_string(), + "system:authenticated".to_string(), + ]), + extra: Some( + extra + .into_iter() + .map(|(key, value)| (key.to_string(), vec![value.to_string()])) + .collect(), + ), + }), + } + } + + #[test] + fn peer_token_review_identity_extracts_pod_binding() { + let status = token_review_status( + true, + vec![DEFAULT_PEER_TOKEN_AUDIENCE], + "system:serviceaccount:openshell:openshell", + vec![(POD_NAME_EXTRA, "openshell-0"), (POD_UID_EXTRA, "uid-a")], + ); + + let identity = peer_token_review_identity( + &status, + DEFAULT_PEER_TOKEN_AUDIENCE, + "openshell", + "openshell", + ) + .unwrap() + .expect("authenticated token should resolve"); + + assert_eq!(identity.pod_name, "openshell-0"); + assert_eq!(identity.pod_uid, "uid-a"); + } + + #[test] + fn peer_token_review_identity_rejects_wrong_service_account() { + let status = token_review_status( + true, + vec![DEFAULT_PEER_TOKEN_AUDIENCE], + "system:serviceaccount:openshell:default", + vec![(POD_NAME_EXTRA, "openshell-0"), (POD_UID_EXTRA, "uid-a")], + ); + + let err = peer_token_review_identity( + &status, + DEFAULT_PEER_TOKEN_AUDIENCE, + "openshell", + "openshell", + ) + .expect_err("wrong service account must fail closed"); + assert_eq!(err.code(), tonic::Code::PermissionDenied); + } + + #[test] + fn validate_required_pod_labels_rejects_mismatch() { + let pod = Pod { + metadata: ObjectMeta { + name: Some("openshell-0".to_string()), + labels: Some(BTreeMap::from([( + "app.kubernetes.io/name".to_string(), + "openshell".to_string(), + )])), + ..Default::default() + }, + ..Default::default() + }; + + let err = validate_required_pod_labels( + &pod, + &[( + "app.kubernetes.io/instance".to_string(), + "release-a".to_string(), + )], + ) + .expect_err("missing required label must fail"); + assert_eq!(err.code(), tonic::Code::PermissionDenied); + } + + #[tokio::test] + async fn authenticator_uses_resolved_pod_name_as_replica() { + let resolver = Arc::new(FakeGatewayPeerResolver::returning(Ok(Some( + ResolvedGatewayPeerIdentity { + pod_name: "openshell-0".to_string(), + pod_uid: "uid-a".to_string(), + }, + )))); + let auth = PeerServiceAccountAuthenticator::new(resolver); + + let principal = auth + .authenticate(&bearer_headers("token-a"), PEER_RELAY_PATH) + .await + .unwrap() + .expect("principal"); + + let Principal::Peer(peer) = principal else { + panic!("expected peer principal"); + }; + assert_eq!(peer.replica_id, "openshell-0"); + assert_eq!(peer.pod_uid, "uid-a"); + } + + #[test] + fn parse_required_pod_labels_accepts_comma_list() { + let labels = parse_required_pod_labels( + "app.kubernetes.io/name=openshell,app.kubernetes.io/instance=dev", + ) + .unwrap(); + assert_eq!( + labels, + vec![ + ( + "app.kubernetes.io/name".to_string(), + "openshell".to_string() + ), + ("app.kubernetes.io/instance".to_string(), "dev".to_string()) + ] + ); + } +} diff --git a/crates/openshell-server/src/auth/principal.rs b/crates/openshell-server/src/auth/principal.rs index 1d4cb7276c..ead3179a57 100644 --- a/crates/openshell-server/src/auth/principal.rs +++ b/crates/openshell-server/src/auth/principal.rs @@ -28,6 +28,8 @@ pub enum Principal { /// sandbox UUID. The wrapped `sandbox_id` MUST match any sandbox referenced /// in the request body for sandbox-class methods. Sandbox(#[allow(dead_code)] SandboxPrincipal), + /// Gateway replica authenticated for internal peer RPCs. + Peer(PeerPrincipal), /// Truly unauthenticated caller (health probes, reflection). Sandbox-class /// and user-class methods reject this variant. #[allow(dead_code)] @@ -57,6 +59,15 @@ pub struct SandboxPrincipal { pub trust_domain: Option, } +/// Gateway peer caller. +#[derive(Debug, Clone)] +pub struct PeerPrincipal { + /// Peer replica id supplied by the authenticated caller. + pub replica_id: String, + /// UID of the authenticated Kubernetes pod. + pub pod_uid: String, +} + /// How a [`SandboxPrincipal`] was authenticated. /// /// Variant fields are populated by the producing authenticator and consumed diff --git a/crates/openshell-server/src/auth/sandbox_methods.rs b/crates/openshell-server/src/auth/sandbox_methods.rs index 89f34d1253..db7aa194ea 100644 --- a/crates/openshell-server/src/auth/sandbox_methods.rs +++ b/crates/openshell-server/src/auth/sandbox_methods.rs @@ -26,6 +26,7 @@ mod tests { "/openshell.v1.OpenShell/ConnectSupervisor" )); assert!(is_sandbox_callable("/openshell.v1.OpenShell/RelayStream")); + assert!(!is_sandbox_callable("/openshell.v1.OpenShell/PeerRelay")); assert!(is_sandbox_callable( "/openshell.v1.OpenShell/GetSandboxConfig" )); diff --git a/crates/openshell-server/src/auth/workspace_authz.rs b/crates/openshell-server/src/auth/workspace_authz.rs index e23d2287a3..508c8b4a69 100644 --- a/crates/openshell-server/src/auth/workspace_authz.rs +++ b/crates/openshell-server/src/auth/workspace_authz.rs @@ -124,6 +124,9 @@ pub async fn authorize_workspace( workspace, grant: AuthGrant::Sandbox, }), + Principal::Peer(_) => Err(Status::permission_denied( + "gateway peer principals cannot perform workspace operations", + )), Principal::Anonymous => Err(Status::unauthenticated("authentication required")), } } @@ -159,6 +162,9 @@ pub fn require_platform_admin(admin_role: &str, principal: &Principal) -> Result Principal::Sandbox(_) => Err(Status::permission_denied( "sandbox principals cannot perform cross-workspace operations", )), + Principal::Peer(_) => Err(Status::permission_denied( + "gateway peer principals cannot perform cross-workspace operations", + )), Principal::Anonymous => Err(Status::unauthenticated("authentication required")), } } diff --git a/crates/openshell-server/src/compute/mod.rs b/crates/openshell-server/src/compute/mod.rs index 30a1303bd5..f763b72a8e 100644 --- a/crates/openshell-server/src/compute/mod.rs +++ b/crates/openshell-server/src/compute/mod.rs @@ -25,6 +25,7 @@ use crate::persistence::{ }; use crate::sandbox_index::SandboxIndex; use crate::sandbox_watch::SandboxWatchBus; +use crate::supervisor_owner::{OWNER_TTL, SupervisorOwnerIndex}; use crate::supervisor_session::SupervisorSessionRegistry; use crate::tracing_bus::TracingLogBus; use futures::{Stream, StreamExt}; @@ -570,6 +571,13 @@ pub struct ComputeRuntime { replica_id: String, } +pub struct SandboxSyncGuard { + // Drop the database guard before the local mutex so another local waiter + // cannot race ahead while this replica still owns the cluster-wide lock. + _distributed: crate::persistence::DistributedMutationGuard, + _local: tokio::sync::OwnedMutexGuard<()>, +} + impl fmt::Debug for ComputeRuntime { fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { f.debug_struct("ComputeRuntime").finish_non_exhaustive() @@ -690,13 +698,21 @@ impl ComputeRuntime { } /// Serializes sandbox/provider-profile invariant checks and object writes - /// within this gateway process. + /// across gateway replicas. /// - /// This is a temporary single-gateway guard for cross-object invariants. - /// It is not HA-safe; replace it with DB-backed CAS/resource-version writes - /// tracked by #1255 before enabling multiple gateway writers. - pub(crate) async fn sandbox_sync_guard(&self) -> tokio::sync::OwnedMutexGuard<()> { - self.sync_lock.clone().lock_owned().await + /// The local mutex preserves lock ordering within one process. `PostgreSQL` + /// deployments additionally acquire a database advisory lock so the + /// validation and related writes remain atomic with respect to other + /// gateway replicas. + pub(crate) async fn sandbox_sync_guard( + &self, + ) -> crate::persistence::PersistenceResult { + let local = self.sync_lock.clone().lock_owned().await; + let distributed = self.store.acquire_distributed_mutation_guard().await?; + Ok(SandboxSyncGuard { + _distributed: distributed, + _local: local, + }) } /// Acquires the process-wide lock for code that already holds the @@ -2767,7 +2783,7 @@ impl ComputeRuntime { expected_resource_version: u64, existing_phase: SandboxPhase, ) -> Result<(), String> { - let session_connected = self.supervisor_sessions.has_session(&incoming.id); + let session_connected = self.supervisor_session_ready(&incoming.id).await?; let sandbox = self .store .update_message_cas::( @@ -2802,30 +2818,85 @@ impl ComputeRuntime { sandbox_id: &str, instance_id: &str, ) -> Result<(), String> { - self.set_supervisor_session_state(sandbox_id, true, Some(instance_id)) - .await + let _guard = self.sync_lock.lock().await; + let sandbox = self + .update_supervisor_session_state(sandbox_id, true, Some(instance_id)) + .await?; + self.publish_supervisor_session_state(sandbox_id, sandbox); + Ok(()) } pub async fn supervisor_session_disconnected(&self, sandbox_id: &str) -> Result<(), String> { - self.set_supervisor_session_state(sandbox_id, false, None) + let _guard = self.sync_lock.lock().await; + + // A replacement session may already be owned by another replica. Do + // not let cleanup from this replica overwrite the new owner's Ready + // state. Recheck after the demotion as well: if ownership changed + // between the first read and the CAS, restore Ready before releasing + // the synchronization lock. A new owner published after the second + // read will run supervisor_session_connected and promote the sandbox. + if self.supervisor_session_ready(sandbox_id).await? { + return Ok(()); + } + + let demoted = self + .update_supervisor_session_state(sandbox_id, false, None) + .await?; + let replacement_instance_id = self + .supervisor_session_owner_instance_id(sandbox_id) + .await?; + let sandbox = if let Some(instance_id) = replacement_instance_id.as_deref() { + self.update_supervisor_session_state(sandbox_id, true, Some(instance_id)) + .await? + } else { + demoted + }; + self.publish_supervisor_session_state(sandbox_id, sandbox); + Ok(()) + } + + async fn supervisor_session_ready(&self, sandbox_id: &str) -> Result { + if self.supervisor_sessions.has_session(sandbox_id) { + return Ok(true); + } + + Ok(self + .supervisor_session_owner_instance_id(sandbox_id) + .await? + .is_some()) + } + + async fn supervisor_session_owner_instance_id( + &self, + sandbox_id: &str, + ) -> Result, String> { + let owner_index = SupervisorOwnerIndex::new(self.store.clone(), OWNER_TTL); + let Some(owner) = owner_index + .read(sandbox_id) .await + .map_err(|err| err.to_string())? + else { + return Ok(None); + }; + + let age_ms = openshell_core::time::now_ms() - owner.updated_at_ms; + let ttl_ms = i64::try_from(OWNER_TTL.as_millis()).unwrap_or(i64::MAX); + Ok((age_ms < ttl_ms).then_some(owner.supervisor_instance_id)) } - async fn set_supervisor_session_state( + async fn update_supervisor_session_state( &self, sandbox_id: &str, connected: bool, instance_id: Option<&str>, - ) -> Result<(), String> { - let _guard = self.sync_lock.lock().await; - + ) -> Result, String> { let Some(existing) = self .store .get_message::(sandbox_id) .await .map_err(|err| err.to_string())? else { - return Ok(()); + return Ok(None); }; let current_phase = SandboxPhase::try_from(existing.phase()).unwrap_or(SandboxPhase::Unknown); @@ -2836,10 +2907,10 @@ impl ComputeRuntime { | SandboxPhase::Stopping | SandboxPhase::Stopped ) { - return Ok(()); + return Ok(None); } if !connected && current_phase != SandboxPhase::Ready { - return Ok(()); + return Ok(None); } let expected_resource_version = sandbox_resource_version(&existing); @@ -2867,7 +2938,7 @@ impl ComputeRuntime { Err(crate::persistence::PersistenceError::Database(ref msg)) if msg.contains("not found") => { - return Ok(()); + return Ok(None); } Err(crate::persistence::PersistenceError::Conflict { current_resource_version, @@ -2881,9 +2952,14 @@ impl ComputeRuntime { Err(e) => return Err(e.to_string()), }; - self.sandbox_index.update_from_sandbox(&sandbox); - self.sandbox_watch_bus.notify(sandbox_id); - Ok(()) + Ok(Some(sandbox)) + } + + fn publish_supervisor_session_state(&self, sandbox_id: &str, sandbox: Option) { + if let Some(sandbox) = sandbox { + self.sandbox_index.update_from_sandbox(&sandbox); + self.sandbox_watch_bus.notify(sandbox_id); + } } /// Persist a terminal canonical-process result. Exit code zero is still a @@ -3927,7 +4003,7 @@ fn ensure_supervisor_not_ready_status(status: &mut Option, sandbo r#type: "Ready".to_string(), status: "False".to_string(), reason: "DependenciesNotReady".to_string(), - message: "Supervisor session disconnected".to_string(), + message: "Supervisor session not connected".to_string(), last_transition_time: String::new(), }, ); @@ -7880,7 +7956,43 @@ mod tests { .unwrap(); assert_eq!(ready.status, "False"); assert_eq!(ready.reason, "DependenciesNotReady"); - assert_eq!(ready.message, "Supervisor session disconnected"); + assert_eq!(ready.message, "Supervisor session not connected"); + } + + #[tokio::test] + async fn stale_session_disconnect_preserves_new_remote_owner_ready_state() { + let runtime = test_runtime(Arc::new(TestDriver::default())).await; + let mut sandbox = sandbox_record("sb-1", "sandbox-a", SandboxPhase::Ready); + sandbox.set_phase(SandboxPhase::Ready as i32); + runtime.store.put_message(&sandbox).await.unwrap(); + + SupervisorOwnerIndex::new(runtime.store.clone(), OWNER_TTL) + .publish( + "sb-1", + "replacement-session", + "supervisor-instance", + 2, + "gateway-b", + "http://gateway-b:8080", + ) + .await + .unwrap(); + + runtime + .supervisor_session_disconnected("sb-1") + .await + .unwrap(); + + let stored = runtime + .store + .get_message::("sb-1") + .await + .unwrap() + .unwrap(); + assert_eq!( + SandboxPhase::try_from(stored.phase()).unwrap(), + SandboxPhase::Ready + ); } // --- Composition rule tests --- diff --git a/crates/openshell-server/src/grpc/mod.rs b/crates/openshell-server/src/grpc/mod.rs index f502369bcf..9019ea7371 100644 --- a/crates/openshell-server/src/grpc/mod.rs +++ b/crates/openshell-server/src/grpc/mod.rs @@ -40,7 +40,7 @@ use openshell_core::proto::{ ListProvidersResponse, ListSandboxPoliciesRequest, ListSandboxPoliciesResponse, ListSandboxProvidersRequest, ListSandboxProvidersResponse, ListSandboxesRequest, ListSandboxesResponse, ListServicesRequest, ListServicesResponse, ListWorkspaceMembersRequest, - ListWorkspaceMembersResponse, ListWorkspacesRequest, ListWorkspacesResponse, + ListWorkspaceMembersResponse, ListWorkspacesRequest, ListWorkspacesResponse, PeerRelayFrame, ProviderProfileResponse, ProviderResponse, PushSandboxLogsRequest, PushSandboxLogsResponse, RefreshSandboxTokenRequest, RefreshSandboxTokenResponse, RejectDraftChunkRequest, RejectDraftChunkResponse, RelayFrame, RemoveWorkspaceMemberRequest, @@ -755,6 +755,16 @@ impl OpenShell for OpenShellService { ) -> Result, Status> { workspace::handle_list_workspace_members(&self.state, request).await } + + type PeerRelayStream = + Pin> + Send + 'static>>; + + async fn peer_relay( + &self, + request: Request>, + ) -> Result, Status> { + crate::supervisor_session::handle_peer_relay(&self.state, request).await + } } // --------------------------------------------------------------------------- diff --git a/crates/openshell-server/src/grpc/policy.rs b/crates/openshell-server/src/grpc/policy.rs index 114b43aba0..00d91ab927 100644 --- a/crates/openshell-server/src/grpc/policy.rs +++ b/crates/openshell-server/src/grpc/policy.rs @@ -2300,6 +2300,9 @@ async fn resolve_sandbox_by_name_for_principal( Ok(sandbox) } Principal::User(_) => sandbox.ok_or_else(|| Status::not_found("sandbox not found")), + Principal::Peer(_) => Err(Status::permission_denied( + "gateway peer principals may not resolve sandboxes by name", + )), Principal::Anonymous => Err(Status::unauthenticated( "sandbox-scoped methods require an authenticated caller", )), @@ -3685,7 +3688,9 @@ async fn handle_update_config_inner( } let _sandbox_sync_guard = if backfill_policy.is_some() { - Some(state.compute.sandbox_sync_guard().await) + Some(state.compute.sandbox_sync_guard().await.map_err(|err| { + super::persistence_error_to_status(err, "acquire policy mutation lock") + })?) } else { None }; @@ -4025,7 +4030,9 @@ pub(super) async fn handle_report_policy_status( // Update current_policy_version using CAS // TODO: Accept expected_version from UpdateConfigRequest for proper client-driven CAS - let _sandbox_sync_guard = state.compute.sandbox_sync_guard().await; + let _sandbox_sync_guard = state.compute.sandbox_sync_guard().await.map_err(|err| { + super::persistence_error_to_status(err, "acquire policy mutation lock") + })?; let version_to_set = req.version; state .store diff --git a/crates/openshell-server/src/grpc/provider.rs b/crates/openshell-server/src/grpc/provider.rs index e20edc055f..cadb5d3a4d 100644 --- a/crates/openshell-server/src/grpc/provider.rs +++ b/crates/openshell-server/src/grpc/provider.rs @@ -2555,7 +2555,10 @@ pub(super) async fn handle_import_provider_profiles( .ensure_active()?; let (profiles, mut diagnostics) = profiles_from_import_items(&request.profiles); add_empty_profile_set_diagnostic(&profiles, &mut diagnostics); - let _sandbox_sync_guard = state.compute.sandbox_sync_guard().await; + let _sandbox_sync_guard = + state.compute.sandbox_sync_guard().await.map_err(|err| { + super::persistence_error_to_status(err, "acquire provider mutation lock") + })?; let catalog = state .provider_profile_sources .snapshot_catalog(state.store.as_ref(), &workspace) @@ -2645,7 +2648,10 @@ pub(super) async fn handle_update_provider_profiles( let (profiles, mut diagnostics) = profiles_from_import_items(&items); add_empty_profile_set_diagnostic(&profiles, &mut diagnostics); let target_id = normalize_profile_id_request(&request.id)?; - let _sandbox_sync_guard = state.compute.sandbox_sync_guard().await; + let _sandbox_sync_guard = + state.compute.sandbox_sync_guard().await.map_err(|err| { + super::persistence_error_to_status(err, "acquire provider mutation lock") + })?; let catalog = state .provider_profile_sources .snapshot_catalog(state.store.as_ref(), &workspace) @@ -2804,7 +2810,10 @@ pub(super) async fn handle_delete_provider_profile( .name; let id = req.id; let id = normalize_profile_id_request(&id)?; - let _sandbox_sync_guard = state.compute.sandbox_sync_guard().await; + let _sandbox_sync_guard = + state.compute.sandbox_sync_guard().await.map_err(|err| { + super::persistence_error_to_status(err, "acquire provider mutation lock") + })?; let catalog = state .provider_profile_sources .snapshot_catalog(state.store.as_ref(), &workspace) @@ -4236,7 +4245,10 @@ pub(super) async fn handle_configure_provider_refresh( // configures of providers attached to the same sandbox could each pass // validation before either persisted and both reserve the same key (CWE-362). // This is the same guard sandbox create/attach and profile changes take. - let _sandbox_sync_guard = state.compute.sandbox_sync_guard().await; + let _sandbox_sync_guard = + state.compute.sandbox_sync_guard().await.map_err(|err| { + super::persistence_error_to_status(err, "acquire provider mutation lock") + })?; let provider = state .store @@ -5082,7 +5094,7 @@ mod tests { #[tokio::test] async fn import_provider_profile_waits_for_sandbox_sync_guard() { let state = test_server_state().await; - let guard = state.compute.sandbox_sync_guard().await; + let guard = state.compute.sandbox_sync_guard().await.unwrap(); let task_state = state.clone(); let task = tokio::spawn(async move { handle_import_provider_profiles( @@ -7675,7 +7687,7 @@ mod tests { .await .unwrap(); - let guard = state.compute.sandbox_sync_guard().await; + let guard = state.compute.sandbox_sync_guard().await.unwrap(); let task_state = state.clone(); let task = tokio::spawn(async move { handle_delete_provider_profile( diff --git a/crates/openshell-server/src/grpc/sandbox.rs b/crates/openshell-server/src/grpc/sandbox.rs index 89f8c942ea..f7877ab535 100644 --- a/crates/openshell-server/src/grpc/sandbox.rs +++ b/crates/openshell-server/src/grpc/sandbox.rs @@ -253,7 +253,9 @@ async fn handle_create_sandbox_inner( let _sandbox_sync_guard = if spec.providers.is_empty() { None } else { - Some(state.compute.sandbox_sync_guard().await) + Some(state.compute.sandbox_sync_guard().await.map_err(|err| { + super::persistence_error_to_status(err, "acquire sandbox mutation lock") + })?) }; // Validate provider names exist (fail fast). @@ -543,7 +545,10 @@ pub(super) async fn handle_attach_sandbox_provider( } })?; - let _sandbox_sync_guard = state.compute.sandbox_sync_guard().await; + let _sandbox_sync_guard = + state.compute.sandbox_sync_guard().await.map_err(|err| { + super::persistence_error_to_status(err, "acquire sandbox mutation lock") + })?; let sandbox = sandbox_by_name(state, &workspace, &request.sandbox_name).await?; let sandbox_id = sandbox .metadata @@ -678,7 +683,10 @@ pub(super) async fn handle_detach_sandbox_provider( ))); } - let _sandbox_sync_guard = state.compute.sandbox_sync_guard().await; + let _sandbox_sync_guard = + state.compute.sandbox_sync_guard().await.map_err(|err| { + super::persistence_error_to_status(err, "acquire sandbox mutation lock") + })?; let sandbox = sandbox_by_name(state, &workspace, &request.sandbox_name).await?; let sandbox_id = sandbox .metadata @@ -1213,11 +1221,15 @@ pub(super) async fn handle_exec_sandbox( // Open a relay channel through the supervisor session. Use a 15s // session-wait timeout, enough to cover a transient supervisor reconnect // while still failing quickly during normal operation. - let (channel_id, relay_rx) = state - .supervisor_sessions - .open_relay(sandbox.object_id(), std::time::Duration::from_secs(15)) - .await - .map_err(|e| Status::unavailable(format!("supervisor relay failed: {e}")))?; + let (channel_id, relay_rx) = crate::supervisor_session::open_routed_relay_with_target( + state, + sandbox.object_id(), + relay_open::Target::Ssh(SshRelayTarget {}), + String::new(), + std::time::Duration::from_secs(15), + ) + .await + .map_err(|e| Status::unavailable(format!("supervisor relay failed: {e}")))?; let command_str = build_remote_exec_command(&req) .map_err(|e| Status::invalid_argument(format!("command construction failed: {e}")))?; @@ -1321,16 +1333,15 @@ pub(super) async fn handle_forward_tcp( } let connection_guard = acquire_forward_connection_guard(state, &init, &sandbox).await?; - let (channel_id, relay_rx) = state - .supervisor_sessions - .open_relay_with_target( - sandbox.object_id(), - target, - init.service_id.clone(), - std::time::Duration::from_secs(15), - ) - .await - .map_err(|e| Status::unavailable(format!("supervisor relay failed: {e}")))?; + let (channel_id, relay_rx) = crate::supervisor_session::open_routed_relay_with_target( + state, + sandbox.object_id(), + target, + init.service_id.clone(), + std::time::Duration::from_secs(15), + ) + .await + .map_err(|e| Status::unavailable(format!("supervisor relay failed: {e}")))?; let sandbox_id = sandbox.object_id().to_string(); let (tx, rx) = mpsc::channel::>(256); @@ -1642,11 +1653,15 @@ pub(super) async fn handle_exec_sandbox_interactive( return Err(Status::failed_precondition("sandbox is not ready")); } - let (channel_id, relay_rx) = state - .supervisor_sessions - .open_relay(sandbox.object_id(), std::time::Duration::from_secs(15)) - .await - .map_err(|e| Status::unavailable(format!("supervisor relay failed: {e}")))?; + let (channel_id, relay_rx) = crate::supervisor_session::open_routed_relay_with_target( + state, + sandbox.object_id(), + relay_open::Target::Ssh(SshRelayTarget {}), + String::new(), + std::time::Duration::from_secs(15), + ) + .await + .map_err(|e| Status::unavailable(format!("supervisor relay failed: {e}")))?; let command_str = build_remote_exec_command(&req) .map_err(|e| Status::invalid_argument(format!("command construction failed: {e}")))?; @@ -2858,7 +2873,7 @@ mod tests { // Hold the global guard so the handler can resolve the original ID and // acquire its delete gate, but cannot yet revalidate or mutate it. - let global_guard = state.compute.sandbox_sync_guard().await; + let global_guard = state.compute.sandbox_sync_guard().await.unwrap(); let delete_state = state.clone(); let delete = tokio::spawn(async move { handle_delete_sandbox_inner( @@ -3616,7 +3631,7 @@ mod tests { .await .unwrap(); - let guard = state.compute.sandbox_sync_guard().await; + let guard = state.compute.sandbox_sync_guard().await.unwrap(); let task_state = state.clone(); let task = tokio::spawn(async move { handle_create_sandbox( diff --git a/crates/openshell-server/src/grpc/workspace.rs b/crates/openshell-server/src/grpc/workspace.rs index d83ffab0e8..1ec91a30e5 100644 --- a/crates/openshell-server/src/grpc/workspace.rs +++ b/crates/openshell-server/src/grpc/workspace.rs @@ -66,6 +66,9 @@ fn membership_filter_subject<'a>( } } Principal::Sandbox(_) => Ok(None), + Principal::Peer(_) => Err(Status::permission_denied( + "gateway peer principals cannot list workspaces", + )), Principal::Anonymous => Err(Status::unauthenticated("authentication required")), } } diff --git a/crates/openshell-server/src/inference.rs b/crates/openshell-server/src/inference.rs index b83fd6be4f..2ba0d5745f 100644 --- a/crates/openshell-server/src/inference.rs +++ b/crates/openshell-server/src/inference.rs @@ -1021,7 +1021,9 @@ fn authorize_inference_bundle( ) -> Result { match principal { Some(crate::auth::principal::Principal::Sandbox(s)) => Ok(s.sandbox_id.clone()), - Some(crate::auth::principal::Principal::User(_)) => Err(Status::permission_denied( + Some( + crate::auth::principal::Principal::User(_) | crate::auth::principal::Principal::Peer(_), + ) => Err(Status::permission_denied( "GetInferenceBundle requires a sandbox principal", )), Some(crate::auth::principal::Principal::Anonymous) | None => Err(Status::unauthenticated( diff --git a/crates/openshell-server/src/lib.rs b/crates/openshell-server/src/lib.rs index 2667611bcc..5e24da53d3 100644 --- a/crates/openshell-server/src/lib.rs +++ b/crates/openshell-server/src/lib.rs @@ -36,6 +36,7 @@ mod sandbox_index; mod sandbox_watch; mod service_routing; mod ssh_sessions; +mod supervisor_owner; pub mod supervisor_session; mod telemetry; #[cfg(any(test, feature = "test-support"))] @@ -295,6 +296,12 @@ pub struct ServerState { /// query session state to surface supervisor readiness. pub supervisor_sessions: Arc, + /// Stable identity for this gateway process. + pub replica_id: String, + + /// Internal endpoint other gateway replicas can dial for peer RPCs. + pub peer_endpoint: Option, + /// Validated built-in and operator-registered supervisor middleware. pub middleware_registry: Arc, @@ -317,6 +324,9 @@ pub struct ServerState { /// runs in-cluster. pub k8s_sa_authenticator: Option>, + /// Optional K8s `ServiceAccount` authenticator for gateway peer RPCs. + pub peer_authenticator: Option>, + /// Gateway-wide gRPC request rate limiter shared by every multiplex path. pub(crate) grpc_rate_limiter: Option, @@ -393,6 +403,8 @@ impl ServerState { oidc_cache: Option>, credentials: credentials::CredentialRuntime, ) -> Self { + let replica_id = compute::lease::replica_id(); + let peer_endpoint = derive_peer_endpoint(&config); let grpc_rate_limiter = multiplex::GrpcRateLimiter::from_config(&config); let admin_role = config .oidc @@ -411,12 +423,15 @@ impl ServerState { ssh_connections_by_sandbox: Mutex::new(HashMap::new()), settings_mutex: tokio::sync::Mutex::new(()), supervisor_sessions, + replica_id, + peer_endpoint, extension_mint_limiter: auth::extension_mint_limit::ExtensionMintLimiter::default(), middleware_registry: Arc::new(MiddlewareRegistry::default()), oidc_cache, sandbox_jwt_issuer: None, sandbox_jwt_authenticator: None, k8s_sa_authenticator: None, + peer_authenticator: None, grpc_rate_limiter, gateway_interceptors: None, provider_profile_sources: @@ -426,6 +441,34 @@ impl ServerState { } } +fn derive_peer_endpoint(config: &Config) -> Option { + if let Ok(endpoint) = std::env::var("OPENSHELL_PEER_ENDPOINT") + && !endpoint.trim().is_empty() + { + return Some(endpoint.trim().to_string()); + } + + let pod_name = std::env::var("OPENSHELL_POD_NAME").ok()?; + let namespace = std::env::var("OPENSHELL_POD_NAMESPACE").ok()?; + let service = std::env::var("OPENSHELL_PEER_SERVICE_NAME").ok()?; + if pod_name.trim().is_empty() || namespace.trim().is_empty() || service.trim().is_empty() { + return None; + } + + let scheme = if config.tls.is_some() { + "https" + } else { + "http" + }; + Some(format!( + "{scheme}://{pod}.{service}.{namespace}.svc.cluster.local:{port}", + pod = pod_name.trim(), + service = service.trim(), + namespace = namespace.trim(), + port = config.bind_address.port() + )) +} + /// Run the `OpenShell` server. /// /// This starts a multiplexed gRPC/HTTP server on the configured bind address. @@ -699,6 +742,51 @@ pub(crate) async fn run_server( } } + if std::env::var_os("KUBERNETES_SERVICE_HOST").is_some() { + let namespace = std::env::var("OPENSHELL_POD_NAMESPACE").ok(); + let service_account = std::env::var("OPENSHELL_SERVICE_ACCOUNT_NAME").ok(); + match (namespace, service_account) { + (Some(namespace), Some(service_account)) + if !namespace.trim().is_empty() && !service_account.trim().is_empty() => + { + let required_labels = + auth::peer::required_pod_labels_from_env().map_err(Error::config)?; + match kube::Client::try_default().await { + Ok(client) => { + let audience = auth::peer::peer_token_audience_from_env(); + let resolver = Arc::new(auth::peer::LiveGatewayPeerResolver::new( + client, + namespace.trim(), + audience.clone(), + service_account.trim().to_string(), + required_labels, + )); + let authenticator = + auth::peer::PeerServiceAccountAuthenticator::new(resolver); + state.peer_authenticator = Some(Arc::new(authenticator)); + info!( + namespace = %namespace.trim(), + service_account = %service_account.trim(), + audience, + "gateway peer ServiceAccount TokenReview authentication enabled" + ); + } + Err(err) => warn!( + error = %err, + "in-cluster K8s client construction failed; \ + gateway peer ServiceAccount authentication is disabled" + ), + } + } + _ => { + debug!( + "OPENSHELL_POD_NAMESPACE or OPENSHELL_SERVICE_ACCOUNT_NAME missing; \ + gateway peer ServiceAccount authentication disabled" + ); + } + } + } + let state = Arc::new(state); // Reconcile local-driver running intent before watchers spawn so their @@ -717,6 +805,12 @@ pub(crate) async fn run_server( } state.compute.spawn_watchers(shutdown_rx.clone()); + sandbox_watch::spawn_store_poller( + store.clone(), + state.sandbox_watch_bus.clone(), + Duration::from_secs(1), + shutdown_rx.clone(), + ); ssh_sessions::spawn_session_reaper(store.clone(), Duration::from_secs(3600)); supervisor_session::spawn_relay_reaper(state.clone(), Duration::from_secs(30)); provider_refresh::spawn_refresh_worker(state.clone(), Duration::from_secs(60)); diff --git a/crates/openshell-server/src/multiplex.rs b/crates/openshell-server/src/multiplex.rs index 7a7125dcc3..ba138edd9a 100644 --- a/crates/openshell-server/src/multiplex.rs +++ b/crates/openshell-server/src/multiplex.rs @@ -630,6 +630,11 @@ fn gateway_principal_fields(principal: &Principal) -> BTreeMap { fields.insert("trust_domain".to_string(), trust_domain.clone()); } } + Principal::Peer(peer) => { + fields.insert("kind".to_string(), "peer".to_string()); + fields.insert("replica_id".to_string(), peer.replica_id.clone()); + fields.insert("pod_uid".to_string(), peer.pod_uid.clone()); + } Principal::Anonymous => { fields.insert("kind".to_string(), "anonymous".to_string()); } @@ -853,13 +858,16 @@ where /// Assemble the authenticator chain for the gateway. /// /// Chain order (first-match-wins): -/// 1. `K8sServiceAccountAuthenticator` (path-scoped to `IssueSandboxToken`) +/// 1. `PeerServiceAccountAuthenticator` (path-scoped to `PeerRelay`) +/// — validates gateway replica projected `ServiceAccount` tokens with +/// `TokenReview` for internal peer relay calls. No-op on every other path. +/// 2. `K8sServiceAccountAuthenticator` (path-scoped to `IssueSandboxToken`) /// — exchanges a projected SA token for a `Principal::Sandbox` so the /// `IssueSandboxToken` handler can mint a gateway JWT. No-op on every /// other path; only present when the gateway runs in-cluster. -/// 2. `SandboxJwtAuthenticator` — validates gateway-minted JWTs. Recognized +/// 3. `SandboxJwtAuthenticator` — validates gateway-minted JWTs. Recognized /// via a distinctive `kid` so non-matching Bearer tokens fall through. -/// 3. `OidcAuthenticator` — validates user Bearer tokens against the +/// 4. `OidcAuthenticator` — validates user Bearer tokens against the /// configured OIDC issuer. Returns `Unauthenticated` for missing /// Bearer headers so non-OIDC clients can't sneak through. /// @@ -874,6 +882,9 @@ where /// to pass-through unless mTLS or local unauthenticated users are enabled. fn build_authenticator_chain(state: &ServerState) -> Option { let mut authenticators: Vec> = Vec::new(); + if let Some(peer) = state.peer_authenticator.clone() { + authenticators.push(peer); + } if let Some(k8s) = state.k8s_sa_authenticator.clone() { authenticators.push(k8s); } @@ -1041,6 +1052,13 @@ where ))); } } + Principal::Peer(_) => { + if !crate::auth::method_authz::is_peer_callable(&path) { + return Ok(status_response(tonic::Status::permission_denied( + "gateway peer principals may not call this method", + ))); + } + } Principal::Anonymous => { return Ok(status_response(tonic::Status::unauthenticated( "anonymous callers may not call authenticated methods", diff --git a/crates/openshell-server/src/persistence/mod.rs b/crates/openshell-server/src/persistence/mod.rs index 516faf4fe4..f66590874a 100644 --- a/crates/openshell-server/src/persistence/mod.rs +++ b/crates/openshell-server/src/persistence/mod.rs @@ -127,6 +127,11 @@ pub enum Store { Sqlite(SqliteStore), } +/// RAII guard for the database-backed cross-object mutation lock. +pub struct DistributedMutationGuard { + _postgres: Option, +} + /// Trait for inferring an object type string from a message type. pub trait ObjectType { fn object_type() -> &'static str; @@ -201,6 +206,23 @@ impl Store { matches!(self, Self::Sqlite(_)) } + /// Serialize mutations whose invariants span multiple persisted objects. + /// + /// `SQLite` deployments are single-replica and use only the caller's local + /// mutex. `PostgreSQL` deployments additionally hold a session-level + /// advisory lock so concurrent gateway replicas cannot validate and write + /// the same cross-object invariant independently. + pub async fn acquire_distributed_mutation_guard( + &self, + ) -> PersistenceResult { + match self { + Self::Postgres(store) => Ok(DistributedMutationGuard { + _postgres: Some(store.acquire_cross_object_lock().await?), + }), + Self::Sqlite(_) => Ok(DistributedMutationGuard { _postgres: None }), + } + } + /// Connect to a persistence store based on the database URL. pub async fn connect(url: &str) -> CoreResult { if url.starts_with("postgres://") || url.starts_with("postgresql://") { diff --git a/crates/openshell-server/src/persistence/postgres.rs b/crates/openshell-server/src/persistence/postgres.rs index 8bab0ada96..5014f85163 100644 --- a/crates/openshell-server/src/persistence/postgres.rs +++ b/crates/openshell-server/src/persistence/postgres.rs @@ -13,8 +13,9 @@ use crate::policy_store::{ use openshell_core::SetResourceVersion; use openshell_core::proto::Sandbox; use prost::Message; +use sqlx::pool::PoolConnection; use sqlx::postgres::PgPoolOptions; -use sqlx::{Connection, PgPool, Row}; +use sqlx::{Connection, PgPool, Postgres, Row}; static POSTGRES_MIGRATOR: sqlx::migrate::Migrator = sqlx::migrate!("./migrations/postgres"); @@ -25,6 +26,18 @@ pub struct PostgresStore { pool: PgPool, } +// Stable cluster-wide key for serializing sandbox/provider cross-object +// mutations. The bytes spell "OPENSHLL" and stay within PostgreSQL's signed +// 64-bit advisory-lock key space. +const CROSS_OBJECT_ADVISORY_LOCK_KEY: i64 = 0x4f50_454e_5348_4c4c; + +pub(super) struct PostgresAdvisoryLockGuard { + // `close_on_drop` is set before this guard is constructed. Closing the + // dedicated session releases the session-level advisory lock even when a + // request is cancelled or returns early. + _connection: PoolConnection, +} + impl PostgresStore { pub async fn connect(url: &str) -> PersistenceResult { let pool = PgPoolOptions::new() @@ -50,6 +63,21 @@ impl PostgresStore { conn.ping().await.map_err(|e| map_db_error(&e)) } + pub(super) async fn acquire_cross_object_lock( + &self, + ) -> PersistenceResult { + let mut connection = self.pool.acquire().await.map_err(|e| map_db_error(&e))?; + connection.close_on_drop(); + sqlx::query("SELECT pg_advisory_lock($1)") + .bind(CROSS_OBJECT_ADVISORY_LOCK_KEY) + .execute(&mut *connection) + .await + .map_err(|e| map_db_error(&e))?; + Ok(PostgresAdvisoryLockGuard { + _connection: connection, + }) + } + /// Test support only: close the underlying connection pool. /// /// Do not call from runtime code; this tears down the active pool. diff --git a/crates/openshell-server/src/sandbox_watch.rs b/crates/openshell-server/src/sandbox_watch.rs index ac38eba8db..0fbe485f77 100644 --- a/crates/openshell-server/src/sandbox_watch.rs +++ b/crates/openshell-server/src/sandbox_watch.rs @@ -3,12 +3,16 @@ //! In-memory buses to support sandbox watch streaming. -use std::collections::HashMap; +use std::collections::{HashMap, HashSet}; use std::sync::{Arc, Mutex}; +use std::time::Duration; -use tokio::sync::broadcast; +use tokio::sync::{broadcast, watch}; use tonic::Status; +use crate::persistence::Store; +use openshell_core::proto::Sandbox; + /// Broadcast bus of sandbox updates keyed by sandbox id. /// /// Producers call [`SandboxWatchBus::notify`] whenever the persisted sandbox record changes. @@ -57,6 +61,71 @@ impl SandboxWatchBus { let mut inner = self.inner.lock().expect("sandbox watch bus lock poisoned"); inner.remove(sandbox_id); } + + fn active_sandbox_ids(&self) -> HashSet { + self.inner + .lock() + .expect("sandbox watch bus lock poisoned") + .iter() + .filter(|(_, sender)| sender.receiver_count() > 0) + .map(|(sandbox_id, _)| sandbox_id.clone()) + .collect() + } +} + +/// Poll persisted sandbox resource versions once per gateway and notify the +/// existing in-memory watch bus when another replica changes a record. +/// +/// The poller performs at most one lookup per actively watched sandbox per +/// interval, regardless of how many clients are watching that sandbox. +pub fn spawn_store_poller( + store: Arc, + bus: SandboxWatchBus, + interval: Duration, + mut shutdown_rx: watch::Receiver, +) { + tokio::spawn(async move { + let mut known_versions: HashMap> = HashMap::new(); + let mut timer = tokio::time::interval(interval); + timer.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Skip); + + loop { + tokio::select! { + changed = shutdown_rx.changed() => { + if changed.is_err() || *shutdown_rx.borrow() { + break; + } + } + _ = timer.tick() => { + let active = bus.active_sandbox_ids(); + known_versions.retain(|sandbox_id, _| active.contains(sandbox_id)); + + for sandbox_id in active { + let current = match store.get_message::(&sandbox_id).await { + Ok(sandbox) => sandbox.map(|sandbox| { + sandbox.metadata.as_ref().map_or(0, |metadata| metadata.resource_version) + }), + Err(err) => { + tracing::warn!( + sandbox_id, + error = %err, + "sandbox watch poller: failed to read persisted sandbox" + ); + continue; + } + }; + + let changed = known_versions + .insert(sandbox_id.clone(), current) + .is_none_or(|previous| previous != current); + if changed { + bus.notify(&sandbox_id); + } + } + } + } + } + }); } /// Helper to translate broadcast lag into a gRPC status. @@ -72,6 +141,7 @@ pub fn broadcast_to_status(err: broadcast::error::RecvError) -> Status { #[cfg(test)] mod tests { use super::*; + use openshell_core::proto::datamodel::v1::ObjectMeta; #[test] fn sandbox_watch_bus_remove_cleans_up() { @@ -114,4 +184,42 @@ mod tests { // Should not panic bus.remove("nonexistent"); } + + #[tokio::test] + async fn shared_store_poller_notifies_remote_resource_version_change() { + let store = Arc::new(crate::persistence::test_store().await); + let bus = SandboxWatchBus::new(); + let sandbox = Sandbox { + metadata: Some(ObjectMeta { + id: "sb-1".to_string(), + name: "sandbox-a".to_string(), + workspace: "default".to_string(), + ..Default::default() + }), + ..Default::default() + }; + store.put_message(&sandbox).await.unwrap(); + + let mut rx = bus.subscribe("sb-1"); + let (shutdown_tx, shutdown_rx) = watch::channel(false); + spawn_store_poller(store.clone(), bus, Duration::from_millis(10), shutdown_rx); + + tokio::time::timeout(Duration::from_secs(1), rx.recv()) + .await + .expect("poller should publish its initial observation") + .unwrap(); + + store + .update_message_cas::("sb-1", 0, |stored| { + stored.set_phase(1); + }) + .await + .unwrap(); + tokio::time::timeout(Duration::from_secs(1), rx.recv()) + .await + .expect("poller should observe a remote store update") + .unwrap(); + + shutdown_tx.send(true).unwrap(); + } } diff --git a/crates/openshell-server/src/service_routing.rs b/crates/openshell-server/src/service_routing.rs index 3e80bc26f5..0c1cba886e 100644 --- a/crates/openshell-server/src/service_routing.rs +++ b/crates/openshell-server/src/service_routing.rs @@ -325,17 +325,16 @@ async fn proxy_to_endpoint( let websocket_upgrade = is_websocket_upgrade(&req); let downstream_upgrade = websocket_upgrade.then(|| hyper::upgrade::on(&mut req)); - let (_channel_id, relay_rx) = state - .supervisor_sessions - .open_relay_with_target( - sandbox.object_id(), - relay_open::Target::Tcp(TcpRelayTarget { - host: RELAY_TARGET_HOST.to_string(), - port: u32::from(target_port), - }), - endpoint.object_id().to_string(), - Duration::from_secs(15), - ) + let (_channel_id, relay_rx) = crate::supervisor_session::open_routed_relay_with_target( + &state, + sandbox.object_id(), + relay_open::Target::Tcp(TcpRelayTarget { + host: RELAY_TARGET_HOST.to_string(), + port: u32::from(target_port), + }), + endpoint.object_id().to_string(), + Duration::from_secs(15), + ) .await .map_err(|err| { warn!(error = %err, sandbox_id = %endpoint.sandbox_id, "sandbox service routing: supervisor relay unavailable"); diff --git a/crates/openshell-server/src/supervisor_owner.rs b/crates/openshell-server/src/supervisor_owner.rs new file mode 100644 index 0000000000..19d371d722 --- /dev/null +++ b/crates/openshell-server/src/supervisor_owner.rs @@ -0,0 +1,344 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! Shared supervisor-session ownership index for HA gateway replicas. + +use crate::persistence::{PersistenceError, Store, WriteCondition}; +use openshell_core::time::now_ms; +use serde::{Deserialize, Serialize}; +use std::sync::Arc; +use std::time::Duration; +use thiserror::Error; + +const OWNER_OBJECT_TYPE: &str = "supervisor_session_owner"; + +pub const OWNER_TTL: Duration = Duration::from_secs(45); + +fn owner_object_id(sandbox_id: &str) -> String { + format!("supervisor-owner:{sandbox_id}") +} + +#[derive(Debug, Error)] +pub enum OwnerError { + #[error("supervisor session is owned by another active gateway replica")] + AlreadyOwned, + #[error("supervisor owner record CAS conflict")] + Conflict, + #[error("persistence error: {0}")] + Store(#[from] PersistenceError), +} + +#[derive(Debug, Clone, Serialize, Deserialize)] +struct OwnerPayload { + sandbox_id: String, + session_id: String, + supervisor_instance_id: String, + connection_epoch: u64, + owner_replica_id: String, + owner_peer_endpoint: String, + connected_at_ms: i64, +} + +#[derive(Debug, Clone)] +pub struct OwnerRecord { + pub session_id: String, + pub supervisor_instance_id: String, + pub connection_epoch: u64, + pub owner_replica_id: String, + pub owner_peer_endpoint: String, + #[allow(dead_code)] + pub connected_at_ms: i64, + pub updated_at_ms: i64, + pub resource_version: u64, +} + +#[derive(Debug, Clone)] +pub struct OwnerGuard { + pub sandbox_id: String, + pub session_id: String, + pub supervisor_instance_id: String, + pub connection_epoch: u64, + pub owner_replica_id: String, + pub owner_peer_endpoint: String, + connected_at_ms: i64, + resource_version: u64, +} + +pub struct SupervisorOwnerIndex { + store: Arc, + ttl: Duration, +} + +impl SupervisorOwnerIndex { + pub fn new(store: Arc, ttl: Duration) -> Self { + Self { store, ttl } + } + + pub async fn publish( + &self, + sandbox_id: &str, + session_id: &str, + supervisor_instance_id: &str, + connection_epoch: u64, + owner_replica_id: &str, + owner_peer_endpoint: &str, + ) -> Result { + let connected_at_ms = now_ms(); + let payload = OwnerPayload { + sandbox_id: sandbox_id.to_string(), + session_id: session_id.to_string(), + supervisor_instance_id: supervisor_instance_id.to_string(), + connection_epoch, + owner_replica_id: owner_replica_id.to_string(), + owner_peer_endpoint: owner_peer_endpoint.to_string(), + connected_at_ms, + }; + + let condition = match self.read(sandbox_id).await? { + None => WriteCondition::MustCreate, + Some(existing) + if can_supersede( + &existing, + supervisor_instance_id, + connection_epoch, + self.ttl, + ) => + { + WriteCondition::MatchResourceVersion(existing.resource_version) + } + Some(_) => return Err(OwnerError::AlreadyOwned), + }; + + let result = self.write_payload(sandbox_id, &payload, condition).await?; + Ok(OwnerGuard { + sandbox_id: sandbox_id.to_string(), + session_id: session_id.to_string(), + supervisor_instance_id: supervisor_instance_id.to_string(), + connection_epoch, + owner_replica_id: owner_replica_id.to_string(), + owner_peer_endpoint: owner_peer_endpoint.to_string(), + connected_at_ms, + resource_version: result.resource_version, + }) + } + + pub async fn renew(&self, guard: &mut OwnerGuard) -> Result<(), OwnerError> { + let payload = OwnerPayload { + sandbox_id: guard.sandbox_id.clone(), + session_id: guard.session_id.clone(), + supervisor_instance_id: guard.supervisor_instance_id.clone(), + connection_epoch: guard.connection_epoch, + owner_replica_id: guard.owner_replica_id.clone(), + owner_peer_endpoint: guard.owner_peer_endpoint.clone(), + connected_at_ms: guard.connected_at_ms, + }; + + match self + .write_payload( + &guard.sandbox_id, + &payload, + WriteCondition::MatchResourceVersion(guard.resource_version), + ) + .await + { + Ok(result) => { + guard.resource_version = result.resource_version; + Ok(()) + } + Err(OwnerError::Store(PersistenceError::Conflict { .. })) => Err(OwnerError::Conflict), + Err(err) => Err(err), + } + } + + pub async fn release_if_current(&self, guard: &OwnerGuard) -> Result<(), OwnerError> { + let Some(record) = self.read(&guard.sandbox_id).await? else { + return Ok(()); + }; + if record.session_id != guard.session_id + || record.owner_replica_id != guard.owner_replica_id + { + return Ok(()); + } + match self + .store + .delete_if( + OWNER_OBJECT_TYPE, + &owner_object_id(&guard.sandbox_id), + record.resource_version, + ) + .await + { + Ok(_) => Ok(()), + Err(PersistenceError::Conflict { .. }) => Err(OwnerError::Conflict), + Err(err) => Err(OwnerError::Store(err)), + } + } + + pub async fn read(&self, sandbox_id: &str) -> Result, OwnerError> { + let Some(record) = self + .store + .get(OWNER_OBJECT_TYPE, &owner_object_id(sandbox_id)) + .await + .map_err(OwnerError::Store)? + else { + return Ok(None); + }; + + let payload: OwnerPayload = serde_json::from_slice(&record.payload) + .map_err(|err| PersistenceError::Decode(err.to_string()))?; + Ok(Some(OwnerRecord { + session_id: payload.session_id, + supervisor_instance_id: payload.supervisor_instance_id, + connection_epoch: payload.connection_epoch, + owner_replica_id: payload.owner_replica_id, + owner_peer_endpoint: payload.owner_peer_endpoint, + connected_at_ms: payload.connected_at_ms, + updated_at_ms: record.updated_at_ms, + resource_version: record.resource_version, + })) + } + + async fn write_payload( + &self, + sandbox_id: &str, + payload: &OwnerPayload, + condition: WriteCondition, + ) -> Result { + let payload_bytes = + serde_json::to_vec(payload).map_err(|err| PersistenceError::Encode(err.to_string())); + let payload_bytes = payload_bytes.map_err(OwnerError::Store)?; + match self + .store + .put_if( + OWNER_OBJECT_TYPE, + &owner_object_id(sandbox_id), + sandbox_id, + "", + &payload_bytes, + None, + condition, + ) + .await + { + Ok(result) => Ok(result), + Err(PersistenceError::UniqueViolation { .. }) => Err(OwnerError::AlreadyOwned), + Err(PersistenceError::Conflict { .. }) => Err(OwnerError::Conflict), + Err(err) => Err(OwnerError::Store(err)), + } + } +} + +fn can_supersede( + existing: &OwnerRecord, + supervisor_instance_id: &str, + connection_epoch: u64, + ttl: Duration, +) -> bool { + let age_ms = now_ms() - existing.updated_at_ms; + let ttl_ms = i64::try_from(ttl.as_millis()).unwrap_or(i64::MAX); + if age_ms >= ttl_ms { + return true; + } + + existing.supervisor_instance_id == supervisor_instance_id + && connection_epoch > existing.connection_epoch +} + +#[cfg(test)] +mod tests { + use super::*; + + async fn test_index(ttl: Duration) -> SupervisorOwnerIndex { + let store = Arc::new(crate::persistence::test_store().await); + SupervisorOwnerIndex::new(store, ttl) + } + + #[tokio::test] + async fn publish_creates_owner() { + let index = test_index(OWNER_TTL).await; + let guard = index + .publish("sbx", "s1", "inst", 1, "gw-1", "http://gw-1") + .await + .unwrap(); + let record = index.read("sbx").await.unwrap().unwrap(); + assert_eq!(record.session_id, guard.session_id); + assert_eq!(record.owner_replica_id, "gw-1"); + } + + #[tokio::test] + async fn publish_does_not_collide_with_sandbox_object_id() { + let index = test_index(OWNER_TTL).await; + index + .store + .put("sandbox", "sbx", "sandbox-a", "default", br"{}", None) + .await + .unwrap(); + + index + .publish("sbx", "s1", "inst", 1, "gw-1", "http://gw-1") + .await + .unwrap(); + + let record = index.read("sbx").await.unwrap().unwrap(); + assert_eq!(record.session_id, "s1"); + assert_eq!(record.owner_replica_id, "gw-1"); + } + + #[tokio::test] + async fn publish_rejects_active_different_instance() { + let index = test_index(OWNER_TTL).await; + index + .publish("sbx", "s1", "inst-a", 1, "gw-1", "http://gw-1") + .await + .unwrap(); + let err = index + .publish("sbx", "s2", "inst-b", 1, "gw-2", "http://gw-2") + .await + .unwrap_err(); + assert!(matches!(err, OwnerError::AlreadyOwned)); + } + + #[tokio::test] + async fn publish_supersedes_same_instance_higher_epoch() { + let index = test_index(OWNER_TTL).await; + index + .publish("sbx", "s1", "inst", 1, "gw-1", "http://gw-1") + .await + .unwrap(); + let guard = index + .publish("sbx", "s2", "inst", 2, "gw-2", "http://gw-2") + .await + .unwrap(); + let record = index.read("sbx").await.unwrap().unwrap(); + assert_eq!(record.session_id, guard.session_id); + assert_eq!(record.owner_replica_id, "gw-2"); + } + + #[tokio::test] + async fn release_if_current_ignores_stale_guard() { + let index = test_index(OWNER_TTL).await; + let old = index + .publish("sbx", "s1", "inst", 1, "gw-1", "http://gw-1") + .await + .unwrap(); + let new = index + .publish("sbx", "s2", "inst", 2, "gw-2", "http://gw-2") + .await + .unwrap(); + index.release_if_current(&old).await.unwrap(); + let record = index.read("sbx").await.unwrap().unwrap(); + assert_eq!(record.session_id, new.session_id); + } + + #[tokio::test] + async fn renew_updates_resource_version() { + let index = test_index(OWNER_TTL).await; + let mut guard = index + .publish("sbx", "s1", "inst", 1, "gw-1", "http://gw-1") + .await + .unwrap(); + let before = guard.resource_version; + index.renew(&mut guard).await.unwrap(); + assert!(guard.resource_version > before); + } +} diff --git a/crates/openshell-server/src/supervisor_session.rs b/crates/openshell-server/src/supervisor_session.rs index fbff0e276c..7c90bf8e13 100644 --- a/crates/openshell-server/src/supervisor_session.rs +++ b/crates/openshell-server/src/supervisor_session.rs @@ -8,19 +8,23 @@ use std::time::{Duration, Instant}; use tokio::sync::{mpsc, oneshot}; use tokio_stream::wrappers::ReceiverStream; +use tonic::metadata::{Ascii, MetadataValue}; +use tonic::transport::{Certificate, Channel, ClientTlsConfig, Endpoint, Identity}; use tonic::{Request, Response, Status}; use tracing::{debug, info, warn}; use uuid::Uuid; use openshell_core::proto::{ - GatewayMessage, RelayFrame, RelayInit, RelayOpen, ReportMainProcessExitRequest, - ReportMainProcessExitResponse, Sandbox, SandboxPhase, SessionAccepted, SshRelayTarget, - SupervisorMessage, gateway_message, relay_open, supervisor_message, + GatewayMessage, PeerRelayFrame, PeerRelayInit, RelayFrame, RelayInit, RelayOpen, + ReportMainProcessExitRequest, ReportMainProcessExitResponse, Sandbox, SandboxPhase, + SessionAccepted, SshRelayTarget, SupervisorMessage, gateway_message, open_shell_client, + peer_relay_frame, relay_open, supervisor_message, }; use openshell_core::transport_errors::is_expected_transport_close_status; use crate::ServerState; use crate::auth::principal::Principal; +use crate::supervisor_owner::{OWNER_TTL, OwnerError, OwnerGuard, SupervisorOwnerIndex}; const HEARTBEAT_INTERVAL_SECS: u32 = 15; const RELAY_PENDING_TIMEOUT: Duration = Duration::from_secs(10); @@ -38,6 +42,79 @@ const MAX_PENDING_RELAYS: usize = 256; /// consume the entire global budget. Sits above the SSH-tunnel per-sandbox /// cap (20) so tunnel-specific limits still fire first for that caller. const MAX_PENDING_RELAYS_PER_SANDBOX: usize = 32; +const PEER_TLS_CA_FILE_ENV: &str = "OPENSHELL_PEER_TLS_CA_FILE"; +const PEER_TLS_CERT_FILE_ENV: &str = "OPENSHELL_PEER_TLS_CERT_FILE"; +const PEER_TLS_KEY_FILE_ENV: &str = "OPENSHELL_PEER_TLS_KEY_FILE"; +const PEER_TLS_SERVER_NAME_ENV: &str = "OPENSHELL_PEER_TLS_SERVER_NAME"; + +#[derive(Debug, Default)] +struct PeerTlsClientConfig { + ca_file: Option, + cert_file: Option, + key_file: Option, + server_name: Option, +} + +impl PeerTlsClientConfig { + fn from_env() -> Self { + Self { + ca_file: nonempty_env(PEER_TLS_CA_FILE_ENV).map(Into::into), + cert_file: nonempty_env(PEER_TLS_CERT_FILE_ENV).map(Into::into), + key_file: nonempty_env(PEER_TLS_KEY_FILE_ENV).map(Into::into), + server_name: nonempty_env(PEER_TLS_SERVER_NAME_ENV), + } + } + + fn load(&self) -> Result { + let mut tls = if let Some(path) = self.ca_file.as_deref() { + let pem = std::fs::read(path).map_err(|err| { + Status::failed_precondition(format!( + "failed to read gateway peer TLS CA {}: {err}", + path.display() + )) + })?; + ClientTlsConfig::new().ca_certificate(Certificate::from_pem(pem)) + } else { + ClientTlsConfig::new().with_native_roots() + }; + + match (self.cert_file.as_deref(), self.key_file.as_deref()) { + (Some(cert_path), Some(key_path)) => { + let cert = std::fs::read(cert_path).map_err(|err| { + Status::failed_precondition(format!( + "failed to read gateway peer TLS certificate {}: {err}", + cert_path.display() + )) + })?; + let key = std::fs::read(key_path).map_err(|err| { + Status::failed_precondition(format!( + "failed to read gateway peer TLS key {}: {err}", + key_path.display() + )) + })?; + tls = tls.identity(Identity::from_pem(cert, key)); + } + (None, None) => {} + _ => { + return Err(Status::failed_precondition(format!( + "{PEER_TLS_CERT_FILE_ENV} and {PEER_TLS_KEY_FILE_ENV} must be configured together" + ))); + } + } + + if let Some(server_name) = self.server_name.as_deref() { + tls = tls.domain_name(server_name); + } + Ok(tls) + } +} + +fn nonempty_env(name: &str) -> Option { + std::env::var(name) + .ok() + .map(|value| value.trim().to_string()) + .filter(|value| !value.is_empty()) +} // --------------------------------------------------------------------------- // Session registry @@ -282,16 +359,36 @@ impl SupervisorSessionRegistry { ), Status, > { - let tx = self - .wait_for_session(sandbox_id, session_wait_timeout) - .await?; - let channel_id = Uuid::new_v4().to_string(); let relay_open = RelayOpen { channel_id: channel_id.clone(), target: Some(target), service_id, }; + self.open_relay_with_message(sandbox_id, relay_open, session_wait_timeout) + .await + } + + pub async fn open_relay_with_message( + &self, + sandbox_id: &str, + relay_open: RelayOpen, + session_wait_timeout: Duration, + ) -> Result< + ( + String, + oneshot::Receiver>, + ), + Status, + > { + if relay_open.channel_id.is_empty() { + return Err(Status::invalid_argument("relay channel_id is required")); + } + let tx = self + .wait_for_session(sandbox_id, session_wait_timeout) + .await?; + + let channel_id = relay_open.channel_id.clone(); // Register the pending relay before sending RelayOpen to avoid a race. // Both caps are checked and the insert happens under a single lock hold @@ -453,6 +550,18 @@ pub fn spawn_relay_reaper(state: Arc, interval: Duration) { }); } +fn owner_error_to_status(err: OwnerError) -> Status { + match err { + OwnerError::AlreadyOwned => { + Status::unavailable("supervisor session owned by another gateway replica") + } + OwnerError::Conflict => Status::aborted("supervisor owner record changed concurrently"), + OwnerError::Store(err) => { + Status::internal(format!("supervisor owner persistence failed: {err}")) + } + } +} + async fn require_persisted_sandbox( store: &Arc, sandbox_id: &str, @@ -672,6 +781,422 @@ async fn expected_transport_close_during_session_teardown( ) } +// --------------------------------------------------------------------------- +// PeerRelay gRPC handler and client-side forwarding +// --------------------------------------------------------------------------- + +#[derive(Clone)] +struct PeerAuthInterceptor { + bearer: MetadataValue, + replica_id: MetadataValue, +} + +impl PeerAuthInterceptor { + fn new(token: &str, replica_id: &str) -> Result { + let bearer = MetadataValue::try_from(format!("Bearer {token}")) + .map_err(|_| Status::internal("invalid gateway peer SA token header value"))?; + let replica_id = MetadataValue::try_from(replica_id.to_string()) + .map_err(|_| Status::internal("invalid gateway replica id header value"))?; + Ok(Self { bearer, replica_id }) + } +} + +impl tonic::service::Interceptor for PeerAuthInterceptor { + fn call(&mut self, mut req: Request<()>) -> Result, Status> { + req.metadata_mut() + .insert("authorization", self.bearer.clone()); + req.metadata_mut() + .insert("x-openshell-peer-replica", self.replica_id.clone()); + Ok(req) + } +} + +async fn build_peer_channel(endpoint: &str) -> Result { + let mut ep = Endpoint::from_shared(endpoint.to_string()) + .map_err(|err| Status::internal(format!("invalid gateway peer endpoint: {err}")))? + .connect_timeout(Duration::from_secs(10)) + .http2_keep_alive_interval(Duration::from_secs(10)) + .keep_alive_while_idle(true) + .keep_alive_timeout(Duration::from_secs(20)) + .http2_adaptive_window(true); + + if endpoint.starts_with("https://") { + let peer_tls = PeerTlsClientConfig::from_env().load()?; + ep = ep + .tls_config(peer_tls) + .map_err(|err| Status::internal(format!("failed to configure peer TLS: {err}")))?; + } + + ep.connect() + .await + .map_err(|err| Status::unavailable(format!("gateway peer connection failed: {err}"))) +} + +pub async fn open_routed_relay_with_target( + state: &Arc, + sandbox_id: &str, + target: relay_open::Target, + service_id: String, + session_wait_timeout: Duration, +) -> Result< + ( + String, + oneshot::Receiver>, + ), + Status, +> { + let channel_id = Uuid::new_v4().to_string(); + let relay_open = RelayOpen { + channel_id: channel_id.clone(), + target: Some(target), + service_id, + }; + open_routed_relay_with_message(state, sandbox_id, relay_open, session_wait_timeout).await +} + +pub async fn open_routed_relay_with_message( + state: &Arc, + sandbox_id: &str, + relay_open: RelayOpen, + session_wait_timeout: Duration, +) -> Result< + ( + String, + oneshot::Receiver>, + ), + Status, +> { + let deadline = Instant::now() + session_wait_timeout; + let mut backoff = SESSION_WAIT_INITIAL_BACKOFF; + let owner_index = SupervisorOwnerIndex::new(state.store.clone(), OWNER_TTL); + loop { + if state.supervisor_sessions.has_session(sandbox_id) { + match state + .supervisor_sessions + .open_relay_with_message(sandbox_id, relay_open.clone(), Duration::ZERO) + .await + { + Ok(relay) => return Ok(relay), + Err(status) if status.code() == tonic::Code::Unavailable => { + // The session can migrate after `has_session` but before + // RelayOpen reaches its sender. Fall through and reread the + // persisted owner instead of surfacing a handoff race. + warn!( + sandbox_id, + error = %status, + "local supervisor relay disappeared during open; resolving owner again" + ); + } + Err(status) => return Err(status), + } + } + + if let Some(owner) = owner_index + .read(sandbox_id) + .await + .map_err(owner_error_to_status)? + && owner_is_fresh(&owner) + { + if owner.owner_replica_id == state.replica_id { + warn!( + sandbox_id, + owner_replica_id = %owner.owner_replica_id, + "supervisor owner record points at this replica but no local session is registered; retrying" + ); + if Instant::now() + backoff > deadline { + return Err(Status::unavailable("supervisor session not connected")); + } + tokio::time::sleep(backoff).await; + backoff = (backoff * 2).min(SESSION_WAIT_MAX_BACKOFF); + continue; + } + match open_peer_relay( + state, + owner.owner_peer_endpoint.clone(), + sandbox_id, + relay_open.clone(), + ) + .await + { + Ok(relay) => return Ok(relay), + Err(status) => { + warn!( + sandbox_id, + owner_replica_id = %owner.owner_replica_id, + owner_peer_endpoint = %owner.owner_peer_endpoint, + error = %status, + "gateway peer owner relay open failed; retrying until session wait timeout" + ); + } + } + } + + if Instant::now() + backoff > deadline { + return Err(Status::unavailable("supervisor session not connected")); + } + tokio::time::sleep(backoff).await; + backoff = (backoff * 2).min(SESSION_WAIT_MAX_BACKOFF); + } +} + +fn owner_is_fresh(owner: &crate::supervisor_owner::OwnerRecord) -> bool { + let age_ms = openshell_core::time::now_ms() - owner.updated_at_ms; + let ttl_ms = i64::try_from(OWNER_TTL.as_millis()).unwrap_or(i64::MAX); + age_ms < ttl_ms +} + +async fn open_peer_relay( + state: &Arc, + owner_peer_endpoint: String, + sandbox_id: &str, + relay_open: RelayOpen, +) -> Result< + ( + String, + oneshot::Receiver>, + ), + Status, +> { + let channel_id = relay_open.channel_id.clone(); + let (relay_tx, relay_rx) = oneshot::channel(); + let stream = connect_peer_relay(state, &owner_peer_endpoint, sandbox_id, relay_open).await?; + let _ = relay_tx.send(Ok(stream)); + Ok((channel_id, relay_rx)) +} + +async fn connect_peer_relay( + state: &Arc, + owner_peer_endpoint: &str, + sandbox_id: &str, + relay_open: RelayOpen, +) -> Result { + let token = crate::auth::peer::load_peer_service_account_token_from_env() + .map_err(|err| { + Status::failed_precondition(format!("gateway peer token load failed: {err}")) + })? + .ok_or_else(|| { + Status::failed_precondition("gateway peer ServiceAccount token is not configured") + })?; + let channel = build_peer_channel(owner_peer_endpoint).await?; + let interceptor = PeerAuthInterceptor::new(&token, &state.replica_id)?; + let mut client = open_shell_client::OpenShellClient::with_interceptor(channel, interceptor); + + let (out_tx, out_rx) = mpsc::channel::(16); + out_tx + .send(PeerRelayFrame { + payload: Some(peer_relay_frame::Payload::Init(PeerRelayInit { + sandbox_id: sandbox_id.to_string(), + relay_open: Some(relay_open), + requester_replica_id: state.replica_id.clone(), + })), + }) + .await + .map_err(|_| Status::internal("failed to initialize peer relay stream"))?; + + let response = client + .peer_relay(ReceiverStream::new(out_rx)) + .await + .map_err(|err| Status::unavailable(format!("gateway peer relay RPC failed: {err}")))?; + let inbound = response.into_inner(); + let (gateway_stream, bridge_stream) = tokio::io::duplex(64 * 1024); + spawn_peer_bridge(bridge_stream, inbound, out_tx, sandbox_id.to_string()); + Ok(gateway_stream) +} + +pub async fn handle_peer_relay( + state: &Arc, + request: Request>, +) -> Result< + Response< + Pin> + Send + 'static>>, + >, + Status, +> { + let peer = match request.extensions().get::() { + Some(Principal::Peer(peer)) => peer.clone(), + _ => { + return Err(Status::permission_denied( + "gateway peer principal is required", + )); + } + }; + let mut inbound = request.into_inner(); + + let first = inbound + .message() + .await? + .ok_or_else(|| Status::invalid_argument("empty PeerRelay stream"))?; + let Some(peer_relay_frame::Payload::Init(init)) = first.payload else { + return Err(Status::invalid_argument( + "first PeerRelayFrame must be init", + )); + }; + if init.sandbox_id.is_empty() { + return Err(Status::invalid_argument("sandbox_id is required")); + } + let relay_open = init + .relay_open + .ok_or_else(|| Status::invalid_argument("relay_open is required"))?; + if relay_open.channel_id.is_empty() { + return Err(Status::invalid_argument("relay channel_id is required")); + } + + info!( + sandbox_id = %init.sandbox_id, + channel_id = %relay_open.channel_id, + requester = %peer.replica_id, + "gateway peer relay: opening local supervisor relay" + ); + + let (channel_id, relay_rx) = state + .supervisor_sessions + .open_relay_with_message(&init.sandbox_id, relay_open, Duration::from_secs(5)) + .await?; + let supervisor_stream = match tokio::time::timeout(Duration::from_secs(10), relay_rx).await { + Ok(Ok(Ok(stream))) => stream, + Ok(Ok(Err(status))) => return Err(status), + Ok(Err(_)) => return Err(Status::unavailable("relay channel dropped")), + Err(_) => return Err(Status::deadline_exceeded("relay open timed out")), + }; + + let (out_tx, out_rx) = mpsc::channel::>(16); + spawn_peer_owner_bridge( + supervisor_stream, + inbound, + out_tx, + init.sandbox_id, + channel_id, + ); + let stream: Pin< + Box> + Send + 'static>, + > = Box::pin(ReceiverStream::new(out_rx)); + Ok(Response::new(stream)) +} + +fn spawn_peer_bridge( + bridge_stream: tokio::io::DuplexStream, + mut inbound: tonic::Streaming, + out_tx: mpsc::Sender, + sandbox_id: String, +) { + let (mut read_half, mut write_half) = tokio::io::split(bridge_stream); + let sandbox_id_in = sandbox_id.clone(); + tokio::spawn(async move { + loop { + match inbound.message().await { + Ok(Some(frame)) => { + let Some(peer_relay_frame::Payload::Data(data)) = frame.payload else { + warn!(sandbox_id = %sandbox_id_in, "gateway peer relay: non-data frame after init"); + break; + }; + if data.is_empty() { + continue; + } + if let Err(err) = + tokio::io::AsyncWriteExt::write_all(&mut write_half, &data).await + { + warn!(sandbox_id = %sandbox_id_in, error = %err, "gateway peer relay: write to duplex failed"); + break; + } + } + Ok(None) => break, + Err(err) => { + warn!(sandbox_id = %sandbox_id_in, error = %err, "gateway peer relay: inbound errored"); + break; + } + } + } + let _ = tokio::io::AsyncWriteExt::shutdown(&mut write_half).await; + }); + + tokio::spawn(async move { + let mut buf = vec![0u8; RELAY_STREAM_CHUNK_SIZE]; + loop { + match tokio::io::AsyncReadExt::read(&mut read_half, &mut buf).await { + Ok(0) => break, + Ok(n) => { + if out_tx + .send(PeerRelayFrame { + payload: Some(peer_relay_frame::Payload::Data(buf[..n].to_vec())), + }) + .await + .is_err() + { + break; + } + } + Err(err) => { + warn!(sandbox_id = %sandbox_id, error = %err, "gateway peer relay: read from duplex failed"); + break; + } + } + } + }); +} + +fn spawn_peer_owner_bridge( + supervisor_stream: tokio::io::DuplexStream, + mut inbound: tonic::Streaming, + out_tx: mpsc::Sender>, + sandbox_id: String, + channel_id: String, +) { + let (mut read_half, mut write_half) = tokio::io::split(supervisor_stream); + let sandbox_id_in = sandbox_id.clone(); + let channel_id_in = channel_id.clone(); + tokio::spawn(async move { + loop { + match inbound.message().await { + Ok(Some(frame)) => { + let Some(peer_relay_frame::Payload::Data(data)) = frame.payload else { + warn!(sandbox_id = %sandbox_id_in, channel_id = %channel_id_in, "gateway peer relay owner: non-data frame after init"); + break; + }; + if data.is_empty() { + continue; + } + if let Err(err) = + tokio::io::AsyncWriteExt::write_all(&mut write_half, &data).await + { + warn!(sandbox_id = %sandbox_id_in, channel_id = %channel_id_in, error = %err, "gateway peer relay owner: write to supervisor relay failed"); + break; + } + } + Ok(None) => break, + Err(err) => { + warn!(sandbox_id = %sandbox_id_in, channel_id = %channel_id_in, error = %err, "gateway peer relay owner: inbound errored"); + break; + } + } + } + let _ = tokio::io::AsyncWriteExt::shutdown(&mut write_half).await; + }); + + tokio::spawn(async move { + let mut buf = vec![0u8; RELAY_STREAM_CHUNK_SIZE]; + loop { + match tokio::io::AsyncReadExt::read(&mut read_half, &mut buf).await { + Ok(0) => break, + Ok(n) => { + if out_tx + .send(Ok(PeerRelayFrame { + payload: Some(peer_relay_frame::Payload::Data(buf[..n].to_vec())), + })) + .await + .is_err() + { + break; + } + } + Err(err) => { + warn!(sandbox_id = %sandbox_id, channel_id = %channel_id, error = %err, "gateway peer relay owner: read from supervisor relay failed"); + break; + } + } + } + }); +} + // --------------------------------------------------------------------------- // ConnectSupervisor gRPC handler // --------------------------------------------------------------------------- @@ -707,10 +1232,35 @@ pub async fn handle_connect_supervisor( require_persisted_sandbox(&state.store, &sandbox_id).await?; let session_id = Uuid::new_v4().to_string(); + let owner_peer_endpoint = state.peer_endpoint.clone().unwrap_or_default(); + if !state.store.is_single_replica() && owner_peer_endpoint.is_empty() { + return Err(Status::failed_precondition( + "gateway peer endpoint is required for multi-replica supervisor ownership", + )); + } + let owner_peer_endpoint = if owner_peer_endpoint.is_empty() { + format!("local://{}", state.replica_id) + } else { + owner_peer_endpoint + }; + let owner_index = SupervisorOwnerIndex::new(state.store.clone(), OWNER_TTL); + let owner_guard = owner_index + .publish( + &sandbox_id, + &session_id, + &hello.instance_id, + hello.connection_epoch, + &state.replica_id, + &owner_peer_endpoint, + ) + .await + .map_err(owner_error_to_status)?; info!( sandbox_id = %sandbox_id, session_id = %session_id, instance_id = %hello.instance_id, + connection_epoch = hello.connection_epoch, + replica_id = %state.replica_id, "supervisor session: accepted" ); @@ -744,6 +1294,9 @@ pub async fn handle_connect_supervisor( state .supervisor_sessions .remove_if_current(&sandbox_id, &session_id); + if let Err(err) = owner_index.release_if_current(&owner_guard).await { + warn!(sandbox_id = %sandbox_id, session_id = %session_id, error = %err, "supervisor session: failed to release owner after accept send failure"); + } return Err(Status::internal("failed to send session accepted")); } @@ -773,6 +1326,7 @@ pub async fn handle_connect_supervisor( let state_clone = Arc::clone(state); let sandbox_id_clone = sandbox_id.clone(); tokio::spawn(async move { + let mut owner_guard = owner_guard; run_session_loop( &state_clone, &sandbox_id_clone, @@ -780,11 +1334,16 @@ pub async fn handle_connect_supervisor( &tx, &mut inbound, shutdown_rx, + &mut owner_guard, ) .await; let still_ours = state_clone .supervisor_sessions .remove_if_current(&sandbox_id_clone, &session_id); + let owner_index = SupervisorOwnerIndex::new(state_clone.store.clone(), OWNER_TTL); + if let Err(err) = owner_index.release_if_current(&owner_guard).await { + warn!(sandbox_id = %sandbox_id_clone, session_id = %session_id, error = %err, "supervisor session: failed to release owner record"); + } if still_ours { info!(sandbox_id = %sandbox_id_clone, session_id = %session_id, "supervisor session: ended"); state_clone @@ -846,6 +1405,7 @@ async fn run_session_loop( tx: &mpsc::Sender, inbound: &mut tonic::Streaming, mut shutdown_rx: oneshot::Receiver<()>, + owner_guard: &mut OwnerGuard, ) { let heartbeat_interval = Duration::from_secs(u64::from(HEARTBEAT_INTERVAL_SECS)); let mut heartbeat_timer = tokio::time::interval(heartbeat_interval); @@ -861,7 +1421,9 @@ async fn run_session_loop( msg = inbound.message() => { match msg { Ok(Some(msg)) => { - handle_supervisor_message(state, sandbox_id, session_id, msg); + if !handle_supervisor_message(state, sandbox_id, session_id, msg, owner_guard).await { + break; + } } Ok(None) => { info!(sandbox_id = %sandbox_id, session_id = %session_id, "supervisor session: stream closed by supervisor"); @@ -904,15 +1466,25 @@ async fn run_session_loop( } } -fn handle_supervisor_message( +async fn handle_supervisor_message( state: &Arc, sandbox_id: &str, session_id: &str, msg: SupervisorMessage, -) { + owner_guard: &mut OwnerGuard, +) -> bool { match msg.payload { Some(supervisor_message::Payload::Heartbeat(_)) => { - // Heartbeat received — nothing to do for now. + let owner_index = SupervisorOwnerIndex::new(state.store.clone(), OWNER_TTL); + if let Err(err) = owner_index.renew(owner_guard).await { + warn!( + sandbox_id = %sandbox_id, + session_id = %session_id, + error = %err, + "supervisor session: owner renewal failed; closing session" + ); + return false; + } } Some(supervisor_message::Payload::RelayOpenResult(result)) => { if result.success { @@ -953,6 +1525,7 @@ fn handle_supervisor_message( ); } } + true } // --------------------------------------------------------------------------- @@ -978,6 +1551,42 @@ mod tests { oneshot::channel::<()>().0 } + #[test] + fn peer_tls_client_config_requires_certificate_and_key_together() { + let config = PeerTlsClientConfig { + cert_file: Some("client.crt".into()), + ..Default::default() + }; + + let err = config + .load() + .expect_err("an incomplete peer mTLS identity must fail closed"); + assert_eq!(err.code(), tonic::Code::FailedPrecondition); + assert!(err.message().contains(PEER_TLS_KEY_FILE_ENV)); + } + + #[test] + fn peer_tls_client_config_loads_chart_ca_identity_and_server_name() { + let dir = tempfile::tempdir().unwrap(); + let ca = dir.path().join("ca.crt"); + let cert = dir.path().join("tls.crt"); + let key = dir.path().join("tls.key"); + std::fs::write(&ca, b"test-ca").unwrap(); + std::fs::write(&cert, b"test-cert").unwrap(); + std::fs::write(&key, b"test-key").unwrap(); + + let config = PeerTlsClientConfig { + ca_file: Some(ca), + cert_file: Some(cert), + key_file: Some(key), + server_name: Some("openshell.openshell.svc.cluster.local".to_string()), + }; + + config + .load() + .expect("complete chart peer TLS materials should configure tonic"); + } + fn sandbox_record(id: &str, name: &str) -> Sandbox { Sandbox { metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta { diff --git a/crates/openshell-server/tests/common/mod.rs b/crates/openshell-server/tests/common/mod.rs index 96b620a230..5e1a7215c0 100644 --- a/crates/openshell-server/tests/common/mod.rs +++ b/crates/openshell-server/tests/common/mod.rs @@ -22,10 +22,10 @@ use openshell_core::proto::{ GetSandboxConfigResponse, GetSandboxProviderEnvironmentRequest, GetSandboxProviderEnvironmentResponse, GetSandboxRequest, HealthRequest, HealthResponse, IssueSandboxTokenRequest, IssueSandboxTokenResponse, ListProvidersRequest, - ListProvidersResponse, ListSandboxesRequest, ListSandboxesResponse, ProviderResponse, - RefreshSandboxTokenRequest, RefreshSandboxTokenResponse, RelayFrame, RevokeSshSessionRequest, - RevokeSshSessionResponse, SandboxResponse, SandboxStreamEvent, ServiceStatus, - SupervisorMessage, TcpForwardFrame, UpdateProviderRequest, WatchSandboxRequest, + ListProvidersResponse, ListSandboxesRequest, ListSandboxesResponse, PeerRelayFrame, + ProviderResponse, RefreshSandboxTokenRequest, RefreshSandboxTokenResponse, RelayFrame, + RevokeSshSessionRequest, RevokeSshSessionResponse, SandboxResponse, SandboxStreamEvent, + ServiceStatus, SupervisorMessage, TcpForwardFrame, UpdateProviderRequest, WatchSandboxRequest, open_shell_client::OpenShellClient, open_shell_server::{OpenShell, OpenShellServer}, }; @@ -507,6 +507,15 @@ impl OpenShell for TestOpenShell { Err(Status::unimplemented("not implemented in test")) } + type PeerRelayStream = ReceiverStream>; + + async fn peer_relay( + &self, + _request: tonic::Request>, + ) -> Result, Status> { + Err(Status::unimplemented("not implemented in test")) + } + type ForwardTcpStream = std::pin::Pin> + Send>>; diff --git a/crates/openshell-server/tests/supervisor_relay_integration.rs b/crates/openshell-server/tests/supervisor_relay_integration.rs index 32ef513adb..d7822fa7de 100644 --- a/crates/openshell-server/tests/supervisor_relay_integration.rs +++ b/crates/openshell-server/tests/supervisor_relay_integration.rs @@ -23,7 +23,7 @@ use hyper_util::{ server::conn::auto::Builder, }; use openshell_core::proto::{ - GatewayMessage, RelayFrame, RelayInit, SupervisorMessage, TcpForwardFrame, + GatewayMessage, PeerRelayFrame, RelayFrame, RelayInit, SupervisorMessage, TcpForwardFrame, open_shell_client::OpenShellClient, open_shell_server::{OpenShell, OpenShellServer}, }; @@ -75,6 +75,16 @@ impl OpenShell for RelayGateway { // ------ unused stubs ------ + type PeerRelayStream = + std::pin::Pin> + Send>>; + + async fn peer_relay( + &self, + _: tonic::Request>, + ) -> Result, Status> { + Err(Status::unimplemented("unused")) + } + type ConnectSupervisorStream = ReceiverStream>; async fn connect_supervisor( &self, diff --git a/crates/openshell-supervisor-process/src/supervisor_session.rs b/crates/openshell-supervisor-process/src/supervisor_session.rs index e8a140e483..24668dddcb 100644 --- a/crates/openshell-supervisor-process/src/supervisor_session.rs +++ b/crates/openshell-supervisor-process/src/supervisor_session.rs @@ -315,8 +315,9 @@ async fn run_session_loop( &ssh_socket_path, netns_fd, expected_ssh_peer_pid, - Arc::clone(&terminating), &instance_id, + attempt, + Arc::clone(&terminating), ) .await { @@ -341,14 +342,16 @@ async fn run_session_loop( } } +#[allow(clippy::too_many_arguments)] async fn run_single_session( endpoint: &str, sandbox_id: &str, ssh_socket_path: &std::path::Path, netns_fd: Option, expected_ssh_peer_pid: Option, - terminating: Arc, instance_id: &str, + connection_epoch: u64, + terminating: Arc, ) -> Result<(), Box> { // Connect to the gateway. The same `Channel` is used for both the // long-lived control stream and all data-plane `RelayStream` calls, so @@ -368,6 +371,7 @@ async fn run_single_session( payload: Some(supervisor_message::Payload::Hello(SupervisorHello { sandbox_id: sandbox_id.to_string(), instance_id: instance_id.to_string(), + connection_epoch, })), }) .await diff --git a/deploy/helm/openshell/README.md b/deploy/helm/openshell/README.md index 5879d617ff..950bd74d4b 100644 --- a/deploy/helm/openshell/README.md +++ b/deploy/helm/openshell/README.md @@ -130,6 +130,28 @@ Append these flags to any of the PostgreSQL commands above for OpenShift: --set securityContext.runAsUser=null ``` +### High availability + +Set `replicaCount` above `1` only with `server.externalDbSecret`; the default +SQLite database is per pod and cannot coordinate multiple gateway replicas. +The chart creates a headless peer Service for gateway-to-gateway relay traffic. +StatefulSet pods use stable pod DNS names through that headless Service. +Deployment pods advertise their pod IP with `OPENSHELL_PEER_ENDPOINT`, because +Kubernetes does not assign stable per-pod DNS names to Deployment replicas. + +Gateway peer traffic uses Kubernetes ServiceAccount identity. Each gateway pod +mounts a projected, pod-bound ServiceAccount token with audience +`openshell-gateway-peer`; receiving replicas validate that token with the +Kubernetes TokenReview API, verify the live pod UID and Helm selector labels, +and authorize only the internal `PeerRelay` RPC. The chart does not create or +accept a shared gateway peer Secret. + +With gateway TLS enabled, peer calls use the chart CA and client TLS Secret for +server verification and mTLS. The client verifies the stable gateway Service +DNS name while connecting directly to the owning pod. Custom TLS Secrets must +include that Service DNS name in the server certificate and provide the CA and +client credentials configured by `server.tls`. + ## Secret bootstrap By default, a pre-install/pre-upgrade hook Job runs `openshell-gateway generate-certs` diff --git a/deploy/helm/openshell/README.md.gotmpl b/deploy/helm/openshell/README.md.gotmpl index 73ebb39c88..754a31c720 100644 --- a/deploy/helm/openshell/README.md.gotmpl +++ b/deploy/helm/openshell/README.md.gotmpl @@ -130,6 +130,28 @@ Append these flags to any of the PostgreSQL commands above for OpenShift: --set securityContext.runAsUser=null ``` +### High availability + +Set `replicaCount` above `1` only with `server.externalDbSecret`; the default +SQLite database is per pod and cannot coordinate multiple gateway replicas. +The chart creates a headless peer Service for gateway-to-gateway relay traffic. +StatefulSet pods use stable pod DNS names through that headless Service. +Deployment pods advertise their pod IP with `OPENSHELL_PEER_ENDPOINT`, because +Kubernetes does not assign stable per-pod DNS names to Deployment replicas. + +Gateway peer traffic uses Kubernetes ServiceAccount identity. Each gateway pod +mounts a projected, pod-bound ServiceAccount token with audience +`openshell-gateway-peer`; receiving replicas validate that token with the +Kubernetes TokenReview API, verify the live pod UID and Helm selector labels, +and authorize only the internal `PeerRelay` RPC. The chart does not create or +accept a shared gateway peer Secret. + +With gateway TLS enabled, peer calls use the chart CA and client TLS Secret for +server verification and mTLS. The client verifies the stable gateway Service +DNS name while connecting directly to the owning pod. Custom TLS Secrets must +include that Service DNS name in the server certificate and provide the CA and +client credentials configured by `server.tls`. + ## Secret bootstrap By default, a pre-install/pre-upgrade hook Job runs `openshell-gateway generate-certs` diff --git a/deploy/helm/openshell/skaffold.yaml b/deploy/helm/openshell/skaffold.yaml index ce32c72132..18d6e2f766 100644 --- a/deploy/helm/openshell/skaffold.yaml +++ b/deploy/helm/openshell/skaffold.yaml @@ -143,6 +143,25 @@ profiles: path: /deploy/helm/releases/0/setValues value: server.disableTls: "false" + # Full HA test path: installs Envoy Gateway and layers both HA replicas and + # Gateway API routing values onto the OpenShell release. + - name: high-availability + patches: + - op: add + path: /deploy/helm/releases/0 + value: + name: envoy-gateway + remoteChart: oci://docker.io/envoyproxy/gateway-helm + version: v1.7.2 + namespace: envoy-gateway-system + createNamespace: true + wait: true + - op: add + path: /deploy/helm/releases/1/valuesFiles/- + value: ci/values-high-availability.yaml + - op: add + path: /deploy/helm/releases/1/valuesFiles/- + value: ci/values-gateway.yaml - name: credential-driver-kubernetes-secrets patches: - op: add diff --git a/deploy/helm/openshell/templates/_gateway-workload.tpl b/deploy/helm/openshell/templates/_gateway-workload.tpl index a73acc9810..f66a4aff5c 100644 --- a/deploy/helm/openshell/templates/_gateway-workload.tpl +++ b/deploy/helm/openshell/templates/_gateway-workload.tpl @@ -50,6 +50,50 @@ spec: - {{ .Values.server.dbUrl | quote }} {{- end }} env: + - name: OPENSHELL_REPLICA_ID + valueFrom: + fieldRef: + fieldPath: metadata.name + - name: OPENSHELL_POD_NAME + valueFrom: + fieldRef: + fieldPath: metadata.name + - name: OPENSHELL_POD_NAMESPACE + valueFrom: + fieldRef: + fieldPath: metadata.namespace + {{- if eq (include "openshell.workloadKind" .) "deployment" }} + - name: OPENSHELL_POD_IP + valueFrom: + fieldRef: + fieldPath: status.podIP + - name: OPENSHELL_PEER_ENDPOINT + value: {{ printf "%s://$(OPENSHELL_POD_IP):%d" (ternary "http" "https" (default false .Values.server.disableTls)) (int .Values.service.port) | quote }} + {{- end }} + - name: OPENSHELL_SERVICE_ACCOUNT_NAME + value: {{ include "openshell.serviceAccountName" . | quote }} + - name: OPENSHELL_PEER_SERVICE_NAME + value: {{ include "openshell.peerServiceName" . | quote }} + - name: OPENSHELL_PEER_TOKEN_AUDIENCE + value: "openshell-gateway-peer" + - name: OPENSHELL_PEER_SERVICE_ACCOUNT_TOKEN_FILE + value: /var/run/secrets/openshell-peer/token + - name: OPENSHELL_PEER_POD_LABELS + value: {{ printf "app.kubernetes.io/name=%s,app.kubernetes.io/instance=%s" (include "openshell.name" .) .Release.Name | quote }} + {{- if not .Values.server.disableTls }} + - name: OPENSHELL_PEER_TLS_SERVER_NAME + value: {{ printf "%s.%s.svc.cluster.local" (include "openshell.fullname" .) .Release.Namespace | quote }} + {{- if or .Values.pkiInitJob.enabled .Values.certManager.enabled }} + - name: OPENSHELL_PEER_TLS_CA_FILE + value: /etc/openshell-tls/server/ca.crt + {{- end }} + {{- if or .Values.server.tls.clientCaSecretName (and .Values.pkiInitJob.enabled (not .Values.certManager.enabled)) (and .Values.certManager.enabled .Values.certManager.clientCaFromServerTlsSecret) }} + - name: OPENSHELL_PEER_TLS_CERT_FILE + value: /etc/openshell-tls/peer-client/tls.crt + - name: OPENSHELL_PEER_TLS_KEY_FILE + value: /etc/openshell-tls/peer-client/tls.key + {{- end }} + {{- end }} {{- if not (or .Values.server.credentialDrivers.kubernetesSecrets.enabled .Values.server.credentialDrivers.vault.enabled) }} - name: {{ include "openshell.credentialStorageKeyEncryptionKeyEnvName" . }} valueFrom: @@ -93,6 +137,9 @@ spec: - name: sandbox-jwt mountPath: /etc/openshell-jwt readOnly: true + - name: gateway-peer-token + mountPath: /var/run/secrets/openshell-peer + readOnly: true {{- if not .Values.server.disableTls }} - name: tls-cert mountPath: /etc/openshell-tls/server @@ -103,6 +150,11 @@ spec: readOnly: true {{- end }} {{- if or .Values.server.tls.clientCaSecretName (and .Values.pkiInitJob.enabled (not .Values.certManager.enabled)) (and .Values.certManager.enabled .Values.certManager.clientCaFromServerTlsSecret) }} + - name: peer-client-tls + mountPath: /etc/openshell-tls/peer-client + readOnly: true + {{- end }} + {{- if or .Values.server.tls.clientCaSecretName (and .Values.pkiInitJob.enabled (not .Values.certManager.enabled)) (and .Values.certManager.enabled .Values.certManager.clientCaFromServerTlsSecret) }} - name: tls-client-ca mountPath: /etc/openshell-tls/client-ca readOnly: true @@ -163,6 +215,14 @@ spec: secret: secretName: {{ include "openshell.sandboxJwtSecretName" . }} defaultMode: {{ .Values.server.sandboxJwt.secretDefaultMode | default 0400 }} + - name: gateway-peer-token + projected: + defaultMode: 0400 + sources: + - serviceAccountToken: + path: token + audience: openshell-gateway-peer + expirationSeconds: 3600 {{- if not .Values.server.disableTls }} - name: tls-cert secret: @@ -173,6 +233,11 @@ spec: secretName: {{ include "openshell.fullname" . }}-server-external-tls {{- end }} {{- if or .Values.server.tls.clientCaSecretName (and .Values.pkiInitJob.enabled (not .Values.certManager.enabled)) (and .Values.certManager.enabled .Values.certManager.clientCaFromServerTlsSecret) }} + - name: peer-client-tls + secret: + secretName: {{ .Values.server.tls.clientTlsSecretName }} + {{- end }} + {{- if or .Values.server.tls.clientCaSecretName (and .Values.pkiInitJob.enabled (not .Values.certManager.enabled)) (and .Values.certManager.enabled .Values.certManager.clientCaFromServerTlsSecret) }} - name: tls-client-ca secret: {{- if or (and .Values.pkiInitJob.enabled (not .Values.certManager.enabled)) (and .Values.certManager.enabled .Values.certManager.clientCaFromServerTlsSecret) }} diff --git a/deploy/helm/openshell/templates/_helpers.tpl b/deploy/helm/openshell/templates/_helpers.tpl index 548418abc6..d6342f2346 100644 --- a/deploy/helm/openshell/templates/_helpers.tpl +++ b/deploy/helm/openshell/templates/_helpers.tpl @@ -160,6 +160,10 @@ Name of the Secret holding gateway-minted sandbox JWT signing material. {{- .Values.server.sandboxJwt.signingSecretName | default (printf "%s-jwt-keys" (include "openshell.fullname" .)) -}} {{- end }} +{{- define "openshell.peerServiceName" -}} +{{- printf "%s-peer" (include "openshell.fullname" .) -}} +{{- end }} + {{/* gRPC endpoint sandbox pods use to call back into the gateway. An explicit .Values.server.grpcEndpoint is used verbatim. Otherwise it is derived from diff --git a/deploy/helm/openshell/templates/clusterrole.yaml b/deploy/helm/openshell/templates/clusterrole.yaml index eb1ed8e1d0..a47c603732 100644 --- a/deploy/helm/openshell/templates/clusterrole.yaml +++ b/deploy/helm/openshell/templates/clusterrole.yaml @@ -9,8 +9,8 @@ metadata: labels: {{- include "openshell.labels" . | nindent 4 }} rules: - # Validate projected sandbox ServiceAccount tokens during the - # IssueSandboxToken bootstrap exchange. + # Validate projected ServiceAccount tokens during sandbox bootstrap and + # internal gateway peer authentication. - apiGroups: - authentication.k8s.io resources: diff --git a/deploy/helm/openshell/templates/peer-role.yaml b/deploy/helm/openshell/templates/peer-role.yaml new file mode 100644 index 0000000000..59ae7479ad --- /dev/null +++ b/deploy/helm/openshell/templates/peer-role.yaml @@ -0,0 +1,35 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +apiVersion: rbac.authorization.k8s.io/v1 +kind: Role +metadata: + name: {{ include "openshell.fullname" . }}-peer + namespace: {{ .Release.Namespace }} + labels: + {{- include "openshell.labels" . | nindent 4 }} +rules: + # Gateway peer identity: TokenReview authenticates the projected token, then + # the receiver resolves the returned pod name and UID to the live gateway pod + # in the release namespace. + - apiGroups: + - "" + resources: + - pods + verbs: + - get +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: RoleBinding +metadata: + name: {{ include "openshell.fullname" . }}-peer + namespace: {{ .Release.Namespace }} + labels: + {{- include "openshell.labels" . | nindent 4 }} +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: Role + name: {{ include "openshell.fullname" . }}-peer +subjects: + - kind: ServiceAccount + name: {{ include "openshell.serviceAccountName" . }} + namespace: {{ .Release.Namespace }} diff --git a/deploy/helm/openshell/templates/peer-service.yaml b/deploy/helm/openshell/templates/peer-service.yaml new file mode 100644 index 0000000000..f5d93af245 --- /dev/null +++ b/deploy/helm/openshell/templates/peer-service.yaml @@ -0,0 +1,19 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +apiVersion: v1 +kind: Service +metadata: + name: {{ include "openshell.peerServiceName" . }} + labels: + {{- include "openshell.labels" . | nindent 4 }} +spec: + clusterIP: None + publishNotReadyAddresses: true + ports: + - port: {{ .Values.service.port }} + targetPort: grpc + protocol: TCP + name: grpc + appProtocol: grpc + selector: + {{- include "openshell.selectorLabels" . | nindent 4 }} diff --git a/deploy/helm/openshell/templates/statefulset.yaml b/deploy/helm/openshell/templates/statefulset.yaml index 30571f80ba..10d0839f60 100644 --- a/deploy/helm/openshell/templates/statefulset.yaml +++ b/deploy/helm/openshell/templates/statefulset.yaml @@ -9,7 +9,7 @@ metadata: labels: {{- include "openshell.labels" . | nindent 4 }} spec: - serviceName: {{ include "openshell.fullname" . }} + serviceName: {{ include "openshell.peerServiceName" . }} replicas: {{ .Values.replicaCount }} selector: matchLabels: diff --git a/deploy/helm/openshell/tests/credential_drivers_test.yaml b/deploy/helm/openshell/tests/credential_drivers_test.yaml index 76d8e081a5..931c62ad1d 100644 --- a/deploy/helm/openshell/tests/credential_drivers_test.yaml +++ b/deploy/helm/openshell/tests/credential_drivers_test.yaml @@ -46,15 +46,14 @@ tests: - it: injects the default credential storage key-encryption key Secret into the gateway pod by default template: templates/statefulset.yaml asserts: - - equal: - path: spec.template.spec.containers[0].env[0].name - value: OPENSHELL_GATEWAY_CREDENTIAL_KEY_ENCRYPTION_KEY - - matchRegex: - path: spec.template.spec.containers[0].env[0].valueFrom.secretKeyRef.name - pattern: 'credential-storage-key-encryption-key$' - - equal: - path: spec.template.spec.containers[0].env[0].valueFrom.secretKeyRef.key - value: key-encryption-key + - contains: + path: spec.template.spec.containers[0].env + content: + name: OPENSHELL_GATEWAY_CREDENTIAL_KEY_ENCRYPTION_KEY + valueFrom: + secretKeyRef: + name: openshell-credential-storage-key-encryption-key + key: key-encryption-key - it: renders Kubernetes Secrets credential driver config template: templates/gateway-config.yaml @@ -153,9 +152,14 @@ tests: - equal: path: kind value: Deployment - - equal: - path: spec.template.spec.containers[0].env[0].name - value: OPENSHELL_GATEWAY_CREDENTIAL_KEY_ENCRYPTION_KEY + - contains: + path: spec.template.spec.containers[0].env + content: + name: OPENSHELL_GATEWAY_CREDENTIAL_KEY_ENCRYPTION_KEY + valueFrom: + secretKeyRef: + name: openshell-credential-storage-key-encryption-key + key: key-encryption-key - it: allows default credential storage with multiple replicas and an external database template: templates/statefulset.yaml @@ -167,6 +171,11 @@ tests: - equal: path: spec.replicas value: 2 - - equal: - path: spec.template.spec.containers[0].env[0].name - value: OPENSHELL_GATEWAY_CREDENTIAL_KEY_ENCRYPTION_KEY + - contains: + path: spec.template.spec.containers[0].env + content: + name: OPENSHELL_GATEWAY_CREDENTIAL_KEY_ENCRYPTION_KEY + valueFrom: + secretKeyRef: + name: openshell-credential-storage-key-encryption-key + key: key-encryption-key diff --git a/deploy/helm/openshell/tests/gateway_config_test.yaml b/deploy/helm/openshell/tests/gateway_config_test.yaml index eaa2140862..6935abb05f 100644 --- a/deploy/helm/openshell/tests/gateway_config_test.yaml +++ b/deploy/helm/openshell/tests/gateway_config_test.yaml @@ -5,6 +5,8 @@ suite: gateway TOML config shape templates: - templates/gateway-config.yaml - templates/deployment.yaml + - templates/peer-role.yaml + - templates/peer-service.yaml - templates/statefulset.yaml release: name: openshell @@ -70,16 +72,16 @@ tests: server.oidc.caConfigMapName: openshell-oidc-ca asserts: - equal: - path: spec.template.spec.containers[0].volumeMounts[3].name + path: spec.template.spec.containers[0].volumeMounts[4].name value: oidc-ca - equal: - path: spec.template.spec.containers[0].volumeMounts[3].mountPath + path: spec.template.spec.containers[0].volumeMounts[4].mountPath value: /etc/openshell-tls/oidc-ca - equal: - path: spec.template.spec.volumes[2].name + path: spec.template.spec.volumes[3].name value: oidc-ca - equal: - path: spec.template.spec.volumes[2].configMap.name + path: spec.template.spec.volumes[3].configMap.name value: openshell-oidc-ca # Regression for the P1 bug Drew flagged: grpc_endpoint MUST live in the @@ -400,6 +402,128 @@ tests: path: spec.template.spec.containers[0].args content: "sqlite:/var/openshell/openshell.db" + - it: configures gateway peer identity and projected peer token + template: templates/statefulset.yaml + asserts: + - equal: + path: spec.serviceName + value: openshell-peer + - contains: + path: spec.template.spec.containers[0].env + content: + name: OPENSHELL_PEER_SERVICE_NAME + value: openshell-peer + - contains: + path: spec.template.spec.containers[0].env + content: + name: OPENSHELL_PEER_TOKEN_AUDIENCE + value: openshell-gateway-peer + - contains: + path: spec.template.spec.containers[0].env + content: + name: OPENSHELL_PEER_TLS_SERVER_NAME + value: openshell.my-namespace.svc.cluster.local + - contains: + path: spec.template.spec.containers[0].env + content: + name: OPENSHELL_PEER_TLS_CA_FILE + value: /etc/openshell-tls/server/ca.crt + - contains: + path: spec.template.spec.containers[0].env + content: + name: OPENSHELL_PEER_TLS_CERT_FILE + value: /etc/openshell-tls/peer-client/tls.crt + - contains: + path: spec.template.spec.containers[0].env + content: + name: OPENSHELL_PEER_TLS_KEY_FILE + value: /etc/openshell-tls/peer-client/tls.key + - contains: + path: spec.template.spec.containers[0].volumeMounts + content: + name: gateway-peer-token + mountPath: /var/run/secrets/openshell-peer + readOnly: true + - contains: + path: spec.template.spec.containers[0].volumeMounts + content: + name: peer-client-tls + mountPath: /etc/openshell-tls/peer-client + readOnly: true + - equal: + path: spec.template.spec.volumes[2].projected.sources[0].serviceAccountToken.audience + value: openshell-gateway-peer + + - it: configures gateway peer identity and projected peer token for Deployment + template: templates/deployment.yaml + set: + workload.kind: deployment + server.externalDbSecret: my-pg-secret + server.disableTls: true + asserts: + - contains: + path: spec.template.spec.containers[0].env + content: + name: OPENSHELL_POD_IP + valueFrom: + fieldRef: + fieldPath: status.podIP + - contains: + path: spec.template.spec.containers[0].env + content: + name: OPENSHELL_PEER_ENDPOINT + value: http://$(OPENSHELL_POD_IP):8080 + - contains: + path: spec.template.spec.containers[0].env + content: + name: OPENSHELL_PEER_SERVICE_NAME + value: openshell-peer + - contains: + path: spec.template.spec.containers[0].env + content: + name: OPENSHELL_PEER_TOKEN_AUDIENCE + value: openshell-gateway-peer + - contains: + path: spec.template.spec.containers[0].volumeMounts + content: + name: gateway-peer-token + mountPath: /var/run/secrets/openshell-peer + readOnly: true + - equal: + path: spec.template.spec.volumes[2].projected.sources[0].serviceAccountToken.audience + value: openshell-gateway-peer + + - it: renders headless gateway peer service + template: templates/peer-service.yaml + asserts: + - equal: + path: metadata.name + value: openshell-peer + - equal: + path: spec.clusterIP + value: None + - equal: + path: spec.publishNotReadyAddresses + value: true + + - it: grants release-namespace pod lookup for gateway peer identity validation + template: templates/peer-role.yaml + asserts: + - hasDocuments: + count: 2 + - equal: + path: metadata.namespace + value: my-namespace + documentIndex: 0 + - equal: + path: rules[0].resources[0] + value: pods + documentIndex: 0 + - equal: + path: subjects[0].name + value: openshell + documentIndex: 1 + - it: fails when legacy postgres.enabled is set template: templates/statefulset.yaml set: diff --git a/deploy/helm/openshell/tests/statefulset_client_ca_test.yaml b/deploy/helm/openshell/tests/statefulset_client_ca_test.yaml index 1d744b35aa..4a6986bf3a 100644 --- a/deploy/helm/openshell/tests/statefulset_client_ca_test.yaml +++ b/deploy/helm/openshell/tests/statefulset_client_ca_test.yaml @@ -17,13 +17,13 @@ tests: certManager.enabled: false asserts: - equal: - path: spec.template.spec.volumes[3].name + path: spec.template.spec.volumes[5].name value: tls-client-ca - equal: - path: spec.template.spec.volumes[3].secret.secretName + path: spec.template.spec.volumes[5].secret.secretName value: openshell-server-tls - equal: - path: spec.template.spec.volumes[3].secret.items[0].key + path: spec.template.spec.volumes[5].secret.items[0].key value: ca.crt - it: shares the cert-manager server TLS ca.crt when clientCaFromServerTlsSecret is true @@ -33,13 +33,13 @@ tests: certManager.clientCaFromServerTlsSecret: true asserts: - equal: - path: spec.template.spec.volumes[3].name + path: spec.template.spec.volumes[5].name value: tls-client-ca - equal: - path: spec.template.spec.volumes[3].secret.secretName + path: spec.template.spec.volumes[5].secret.secretName value: openshell-server-tls - equal: - path: spec.template.spec.volumes[3].secret.items[0].key + path: spec.template.spec.volumes[5].secret.items[0].key value: ca.crt # Regression: with cert-manager enabled and pkiInitJob left at its default @@ -56,13 +56,13 @@ tests: server.tls.clientCaSecretName: openshell-ca-tls asserts: - equal: - path: spec.template.spec.volumes[3].name + path: spec.template.spec.volumes[5].name value: tls-client-ca - equal: - path: spec.template.spec.volumes[3].secret.secretName + path: spec.template.spec.volumes[5].secret.secretName value: openshell-ca-tls - notExists: - path: spec.template.spec.volumes[3].secret.items + path: spec.template.spec.volumes[5].secret.items # When cert-manager owns TLS, does not share its CA, and no separate client CA # secret is configured, there is no client CA to mount: the volume must not @@ -77,7 +77,7 @@ tests: asserts: - lengthEqual: path: spec.template.spec.volumes - count: 3 + count: 4 - notContains: path: spec.template.spec.containers[0].volumeMounts content: diff --git a/deploy/kube/manifests/envoy-gateway-openshell.yaml b/deploy/kube/manifests/envoy-gateway-openshell.yaml index 583f2b41ba..68d1ea7d7a 100644 --- a/deploy/kube/manifests/envoy-gateway-openshell.yaml +++ b/deploy/kube/manifests/envoy-gateway-openshell.yaml @@ -15,3 +15,24 @@ metadata: name: eg spec: controllerName: gateway.envoyproxy.io/gatewayclass-controller +--- +# OpenShell gRPC streams can remain active across sandbox create, exec, relay, +# and watch operations. Disable Envoy's backend request and stream duration +# timeouts for the OpenShell GRPCRoute so the proxy does not reset long-running +# HTTP/2 streams while gateway pods rotate behind it. +apiVersion: gateway.envoyproxy.io/v1alpha1 +kind: BackendTrafficPolicy +metadata: + name: openshell-grpc-timeouts + namespace: openshell + labels: + app.kubernetes.io/name: openshell +spec: + targetRefs: + - group: gateway.networking.k8s.io + kind: GRPCRoute + name: openshell + timeout: + http: + requestTimeout: 0s + maxStreamDuration: 0s diff --git a/e2e/rust/Cargo.toml b/e2e/rust/Cargo.toml index 42f989ce42..53444be9fb 100644 --- a/e2e/rust/Cargo.toml +++ b/e2e/rust/Cargo.toml @@ -28,6 +28,7 @@ e2e-docker = ["e2e", "e2e-host-gateway", "e2e-local-container-driver"] e2e-gpu = ["e2e"] e2e-docker-gpu = ["e2e-docker", "e2e-gpu"] e2e-kubernetes = ["e2e"] +e2e-kubernetes-ha = ["e2e-kubernetes"] e2e-kubernetes-credential-drivers = ["e2e-kubernetes"] e2e-kubernetes-workspace-managed = ["e2e-kubernetes"] e2e-kubernetes-workspace-operator = ["e2e-kubernetes"] @@ -96,16 +97,16 @@ name = "vm_gateway_start" path = "tests/vm_gateway_start.rs" required-features = ["e2e-vm"] +[[test]] +name = "kubernetes_ha_rebalancing" +path = "tests/kubernetes_ha_rebalancing.rs" +required-features = ["e2e-kubernetes-ha"] + [[test]] name = "provider_token_exchange" path = "tests/provider_token_exchange.rs" required-features = ["e2e-podman"] -[[test]] -name = "readyz_health" -path = "tests/readyz_health.rs" -required-features = ["e2e-kubernetes"] - [[test]] name = "kubernetes_corporate_proxy" path = "tests/kubernetes_corporate_proxy.rs" diff --git a/e2e/rust/e2e-kubernetes.sh b/e2e/rust/e2e-kubernetes.sh index cf28e35728..125790a31d 100755 --- a/e2e/rust/e2e-kubernetes.sh +++ b/e2e/rust/e2e-kubernetes.sh @@ -12,14 +12,14 @@ # the sandbox-side `host.openshell.internal` alias compile and run. The # wrapper detects the cluster's host-routable IP and wires it into the chart # via `server.hostGatewayIP`. Targeting a cluster where the test host is -# unreachable from pods? Set OPENSHELL_E2E_KUBERNETES_FEATURES=e2e to drop the +# unreachable from pods? Set OPENSHELL_E2E_KUBE_FEATURES=e2e to drop the # alias-dependent tests entirely. set -euo pipefail ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)" -E2E_FEATURES="${OPENSHELL_E2E_KUBERNETES_FEATURES:-e2e,e2e-host-gateway,e2e-kubernetes}" +E2E_FEATURES="${OPENSHELL_E2E_KUBE_FEATURES:-e2e,e2e-host-gateway,e2e-kubernetes}" # Docker and Podman build their local gateway and CLI together in the shared # gateway wrapper. Kubernetes consumes published gateway images, so only its diff --git a/e2e/rust/tests/credential_drivers.rs b/e2e/rust/tests/credential_drivers.rs index ef8069fd03..6d58a0db89 100644 --- a/e2e/rust/tests/credential_drivers.rs +++ b/e2e/rust/tests/credential_drivers.rs @@ -14,7 +14,7 @@ use openshell_e2e::harness::sandbox::SandboxGuard; use sha2::{Digest, Sha256}; use tokio::io::AsyncWriteExt; -const CREDENTIAL_KEY: &str = "OPENAI_API_KEY"; +const CREDENTIAL_KEY: &str = "ANTHROPIC_API_KEY"; const VAULT_POLICY: &str = r#"path "secret/data/openshell/provider-credentials/*" { capabilities = ["create", "read", "update", "delete"] } @@ -41,6 +41,12 @@ fn credential_driver() -> String { .unwrap_or_else(|_| "kubernetes-secrets".to_string()) } +#[derive(Debug)] +struct ProviderIdentity { + id: String, + workspace: String, +} + fn vault_namespace() -> String { std::env::var("OPENSHELL_E2E_VAULT_NAMESPACE").unwrap_or_else(|_| "vault".to_string()) } @@ -53,23 +59,26 @@ fn vault_token() -> String { std::env::var("OPENSHELL_E2E_VAULT_TOKEN").unwrap_or_else(|_| "root".to_string()) } -fn managed_kubernetes_secret_name(provider_name: &str) -> String { +fn managed_credential_hash(identity: &ProviderIdentity, provider_name: &str) -> String { let mut hasher = Sha256::new(); + hasher.update(identity.workspace.as_bytes()); + hasher.update([0]); + hasher.update(identity.id.as_bytes()); + hasher.update([0]); hasher.update(provider_name.as_bytes()); hasher.update([0]); hasher.update(CREDENTIAL_KEY.as_bytes()); let digest = hasher.finalize(); - let hex = format!("{digest:x}"); + format!("{digest:x}") +} + +fn managed_kubernetes_secret_name(identity: &ProviderIdentity, provider_name: &str) -> String { + let hex = managed_credential_hash(identity, provider_name); format!("openshell-cred-{}", &hex[..40]) } -fn managed_vault_path(provider_name: &str) -> String { - let mut hasher = Sha256::new(); - hasher.update(provider_name.as_bytes()); - hasher.update([0]); - hasher.update(CREDENTIAL_KEY.as_bytes()); - let digest = hasher.finalize(); - let hex = format!("{digest:x}"); +fn managed_vault_path(identity: &ProviderIdentity, provider_name: &str) -> String { + let hex = managed_credential_hash(identity, provider_name); format!("openshell/provider-credentials/{}", &hex[..40]) } @@ -203,7 +212,7 @@ async fn create_provider(name: &str, secret_value: &str) -> Result Result Result { + let (output, code) = run_cli(&["provider", "list", "--output", "json"]).await; + let clean = strip_ansi(&output); + if code != 0 { + return Err(format!("provider list failed (exit {code}):\n{clean}")); + } + let providers: Vec = serde_json::from_str(&clean) + .map_err(|err| format!("failed to parse provider list JSON: {err}\n{clean}"))?; + let provider = providers + .iter() + .find(|provider| provider["name"].as_str() == Some(provider_name)) + .ok_or_else(|| format!("provider '{provider_name}' was not returned by provider list"))?; + let id = provider["id"] + .as_str() + .filter(|id| !id.is_empty()) + .ok_or_else(|| format!("provider '{provider_name}' did not include an ID"))?; + let workspace = provider["workspace"] + .as_str() + .ok_or_else(|| format!("provider '{provider_name}' did not include a workspace"))?; + Ok(ProviderIdentity { + id: id.to_string(), + workspace: workspace.to_string(), + }) +} + async fn assert_provider_get_does_not_expose_secret( provider_name: &str, secret_value: &str, @@ -250,9 +284,8 @@ async fn assert_provider_placeholder_available_in_sandbox( "--no-auto-providers", "--no-tty", "--", - "bash", - "-lc", - r#"printf '%s\n' "$OPENAI_API_KEY""#, + "printenv", + CREDENTIAL_KEY, ]) .await?; let clean = strip_ansi(&guard.create_output); @@ -297,11 +330,12 @@ async fn configure_vault_storage() -> Result<(), String> { } async fn assert_kubernetes_secret_stored( + identity: &ProviderIdentity, provider_name: &str, secret_value: &str, ) -> Result<(), String> { let namespace = namespace(); - let secret_name = managed_kubernetes_secret_name(provider_name); + let secret_name = managed_kubernetes_secret_name(identity, provider_name); let encoded = kubectl(&[ "-n", &namespace, @@ -323,9 +357,12 @@ async fn assert_kubernetes_secret_stored( Ok(()) } -async fn assert_kubernetes_secret_deleted(provider_name: &str) -> Result<(), String> { +async fn assert_kubernetes_secret_deleted( + identity: &ProviderIdentity, + provider_name: &str, +) -> Result<(), String> { let namespace = namespace(); - let secret_name = managed_kubernetes_secret_name(provider_name); + let secret_name = managed_kubernetes_secret_name(identity, provider_name); match kubectl(&["-n", &namespace, "get", "secret", &secret_name]).await { Ok(output) => Err(format!( "Kubernetes Secret '{secret_name}' still exists after provider deletion:\n{output}" @@ -334,8 +371,12 @@ async fn assert_kubernetes_secret_deleted(provider_name: &str) -> Result<(), Str } } -async fn assert_vault_secret_stored(provider_name: &str, secret_value: &str) -> Result<(), String> { - let logical_path = managed_vault_path(provider_name); +async fn assert_vault_secret_stored( + identity: &ProviderIdentity, + provider_name: &str, + secret_value: &str, +) -> Result<(), String> { + let logical_path = managed_vault_path(identity, provider_name); let output = bao(&[ "kv", "get", @@ -349,8 +390,11 @@ async fn assert_vault_secret_stored(provider_name: &str, secret_value: &str) -> Ok(()) } -async fn assert_vault_secret_deleted(provider_name: &str) -> Result<(), String> { - let logical_path = managed_vault_path(provider_name); +async fn assert_vault_secret_deleted( + identity: &ProviderIdentity, + provider_name: &str, +) -> Result<(), String> { + let logical_path = managed_vault_path(identity, provider_name); match bao(&[ "kv", "get", @@ -368,20 +412,27 @@ async fn assert_vault_secret_deleted(provider_name: &str) -> Result<(), String> async fn assert_backend_stored( driver: &str, + identity: &ProviderIdentity, provider_name: &str, secret_value: &str, ) -> Result<(), String> { match driver { - "kubernetes-secrets" => assert_kubernetes_secret_stored(provider_name, secret_value).await, - "vault" => assert_vault_secret_stored(provider_name, secret_value).await, + "kubernetes-secrets" => { + assert_kubernetes_secret_stored(identity, provider_name, secret_value).await + } + "vault" => assert_vault_secret_stored(identity, provider_name, secret_value).await, other => Err(format!("unsupported credential driver '{other}'")), } } -async fn assert_backend_deleted(driver: &str, provider_name: &str) -> Result<(), String> { +async fn assert_backend_deleted( + driver: &str, + identity: &ProviderIdentity, + provider_name: &str, +) -> Result<(), String> { match driver { - "kubernetes-secrets" => assert_kubernetes_secret_deleted(provider_name).await, - "vault" => assert_vault_secret_deleted(provider_name).await, + "kubernetes-secrets" => assert_kubernetes_secret_deleted(identity, provider_name).await, + "vault" => assert_vault_secret_deleted(identity, provider_name).await, other => Err(format!("unsupported credential driver '{other}'")), } } @@ -400,7 +451,8 @@ async fn provider_credentials_are_stored_in_configured_backend() { let suffix = unique_suffix(); let driver_slug = driver.replace('-', ""); let provider_name = format!("cred-storage-{driver_slug}-{suffix}"); - let sandbox_name = format!("cred-storage-sandbox-{driver_slug}-{suffix}"); + let sandbox_hash = format!("{:x}", Sha256::digest(suffix.as_bytes())); + let sandbox_name = format!("cred-{}", &sandbox_hash[..14]); let secret_value = format!("example-e2e-{driver_slug}-{suffix}"); delete_provider(&provider_name).await; @@ -410,23 +462,24 @@ async fn provider_credentials_are_stored_in_configured_backend() { .expect("configure Vault storage fixture"); } - let result: Result<(), String> = async { + let result: Result = async { create_provider(&provider_name, &secret_value).await?; assert_provider_get_does_not_expose_secret(&provider_name, &secret_value).await?; - assert_backend_stored(&driver, &provider_name, &secret_value).await?; + let identity = provider_identity(&provider_name).await?; + assert_backend_stored(&driver, &identity, &provider_name, &secret_value).await?; assert_provider_placeholder_available_in_sandbox( &provider_name, &sandbox_name, &secret_value, ) .await?; - Ok(()) + Ok(identity) } .await; delete_provider(&provider_name).await; - assert_backend_deleted(&driver, &provider_name) + let identity = result.expect("credential storage e2e failed"); + assert_backend_deleted(&driver, &identity, &provider_name) .await .expect("credential backend object should be deleted with provider"); - result.expect("credential storage e2e failed"); } diff --git a/e2e/rust/tests/kubernetes_ha_rebalancing.rs b/e2e/rust/tests/kubernetes_ha_rebalancing.rs new file mode 100644 index 0000000000..ca65f1e3eb --- /dev/null +++ b/e2e/rust/tests/kubernetes_ha_rebalancing.rs @@ -0,0 +1,627 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +#![cfg(feature = "e2e-kubernetes-ha")] + +use std::fs; +use std::io::Write; +use std::path::Path; +use std::process::Stdio; +use std::time::{Duration, Instant}; + +use openshell_e2e::harness::binary::openshell_cmd; +use openshell_e2e::harness::output::strip_ansi; +use openshell_e2e::harness::port::{find_free_port, wait_for_port}; +use openshell_e2e::harness::sandbox::SandboxGuard; +use serde_json::Value; +use sha2::{Digest, Sha256}; +use tokio::process::{Child, Command}; + +static KUBE_HA_TEST_LOCK: tokio::sync::Mutex<()> = tokio::sync::Mutex::const_new(()); + +const HA_SYNC_PAYLOAD_BYTES: usize = 32 * 1024 * 1024; +const HA_SYNC_TIMEOUT: Duration = Duration::from_secs(600); + +#[derive(Clone)] +struct KubeTarget { + context: String, + namespace: String, + release: String, +} + +impl KubeTarget { + fn from_env() -> Self { + Self { + context: required_env("OPENSHELL_E2E_KUBE_CONTEXT"), + namespace: std::env::var("OPENSHELL_E2E_KUBE_NAMESPACE") + .unwrap_or_else(|_| "openshell".to_string()), + release: std::env::var("OPENSHELL_E2E_KUBE_RELEASE") + .unwrap_or_else(|_| "openshell".to_string()), + } + } + + async fn kubectl(&self, args: &[&str]) -> Result { + let output = Command::new("kubectl") + .arg("--context") + .arg(&self.context) + .args(args) + .stdout(Stdio::piped()) + .stderr(Stdio::piped()) + .output() + .await + .map_err(|err| format!("failed to spawn kubectl {args:?}: {err}"))?; + + let combined = format!( + "{}{}", + String::from_utf8_lossy(&output.stdout), + String::from_utf8_lossy(&output.stderr) + ); + + if !output.status.success() { + return Err(format!( + "kubectl {args:?} failed with exit {:?}:\n{combined}", + output.status.code() + )); + } + + Ok(combined) + } + + async fn scale_gateway(&self, replicas: usize) -> Result<(), String> { + let resource = self.gateway_workload_resource().await?; + let replicas_arg = replicas.to_string(); + + self.kubectl(&[ + "-n", + &self.namespace, + "scale", + &resource, + "--replicas", + &replicas_arg, + ]) + .await?; + self.kubectl(&[ + "-n", + &self.namespace, + "rollout", + "status", + &resource, + "--timeout=180s", + ]) + .await?; + Ok(()) + } + + async fn gateway_workload_resource(&self) -> Result { + let deployment = format!("deployment/{}", self.release); + if self + .kubectl(&["-n", &self.namespace, "get", &deployment]) + .await + .is_ok() + { + return Ok(deployment); + } + + let statefulset = format!("statefulset/{}", self.release); + if self + .kubectl(&["-n", &self.namespace, "get", &statefulset]) + .await + .is_ok() + { + return Ok(statefulset); + } + + Err(format!( + "no gateway Deployment or StatefulSet named {} found in namespace {}", + self.release, self.namespace + )) + } + + async fn delete_gateway_pod(&self, pod: &str) -> Result<(), String> { + self.kubectl(&[ + "-n", + &self.namespace, + "delete", + "pod", + pod, + "--wait=true", + "--timeout=90s", + ]) + .await?; + Ok(()) + } + + async fn roll_gateway_pods(&self, pods: Vec, expected: usize) -> Result<(), String> { + for pod in pods { + self.delete_gateway_pod(&pod).await?; + self.wait_for_gateway_pods(expected).await?; + } + Ok(()) + } + + async fn wait_for_gateway_pods(&self, expected: usize) -> Result, String> { + let deadline = Instant::now() + Duration::from_secs(240); + let mut last = String::new(); + + while Instant::now() < deadline { + match self.gateway_pods().await { + Ok(pods) => { + if pods.len() == expected && pods.iter().all(|pod| pod.ready) { + return Ok(pods.into_iter().map(|pod| pod.name).collect()); + } + last = format!( + "pods={:?}", + pods.iter() + .map(|pod| format!("{} ready={}", pod.name, pod.ready)) + .collect::>() + ); + } + Err(err) => last = err, + } + tokio::time::sleep(Duration::from_secs(2)).await; + } + + Err(format!( + "gateway pods did not reach expected ready count {expected} within 240s; last={last}" + )) + } + + async fn gateway_pods(&self) -> Result, String> { + let selector = format!("app.kubernetes.io/instance={}", self.release); + let json = self + .kubectl(&[ + "-n", + &self.namespace, + "get", + "pods", + "-l", + &selector, + "-o", + "json", + ]) + .await?; + let value = serde_json::from_str::(&json) + .map_err(|err| format!("failed to parse gateway pod JSON: {err}\n{json}"))?; + let items = value["items"] + .as_array() + .ok_or_else(|| format!("gateway pod JSON missing items array: {value}"))?; + + let mut pods = Vec::new(); + for item in items { + if !item["metadata"]["deletionTimestamp"].is_null() { + continue; + } + let Some(name) = item["metadata"]["name"].as_str() else { + continue; + }; + let ready = item["status"]["conditions"] + .as_array() + .is_some_and(|conditions| { + conditions.iter().any(|condition| { + condition["type"].as_str() == Some("Ready") + && condition["status"].as_str() == Some("True") + }) + }); + pods.push(GatewayPod { + name: name.to_string(), + ready, + }); + } + pods.sort_by(|a, b| a.name.cmp(&b.name)); + Ok(pods) + } +} + +#[derive(Debug)] +struct GatewayPod { + name: String, + ready: bool, +} + +struct PortForward { + port: u16, + child: Child, +} + +impl PortForward { + async fn start(kube: &KubeTarget, pod: &str) -> Result { + let port = find_free_port(); + let mut child = Command::new("kubectl") + .arg("--context") + .arg(&kube.context) + .arg("-n") + .arg(&kube.namespace) + .arg("port-forward") + .arg(format!("pod/{pod}")) + .arg(format!("{port}:8080")) + .stdout(Stdio::null()) + .stderr(Stdio::null()) + .kill_on_drop(true) + .spawn() + .map_err(|err| format!("failed to start kubectl port-forward for {pod}: {err}"))?; + + match wait_for_port("127.0.0.1", port, Duration::from_secs(30)).await { + Ok(()) => Ok(Self { port, child }), + Err(err) => { + let status = child.try_wait().ok().flatten(); + let _ = child.kill().await; + Err(format!( + "port-forward to {pod} did not become ready on {port}: {err}; status={status:?}" + )) + } + } + } +} + +impl Drop for PortForward { + fn drop(&mut self) { + let _ = self.child.start_kill(); + } +} + +fn required_env(name: &str) -> String { + std::env::var(name) + .unwrap_or_else(|_| panic!("{name} is not set; run through e2e/rust/e2e-kubernetes.sh")) +} + +async fn exec_through_pod( + kube: &KubeTarget, + pod: &str, + sandbox_name: &str, + marker: &str, +) -> Result<(), String> { + let port_forward = PortForward::start(kube, pod).await?; + let endpoint = format!("http://127.0.0.1:{}", port_forward.port); + + let mut cmd = openshell_cmd(); + cmd.arg("--gateway-endpoint") + .arg(&endpoint) + .args([ + "sandbox", + "exec", + "--name", + sandbox_name, + "--no-tty", + "--", + "printf", + "%s", + marker, + ]) + .stdout(Stdio::piped()) + .stderr(Stdio::piped()); + let output = cmd + .output() + .await + .map_err(|err| format!("failed to spawn openshell exec via {pod}: {err}"))?; + + let combined = strip_ansi(&format!( + "{}{}", + String::from_utf8_lossy(&output.stdout), + String::from_utf8_lossy(&output.stderr) + )); + if !output.status.success() || !combined.contains(marker) { + return Err(format!( + "exec through {pod} ({endpoint}) failed with exit {:?}; expected marker {marker:?}; output:\n{combined}", + output.status.code() + )); + } + + Ok(()) +} + +async fn exec_through_configured_gateway(sandbox_name: &str, marker: &str) -> Result<(), String> { + let mut cmd = openshell_cmd(); + cmd.args([ + "sandbox", + "exec", + "--name", + sandbox_name, + "--no-tty", + "--", + "printf", + "%s", + marker, + ]) + .stdout(Stdio::piped()) + .stderr(Stdio::piped()); + let output = cmd + .output() + .await + .map_err(|err| format!("failed to spawn openshell exec via configured gateway: {err}"))?; + + let combined = strip_ansi(&format!( + "{}{}", + String::from_utf8_lossy(&output.stdout), + String::from_utf8_lossy(&output.stderr) + )); + if !output.status.success() || !combined.contains(marker) { + return Err(format!( + "exec through configured gateway failed with exit {:?}; expected marker {marker:?}; output:\n{combined}", + output.status.code() + )); + } + + Ok(()) +} + +async fn create_sandbox_through_configured_gateway(phase: &str) -> Result { + let marker = format!("ha-create-watch-{phase}"); + let guard = SandboxGuard::create(&["--", "printf", "%s", &marker]).await?; + let output = strip_ansi(&guard.create_output); + + if !output.contains(&marker) { + return Err(format!( + "sandbox create through configured gateway did not include marker {marker:?}; output:\n{output}" + )); + } + + Ok(guard) +} + +async fn assert_exec_through_all_pods( + kube: &KubeTarget, + pods: &[String], + sandbox_name: &str, + phase: &str, +) -> Result<(), String> { + for pod in pods { + let marker = format!("ha-rebalance-{phase}-{pod}"); + exec_through_pod(kube, pod, sandbox_name, &marker).await?; + } + Ok(()) +} + +fn write_deterministic_payload(path: &Path, size: usize) { + let mut file = fs::File::create(path).expect("create HA sync payload"); + let mut offset = 0usize; + let mut remaining = size; + let mut buf = vec![0_u8; 64 * 1024]; + + while remaining > 0 { + let chunk_len = remaining.min(buf.len()); + for (idx, byte) in buf[..chunk_len].iter_mut().enumerate() { + *byte = u8::try_from((offset + idx) % 251).expect("byte value fits"); + } + file.write_all(&buf[..chunk_len]) + .expect("write HA sync payload chunk"); + offset += chunk_len; + remaining -= chunk_len; + } +} + +fn sha256_file(path: &Path) -> String { + let data = fs::read(path).expect("read file for SHA-256"); + let mut hasher = Sha256::new(); + hasher.update(&data); + hex::encode(hasher.finalize()) +} + +fn upload_command(sandbox_name: &str, local_path: &Path, dest: &str) -> Command { + let mut cmd = openshell_cmd(); + cmd.arg("sandbox") + .arg("upload") + .arg(sandbox_name) + .arg(local_path) + .arg(dest) + .arg("--no-git-ignore"); + cmd +} + +fn download_command(sandbox_name: &str, sandbox_path: &str, local_dest: &Path) -> Command { + let mut cmd = openshell_cmd(); + cmd.arg("sandbox") + .arg("download") + .arg(sandbox_name) + .arg(sandbox_path) + .arg(local_dest); + cmd +} + +async fn run_cli_during_gateway_pod_roll( + kube: &KubeTarget, + mut cmd: Command, + operation: &str, +) -> Result { + let pods = kube.wait_for_gateway_pods(2).await?; + + cmd.stdout(Stdio::piped()) + .stderr(Stdio::piped()) + .kill_on_drop(true); + let child = cmd + .spawn() + .map_err(|err| format!("failed to spawn {operation} command: {err}"))?; + + let (roll_result, output_result) = tokio::time::timeout(HA_SYNC_TIMEOUT, async { + let roll = async { + tokio::time::sleep(Duration::from_millis(250)).await; + kube.roll_gateway_pods(pods, 2).await + }; + tokio::join!(roll, child.wait_with_output()) + }) + .await + .map_err(|_| { + format!( + "{operation} command and gateway pod roll did not finish within {HA_SYNC_TIMEOUT:?}" + ) + })?; + + roll_result.map_err(|err| { + format!("gateway pod roll failed while {operation} command was running: {err}") + })?; + + let output = + output_result.map_err(|err| format!("failed to wait for {operation} command: {err}"))?; + let combined = strip_ansi(&format!( + "{}{}", + String::from_utf8_lossy(&output.stdout), + String::from_utf8_lossy(&output.stderr) + )); + if !output.status.success() { + return Err(format!( + "{operation} command failed with exit {:?} during gateway pod roll:\n{combined}", + output.status.code() + )); + } + + Ok(combined) +} + +#[tokio::test] +async fn sandbox_exec_rebalances_across_gateway_scale_and_rollout() { + let _test_lock = KUBE_HA_TEST_LOCK.lock().await; + let kube = KubeTarget::from_env(); + + let mut pods = kube + .wait_for_gateway_pods(2) + .await + .expect("gateway should start with two ready HA replicas"); + + let mut sandbox = create_sandbox_through_configured_gateway("initial") + .await + .expect("sandbox create and readiness watch should succeed through the configured gateway endpoint initially"); + + assert_exec_through_all_pods(&kube, &pods, &sandbox.name, "initial") + .await + .expect("exec should work through every initial gateway pod"); + exec_through_configured_gateway(&sandbox.name, "ha-rebalance-client-initial") + .await + .expect("exec should work through the configured client gateway endpoint initially"); + + kube.scale_gateway(3) + .await + .expect("scale gateway to three replicas"); + pods = kube + .wait_for_gateway_pods(3) + .await + .expect("gateway should scale to three ready replicas"); + assert_exec_through_all_pods(&kube, &pods, &sandbox.name, "scale-up") + .await + .expect("exec should work through every gateway pod after scale-up"); + exec_through_configured_gateway(&sandbox.name, "ha-rebalance-client-scale-up") + .await + .expect("exec should work through the configured client gateway endpoint after scale-up"); + let mut scale_up_sandbox = create_sandbox_through_configured_gateway("scale-up") + .await + .expect( + "sandbox create and readiness watch should succeed through the configured gateway endpoint after scale-up", + ); + scale_up_sandbox.cleanup().await; + + kube.scale_gateway(2) + .await + .expect("scale gateway back to two replicas"); + pods = kube + .wait_for_gateway_pods(2) + .await + .expect("gateway should scale back to two ready replicas"); + assert_exec_through_all_pods(&kube, &pods, &sandbox.name, "scale-down") + .await + .expect("exec should work through every gateway pod after scale-down"); + exec_through_configured_gateway(&sandbox.name, "ha-rebalance-client-scale-down") + .await + .expect("exec should work through the configured client gateway endpoint after scale-down"); + let mut scale_down_sandbox = create_sandbox_through_configured_gateway("scale-down") + .await + .expect( + "sandbox create and readiness watch should succeed through the configured gateway endpoint after scale-down", + ); + scale_down_sandbox.cleanup().await; + + for (idx, pod) in pods.clone().into_iter().enumerate() { + kube.delete_gateway_pod(&pod) + .await + .unwrap_or_else(|err| panic!("delete gateway pod {pod}: {err}")); + pods = kube.wait_for_gateway_pods(2).await.unwrap_or_else(|err| { + panic!("gateway pods should recover after deleting {pod}: {err}") + }); + assert_exec_through_all_pods(&kube, &pods, &sandbox.name, &format!("delete-{pod}")) + .await + .unwrap_or_else(|err| panic!("exec should work after deleting {pod}: {err}")); + exec_through_configured_gateway( + &sandbox.name, + &format!("ha-rebalance-client-delete-{pod}"), + ) + .await + .unwrap_or_else(|err| { + panic!( + "exec should work through the configured client gateway endpoint after deleting {pod}: {err}" + ) + }); + let mut delete_sandbox = + create_sandbox_through_configured_gateway(&format!("delete-{idx}")) + .await + .unwrap_or_else(|err| { + panic!( + "sandbox create and readiness watch should succeed through the configured gateway endpoint after deleting {pod}: {err}" + ) + }); + delete_sandbox.cleanup().await; + } + + sandbox.cleanup().await; +} + +#[tokio::test] +async fn sandbox_file_sync_survives_gateway_pod_rolls() { + let _test_lock = KUBE_HA_TEST_LOCK.lock().await; + let kube = KubeTarget::from_env(); + + kube.scale_gateway(2) + .await + .expect("gateway should run with two HA replicas for sync outage testing"); + kube.wait_for_gateway_pods(2) + .await + .expect("gateway should have two ready replicas before sync outage testing"); + + let mut sandbox = + SandboxGuard::create_keep(&["sh", "-c", "echo Ready && sleep infinity"], "Ready") + .await + .expect("sandbox create --keep for HA sync testing"); + + let tmpdir = tempfile::tempdir().expect("create HA sync tmpdir"); + let upload_dir = tmpdir.path().join("ha-sync-upload"); + fs::create_dir_all(&upload_dir).expect("create HA sync upload dir"); + fs::write(upload_dir.join("marker.txt"), "ha-sync-marker").expect("write HA sync marker"); + + let payload = upload_dir.join("payload.bin"); + write_deterministic_payload(&payload, HA_SYNC_PAYLOAD_BYTES); + let expected_hash = sha256_file(&payload); + + let upload = upload_command(&sandbox.name, &upload_dir, "/sandbox/ha-sync"); + run_cli_during_gateway_pod_roll(&kube, upload, "upload") + .await + .expect("upload should survive rolling gateway pod outages"); + + let remote_payload = "/sandbox/ha-sync/ha-sync-upload/payload.bin"; + let remote_hash_cmd = format!("sha256sum {remote_payload} | awk '{{print $1}}'"); + let remote_hash = sandbox + .exec(&["sh", "-c", &remote_hash_cmd]) + .await + .expect("uploaded payload should be readable in sandbox"); + assert!( + strip_ansi(&remote_hash).contains(&expected_hash), + "uploaded payload SHA-256 mismatch; expected {expected_hash}, got:\n{remote_hash}" + ); + + let download_dir = tmpdir.path().join("ha-sync-download"); + fs::create_dir_all(&download_dir).expect("create HA sync download dir"); + let download = download_command( + &sandbox.name, + "/sandbox/ha-sync/ha-sync-upload", + &download_dir, + ); + run_cli_during_gateway_pod_roll(&kube, download, "download") + .await + .expect("download should survive rolling gateway pod outages"); + + let actual_hash = sha256_file(&download_dir.join("payload.bin")); + assert_eq!( + expected_hash, actual_hash, + "downloaded payload SHA-256 mismatch after gateway pod rolls" + ); + let marker = fs::read_to_string(download_dir.join("marker.txt")) + .expect("read downloaded HA sync marker"); + assert_eq!(marker, "ha-sync-marker", "downloaded marker mismatch"); + + sandbox.cleanup().await; +} diff --git a/e2e/rust/tests/readyz_health.rs b/e2e/rust/tests/readyz_health.rs deleted file mode 100644 index 8f093dabe8..0000000000 --- a/e2e/rust/tests/readyz_health.rs +++ /dev/null @@ -1,95 +0,0 @@ -// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -// SPDX-License-Identifier: Apache-2.0 - -#![cfg(feature = "e2e-kubernetes")] - -use bytes::Bytes; -use http_body_util::{BodyExt, Empty}; -use hyper::Request; -use hyper_util::rt::TokioIo; -use serde_json::Value; -use std::time::{Duration, Instant}; -use tokio::net::TcpStream; - -fn health_port_from_env() -> u16 { - let raw = std::env::var("OPENSHELL_E2E_HEALTH_PORT").unwrap_or_else(|_| { - panic!( - "OPENSHELL_E2E_HEALTH_PORT is not set. The Kubernetes e2e wrapper \ - (e2e/with-kube-gateway.sh) must export this variable so the \ - /readyz test can reach the gateway health listener." - ) - }); - raw.parse::().unwrap_or_else(|err| { - panic!("OPENSHELL_E2E_HEALTH_PORT=\"{raw}\" is not a valid u16 port: {err}") - }) -} - -async fn http_get_json(port: u16, path: &str) -> Result<(u16, Value), String> { - let stream = TcpStream::connect(("127.0.0.1", port)) - .await - .map_err(|err| format!("connect health endpoint :{port}: {err}"))?; - let (mut sender, conn) = hyper::client::conn::http1::Builder::new() - .handshake(TokioIo::new(stream)) - .await - .map_err(|err| format!("handshake health HTTP/1 client :{port}: {err}"))?; - tokio::spawn(async move { - let _ = conn.await; - }); - - let req = Request::builder() - .method("GET") - .uri(format!("http://127.0.0.1:{port}{path}")) - .body(Empty::::new()) - .map_err(|err| format!("build health request {path}: {err}"))?; - let resp = sender - .send_request(req) - .await - .map_err(|err| format!("send health request {path} to :{port}: {err}"))?; - let status_code = resp.status().as_u16(); - let bytes = resp - .into_body() - .collect() - .await - .map_err(|err| format!("read health response body {path}: {err}"))? - .to_bytes(); - let json = serde_json::from_slice::(&bytes) - .map_err(|err| format!("health endpoint {path} did not return valid JSON: {err}"))?; - - Ok((status_code, json)) -} - -#[tokio::test] -async fn readyz_reports_healthy_database_check() { - let port = health_port_from_env(); - - let deadline = Instant::now() + Duration::from_secs(20); - let timeout_detail = loop { - let observation = match http_get_json(port, "/readyz").await { - Ok((status, payload)) => { - let ready = status == 200 - && payload["status"] == "healthy" - && payload["checks"]["database"]["status"] == "healthy"; - if ready { - assert!( - payload["checks"]["database"]["latency_ms"].is_number(), - "readyz payload should include checks.database.latency_ms: {payload}" - ); - assert!( - payload["checks"]["database"]["error"].is_null(), - "readyz payload should not include checks.database.error when healthy: {payload}" - ); - return; - } - format!("unexpected /readyz response status={status} payload={payload}") - } - Err(err) => err, - }; - - if Instant::now() >= deadline { - break observation; - } - - tokio::time::sleep(Duration::from_secs(1)).await; - }; - panic!("timed out waiting for /readyz healthy response after 20s: {timeout_detail}"); -} diff --git a/e2e/with-kube-gateway.sh b/e2e/with-kube-gateway.sh index f83c8bafe1..4da96c724b 100755 --- a/e2e/with-kube-gateway.sh +++ b/e2e/with-kube-gateway.sh @@ -87,6 +87,12 @@ EXTERNAL_PG_FIXTURE_SERVICE="openshell-e2e-postgres" EXTERNAL_PG_FIXTURE_USER="openshell" EXTERNAL_PG_FIXTURE_PASSWORD="openshell-e2e-postgres" EXTERNAL_PG_FIXTURE_DATABASE="openshell" +ENVOY_RELEASE_NAME="${OPENSHELL_E2E_ENVOY_RELEASE_NAME:-envoy-gateway}" +ENVOY_NAMESPACE="${OPENSHELL_E2E_ENVOY_NAMESPACE:-envoy-gateway-system}" +ENVOY_CHART_VERSION="${OPENSHELL_E2E_ENVOY_VERSION:-v1.7.2}" +ENVOY_GATEWAY_MANIFEST="${ROOT}/deploy/kube/manifests/envoy-gateway-openshell.yaml" +ENVOY_HELM_INSTALLED=0 +ENVOY_GATEWAY_CONFIG_APPLIED=0 VAULT_FIXTURE_DEPLOYED=0 VAULT_NAMESPACE="${OPENSHELL_E2E_VAULT_NAMESPACE:-openbao}" VAULT_RELEASE_NAME="${OPENSHELL_E2E_VAULT_RELEASE_NAME:-openbao}" @@ -147,6 +153,122 @@ deploy_postgres_fixture() { --from-literal=uri="${pg_uri}" } +use_envoy_gateway() { + case "${OPENSHELL_E2E_KUBE_USE_ENVOY:-0}" in + 1 | true | TRUE | yes | YES) return 0 ;; + *) return 1 ;; + esac +} + +install_envoy_gateway() { + echo "Installing Envoy Gateway (${ENVOY_CHART_VERSION})..." + helmctl upgrade --install "${ENVOY_RELEASE_NAME}" \ + oci://docker.io/envoyproxy/gateway-helm \ + --version "${ENVOY_CHART_VERSION}" \ + --namespace "${ENVOY_NAMESPACE}" --create-namespace \ + --wait --timeout 5m + ENVOY_HELM_INSTALLED=1 + + if ! kctl get namespace "${NAMESPACE}" >/dev/null 2>&1; then + kctl create namespace "${NAMESPACE}" + fi + + kctl apply -f "${ENVOY_GATEWAY_MANIFEST}" + ENVOY_GATEWAY_CONFIG_APPLIED=1 +} + +wait_for_envoy_service() { + local svc_ref="" + local svc_namespace="" + + for _ in $(seq 1 60); do + svc_ref="$(kctl get svc -A \ + -l "gateway.envoyproxy.io/owning-gateway-name=${RELEASE_NAME},gateway.envoyproxy.io/owning-gateway-namespace=${NAMESPACE}" \ + -o jsonpath='{range .items[0]}{.metadata.namespace}{"/"}{.metadata.name}{end}' \ + 2>/dev/null || true)" + if [ -n "${svc_ref}" ]; then + svc_namespace="${svc_ref%%/*}" + if kctl -n "${svc_namespace}" wait --for=condition=Ready pod \ + -l "gateway.envoyproxy.io/owning-gateway-name=${RELEASE_NAME},gateway.envoyproxy.io/owning-gateway-namespace=${NAMESPACE}" \ + --timeout=5s >/dev/null 2>&1; then + printf '%s\n' "${svc_ref}" + return 0 + fi + fi + sleep 2 + done + + echo "ERROR: Envoy proxy Service for Gateway ${RELEASE_NAME} was not ready." >&2 + kctl -n "${NAMESPACE}" get gateway,grpcroute -o wide >&2 || true + kctl get svc -A \ + -l "gateway.envoyproxy.io/owning-gateway-name=${RELEASE_NAME},gateway.envoyproxy.io/owning-gateway-namespace=${NAMESPACE}" \ + -o wide >&2 || true + kctl get pods -A \ + -l "gateway.envoyproxy.io/owning-gateway-name=${RELEASE_NAME},gateway.envoyproxy.io/owning-gateway-namespace=${NAMESPACE}" \ + -o wide >&2 || true + return 1 +} + +start_gateway_portforward() { + local elapsed=0 + local pf_timeout=30 + local target_port=8080 + local target_namespace="${NAMESPACE}" + local target_service="${RELEASE_NAME}" + local target_service_ref="" + + LOCAL_PORT="$(e2e_pick_port)" + if use_envoy_gateway; then + target_service_ref="$(wait_for_envoy_service)" + target_namespace="${target_service_ref%%/*}" + target_service="${target_service_ref#*/}" + target_port=80 + echo "Starting kubectl port-forward -n ${target_namespace} svc/${target_service} ${LOCAL_PORT}:${target_port} (Envoy Gateway)..." + else + echo "Starting kubectl port-forward svc/${target_service} ${LOCAL_PORT}:${target_port}..." + fi + + kctl -n "${target_namespace}" port-forward "svc/${target_service}" \ + "${LOCAL_PORT}:${target_port}" >"${PORTFORWARD_LOG}" 2>&1 & + PORTFORWARD_PID=$! + + while [ "${elapsed}" -lt "${pf_timeout}" ]; do + if ! kill -0 "${PORTFORWARD_PID}" 2>/dev/null; then + echo "ERROR: kubectl port-forward exited before becoming reachable" >&2 + cat "${PORTFORWARD_LOG}" >&2 || true + return 1 + fi + if curl -s -o /dev/null --connect-timeout 1 "http://127.0.0.1:${LOCAL_PORT}"; then + return 0 + fi + sleep 1 + elapsed=$((elapsed + 1)) + done + + echo "ERROR: port-forward did not accept TCP within ${pf_timeout}s" >&2 + cat "${PORTFORWARD_LOG}" >&2 || true + return 1 +} + +stop_gateway_portforward() { + local pid + local pid_var + for pid_var in PORTFORWARD_PID PORTFORWARD_HEALTH_PID; do + pid="${!pid_var}" + [ -n "${pid}" ] || continue + kill "${pid}" >/dev/null 2>&1 || true + for _ in $(seq 1 10); do + if ! kill -0 "${pid}" >/dev/null 2>&1; then + break + fi + sleep 0.5 + done + kill -KILL "${pid}" >/dev/null 2>&1 || true + wait "${pid}" >/dev/null 2>&1 || true + printf -v "${pid_var}" '%s' "" + done +} + cleanup_postgres_fixture() { local secret_name="$1" @@ -206,15 +328,7 @@ cleanup_vault_fixture() { cleanup() { local exit_code=$? - if [ -n "${PORTFORWARD_PID}" ]; then - kill "${PORTFORWARD_PID}" >/dev/null 2>&1 || true - wait "${PORTFORWARD_PID}" >/dev/null 2>&1 || true - fi - - if [ -n "${PORTFORWARD_HEALTH_PID}" ]; then - kill "${PORTFORWARD_HEALTH_PID}" >/dev/null 2>&1 || true - wait "${PORTFORWARD_HEALTH_PID}" >/dev/null 2>&1 || true - fi + stop_gateway_portforward if [ "${exit_code}" -ne 0 ] && [ -n "${KUBE_CONTEXT}" ] && [ -n "${NAMESPACE}" ]; then if command -v kubectl >/dev/null 2>&1 \ @@ -250,6 +364,17 @@ cleanup() { cleanup_vault_fixture fi + if [ "${ENVOY_GATEWAY_CONFIG_APPLIED}" = "1" ] && [ -n "${KUBE_CONTEXT}" ]; then + if command -v kubectl >/dev/null 2>&1; then + kctl -n "${NAMESPACE}" delete backendtrafficpolicy.gateway.envoyproxy.io \ + openshell-grpc-timeouts --ignore-not-found --wait=false \ + >/dev/null 2>&1 || true + kctl delete gatewayclass.gateway.networking.k8s.io eg \ + --ignore-not-found --wait=false >/dev/null 2>&1 || true + fi + ENVOY_GATEWAY_CONFIG_APPLIED=0 + fi + if [ "${CORPORATE_PROXY_FIXTURE_DEPLOYED}" = "1" ]; then kctl -n "${NAMESPACE}" delete secret "${CORPORATE_PROXY_FIXTURE_SECRET}" \ --ignore-not-found >/dev/null 2>&1 || true @@ -284,6 +409,18 @@ cleanup() { fi fi + if [ "${ENVOY_HELM_INSTALLED}" = "1" ] && [ -n "${KUBE_CONTEXT}" ]; then + if command -v helm >/dev/null 2>&1; then + helmctl uninstall "${ENVOY_RELEASE_NAME}" --namespace "${ENVOY_NAMESPACE}" \ + --wait --timeout 60s >/dev/null 2>&1 || true + fi + if command -v kubectl >/dev/null 2>&1; then + kctl delete namespace "${ENVOY_NAMESPACE}" --wait=true --timeout=60s \ + --ignore-not-found >/dev/null 2>&1 || true + fi + ENVOY_HELM_INSTALLED=0 + fi + if [ "${CLUSTER_CREATED_BY_US}" = "1" ] && [ -n "${CLUSTER_NAME}" ]; then if command -v k3d >/dev/null 2>&1 && k3d cluster list "${CLUSTER_NAME}" \ >/dev/null 2>&1; then @@ -299,16 +436,7 @@ trap cleanup EXIT # --- DB-scenario helpers (used only when OPENSHELL_E2E_KUBE_DB_SCENARIOS=1) --- scenario_stop_portforward() { - if [ -n "${PORTFORWARD_PID}" ]; then - kill "${PORTFORWARD_PID}" >/dev/null 2>&1 || true - wait "${PORTFORWARD_PID}" >/dev/null 2>&1 || true - PORTFORWARD_PID="" - fi - if [ -n "${PORTFORWARD_HEALTH_PID}" ]; then - kill "${PORTFORWARD_HEALTH_PID}" >/dev/null 2>&1 || true - wait "${PORTFORWARD_HEALTH_PID}" >/dev/null 2>&1 || true - PORTFORWARD_HEALTH_PID="" - fi + stop_gateway_portforward } scenario_cleanup_release() { @@ -344,6 +472,8 @@ run_scenario() { local scenario_label="$1" shift 2 local scenario_exit=0 + local elapsed=0 + local pf_timeout=30 echo "" echo "========================================" @@ -363,34 +493,9 @@ run_scenario() { --wait --timeout 5m HELM_INSTALLED=1 - LOCAL_PORT="$(e2e_pick_port)" - echo "Starting kubectl port-forward svc/openshell ${LOCAL_PORT}:8080..." - kctl -n "${NAMESPACE}" port-forward "svc/openshell" \ - "${LOCAL_PORT}:8080" >"${PORTFORWARD_LOG}" 2>&1 & - PORTFORWARD_PID=$! - - local elapsed=0 pf_timeout=30 - while [ "${elapsed}" -lt "${pf_timeout}" ]; do - if ! kill -0 "${PORTFORWARD_PID}" 2>/dev/null; then - echo "ERROR: kubectl port-forward exited before becoming reachable" >&2 - cat "${PORTFORWARD_LOG}" >&2 || true - DB_FAILED=$((DB_FAILED + 1)) - DB_SCENARIOS_SUMMARY+=("FAIL ${scenario_label}: port-forward died") - scenario_stop_portforward - scenario_cleanup_release - return - fi - if curl -s -o /dev/null --connect-timeout 1 "http://127.0.0.1:${LOCAL_PORT}"; then - break - fi - sleep 1 - elapsed=$((elapsed + 1)) - done - if [ "${elapsed}" -ge "${pf_timeout}" ]; then - echo "ERROR: port-forward did not accept TCP within ${pf_timeout}s" >&2 - cat "${PORTFORWARD_LOG}" >&2 || true + if ! start_gateway_portforward; then DB_FAILED=$((DB_FAILED + 1)) - DB_SCENARIOS_SUMMARY+=("FAIL ${scenario_label}: port-forward timeout") + DB_SCENARIOS_SUMMARY+=("FAIL ${scenario_label}: port-forward failed") scenario_stop_portforward scenario_cleanup_release return @@ -450,6 +555,9 @@ run_scenario() { export CONTAINER_ENGINE="${CONTAINER_ENGINE:-docker}" export OPENSHELL_E2E_KUBE_CONTEXT_ACTIVE="${KUBE_CONTEXT}" export OPENSHELL_E2E_SANDBOX_NAMESPACE="${NAMESPACE}" + export OPENSHELL_E2E_KUBE_CONTEXT="${KUBE_CONTEXT}" + export OPENSHELL_E2E_KUBE_NAMESPACE="${NAMESPACE}" + export OPENSHELL_E2E_KUBE_RELEASE="${RELEASE_NAME}" export OPENSHELL_PROVISION_TIMEOUT="${OPENSHELL_PROVISION_TIMEOUT:-300}" echo "Running e2e command against ${GATEWAY_ENDPOINT}: ${E2E_CMD[*]}" @@ -799,6 +907,10 @@ if [ -n "${OPENSHELL_E2E_KUBE_EXTRA_VALUES:-}" ]; then helm_values_args+=(--values "${values_file}") done fi +if use_envoy_gateway; then + helm_values_args+=(--values "${ROOT}/deploy/helm/openshell/ci/values-gateway.yaml") + install_envoy_gateway +fi if [ "${OPENSHELL_E2E_KUBE_DB_SCENARIOS:-0}" = "1" ]; then # --- Multi-scenario mode: test all database backends --- @@ -858,31 +970,7 @@ else --docker-password=e2e-password fi - LOCAL_PORT="$(e2e_pick_port)" - echo "Starting kubectl port-forward svc/openshell ${LOCAL_PORT}:8080..." - kctl -n "${NAMESPACE}" port-forward "svc/openshell" \ - "${LOCAL_PORT}:8080" >"${PORTFORWARD_LOG}" 2>&1 & - PORTFORWARD_PID=$! - - elapsed=0 - timeout=30 - while [ "${elapsed}" -lt "${timeout}" ]; do - if ! kill -0 "${PORTFORWARD_PID}" 2>/dev/null; then - echo "ERROR: kubectl port-forward exited before becoming reachable" >&2 - cat "${PORTFORWARD_LOG}" >&2 || true - exit 1 - fi - if curl -s -o /dev/null --connect-timeout 1 "http://127.0.0.1:${LOCAL_PORT}"; then - break - fi - sleep 1 - elapsed=$((elapsed + 1)) - done - if [ "${elapsed}" -ge "${timeout}" ]; then - echo "ERROR: port-forward did not accept TCP within ${timeout}s" >&2 - cat "${PORTFORWARD_LOG}" >&2 || true - exit 1 - fi + start_gateway_portforward HEALTH_LOCAL_PORT="$(e2e_pick_port)" WORKLOAD_REF="$(kube_workload_ref "${RELEASE_NAME}")" @@ -930,6 +1018,9 @@ else export CONTAINER_ENGINE="${CONTAINER_ENGINE:-docker}" export OPENSHELL_E2E_KUBE_CONTEXT_ACTIVE="${KUBE_CONTEXT}" export OPENSHELL_E2E_SANDBOX_NAMESPACE="${NAMESPACE}" + export OPENSHELL_E2E_KUBE_CONTEXT="${KUBE_CONTEXT}" + export OPENSHELL_E2E_KUBE_NAMESPACE="${NAMESPACE}" + export OPENSHELL_E2E_KUBE_RELEASE="${RELEASE_NAME}" export OPENSHELL_PROVISION_TIMEOUT="${OPENSHELL_PROVISION_TIMEOUT:-300}" echo "Running e2e command against ${GATEWAY_ENDPOINT}: $*" diff --git a/proto/openshell.proto b/proto/openshell.proto index 6f1481b521..b321f84b5f 100644 --- a/proto/openshell.proto +++ b/proto/openshell.proto @@ -488,6 +488,19 @@ service OpenShell { }; } + // Internal gateway-to-gateway relay forwarding. + // + // A gateway replica that receives a user request for a sandbox whose + // supervisor session is owned by a different replica opens this stream to the + // owner. The first frame carries PeerRelayInit; subsequent frames carry raw + // bytes in either direction. This RPC is authenticated as a gateway peer, not + // as a user or sandbox supervisor. + rpc PeerRelay(stream PeerRelayFrame) returns (stream PeerRelayFrame) { + option (openshell.options.v1.authorization) = { + auth_mode: "peer" + }; + } + // Watch a sandbox and stream updates. // // This stream can include: @@ -2186,6 +2199,9 @@ message SupervisorHello { string sandbox_id = 1; // Supervisor instance ID (e.g. boot id or process epoch). string instance_id = 2; + // Monotonic counter scoped to instance_id. Incremented for each reconnect so + // gateways can distinguish a fresh supervisor connection from stale cleanup. + uint64 connection_epoch = 3; } // Gateway accepts the supervisor session. @@ -2266,6 +2282,25 @@ message RelayFrame { } } +// Initial frame for gateway peer relay forwarding. +message PeerRelayInit { + // Stable sandbox UUID whose supervisor relay should be opened. + string sandbox_id = 1; + // Relay target to ask the owning gateway to open on its local supervisor + // session. The channel_id is assigned by the forwarding gateway. + RelayOpen relay_open = 2; + // Gateway replica id that initiated the peer relay. + string requester_replica_id = 3; +} + +// A single frame on the gateway-to-gateway peer relay RPC. +message PeerRelayFrame { + oneof payload { + PeerRelayInit init = 1; + bytes data = 2; + } +} + // Supervisor reports the result of a relay open request. message RelayOpenResult { // Channel identifier from the RelayOpen request. diff --git a/sdk/go/proto/openshellv1/openshell.pb.go b/sdk/go/proto/openshellv1/openshell.pb.go index 778bf4f985..c606114eaa 100644 --- a/sdk/go/proto/openshellv1/openshell.pb.go +++ b/sdk/go/proto/openshellv1/openshell.pb.go @@ -9616,9 +9616,12 @@ type SupervisorHello struct { // Sandbox ID this supervisor manages. SandboxId string `protobuf:"bytes,1,opt,name=sandbox_id,json=sandboxId,proto3" json:"sandbox_id,omitempty"` // Supervisor instance ID (e.g. boot id or process epoch). - InstanceId string `protobuf:"bytes,2,opt,name=instance_id,json=instanceId,proto3" json:"instance_id,omitempty"` - unknownFields protoimpl.UnknownFields - sizeCache protoimpl.SizeCache + InstanceId string `protobuf:"bytes,2,opt,name=instance_id,json=instanceId,proto3" json:"instance_id,omitempty"` + // Monotonic counter scoped to instance_id. Incremented for each reconnect so + // gateways can distinguish a fresh supervisor connection from stale cleanup. + ConnectionEpoch uint64 `protobuf:"varint,3,opt,name=connection_epoch,json=connectionEpoch,proto3" json:"connection_epoch,omitempty"` + unknownFields protoimpl.UnknownFields + sizeCache protoimpl.SizeCache } func (x *SupervisorHello) Reset() { @@ -9665,6 +9668,13 @@ func (x *SupervisorHello) GetInstanceId() string { return "" } +func (x *SupervisorHello) GetConnectionEpoch() uint64 { + if x != nil { + return x.ConnectionEpoch + } + return 0 +} + // Gateway accepts the supervisor session. type SessionAccepted struct { state protoimpl.MessageState `protogen:"open.v1"` @@ -10272,6 +10282,154 @@ func (*RelayFrame_Init) isRelayFrame_Payload() {} func (*RelayFrame_Data) isRelayFrame_Payload() {} +// Initial frame for gateway peer relay forwarding. +type PeerRelayInit struct { + state protoimpl.MessageState `protogen:"open.v1"` + // Stable sandbox UUID whose supervisor relay should be opened. + SandboxId string `protobuf:"bytes,1,opt,name=sandbox_id,json=sandboxId,proto3" json:"sandbox_id,omitempty"` + // Relay target to ask the owning gateway to open on its local supervisor + // session. The channel_id is assigned by the forwarding gateway. + RelayOpen *RelayOpen `protobuf:"bytes,2,opt,name=relay_open,json=relayOpen,proto3" json:"relay_open,omitempty"` + // Gateway replica id that initiated the peer relay. + RequesterReplicaId string `protobuf:"bytes,3,opt,name=requester_replica_id,json=requesterReplicaId,proto3" json:"requester_replica_id,omitempty"` + unknownFields protoimpl.UnknownFields + sizeCache protoimpl.SizeCache +} + +func (x *PeerRelayInit) Reset() { + *x = PeerRelayInit{} + mi := &file_openshell_proto_msgTypes[144] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) +} + +func (x *PeerRelayInit) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*PeerRelayInit) ProtoMessage() {} + +func (x *PeerRelayInit) ProtoReflect() protoreflect.Message { + mi := &file_openshell_proto_msgTypes[144] + if x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use PeerRelayInit.ProtoReflect.Descriptor instead. +func (*PeerRelayInit) Descriptor() ([]byte, []int) { + return file_openshell_proto_rawDescGZIP(), []int{144} +} + +func (x *PeerRelayInit) GetSandboxId() string { + if x != nil { + return x.SandboxId + } + return "" +} + +func (x *PeerRelayInit) GetRelayOpen() *RelayOpen { + if x != nil { + return x.RelayOpen + } + return nil +} + +func (x *PeerRelayInit) GetRequesterReplicaId() string { + if x != nil { + return x.RequesterReplicaId + } + return "" +} + +// A single frame on the gateway-to-gateway peer relay RPC. +type PeerRelayFrame struct { + state protoimpl.MessageState `protogen:"open.v1"` + // Types that are valid to be assigned to Payload: + // + // *PeerRelayFrame_Init + // *PeerRelayFrame_Data + Payload isPeerRelayFrame_Payload `protobuf_oneof:"payload"` + unknownFields protoimpl.UnknownFields + sizeCache protoimpl.SizeCache +} + +func (x *PeerRelayFrame) Reset() { + *x = PeerRelayFrame{} + mi := &file_openshell_proto_msgTypes[145] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) +} + +func (x *PeerRelayFrame) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*PeerRelayFrame) ProtoMessage() {} + +func (x *PeerRelayFrame) ProtoReflect() protoreflect.Message { + mi := &file_openshell_proto_msgTypes[145] + if x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use PeerRelayFrame.ProtoReflect.Descriptor instead. +func (*PeerRelayFrame) Descriptor() ([]byte, []int) { + return file_openshell_proto_rawDescGZIP(), []int{145} +} + +func (x *PeerRelayFrame) GetPayload() isPeerRelayFrame_Payload { + if x != nil { + return x.Payload + } + return nil +} + +func (x *PeerRelayFrame) GetInit() *PeerRelayInit { + if x != nil { + if x, ok := x.Payload.(*PeerRelayFrame_Init); ok { + return x.Init + } + } + return nil +} + +func (x *PeerRelayFrame) GetData() []byte { + if x != nil { + if x, ok := x.Payload.(*PeerRelayFrame_Data); ok { + return x.Data + } + } + return nil +} + +type isPeerRelayFrame_Payload interface { + isPeerRelayFrame_Payload() +} + +type PeerRelayFrame_Init struct { + Init *PeerRelayInit `protobuf:"bytes,1,opt,name=init,proto3,oneof"` +} + +type PeerRelayFrame_Data struct { + Data []byte `protobuf:"bytes,2,opt,name=data,proto3,oneof"` +} + +func (*PeerRelayFrame_Init) isPeerRelayFrame_Payload() {} + +func (*PeerRelayFrame_Data) isPeerRelayFrame_Payload() {} + // Supervisor reports the result of a relay open request. type RelayOpenResult struct { state protoimpl.MessageState `protogen:"open.v1"` @@ -10287,7 +10445,7 @@ type RelayOpenResult struct { func (x *RelayOpenResult) Reset() { *x = RelayOpenResult{} - mi := &file_openshell_proto_msgTypes[144] + mi := &file_openshell_proto_msgTypes[146] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -10299,7 +10457,7 @@ func (x *RelayOpenResult) String() string { func (*RelayOpenResult) ProtoMessage() {} func (x *RelayOpenResult) ProtoReflect() protoreflect.Message { - mi := &file_openshell_proto_msgTypes[144] + mi := &file_openshell_proto_msgTypes[146] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -10312,7 +10470,7 @@ func (x *RelayOpenResult) ProtoReflect() protoreflect.Message { // Deprecated: Use RelayOpenResult.ProtoReflect.Descriptor instead. func (*RelayOpenResult) Descriptor() ([]byte, []int) { - return file_openshell_proto_rawDescGZIP(), []int{144} + return file_openshell_proto_rawDescGZIP(), []int{146} } func (x *RelayOpenResult) GetChannelId() string { @@ -10349,7 +10507,7 @@ type RelayClose struct { func (x *RelayClose) Reset() { *x = RelayClose{} - mi := &file_openshell_proto_msgTypes[145] + mi := &file_openshell_proto_msgTypes[147] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -10361,7 +10519,7 @@ func (x *RelayClose) String() string { func (*RelayClose) ProtoMessage() {} func (x *RelayClose) ProtoReflect() protoreflect.Message { - mi := &file_openshell_proto_msgTypes[145] + mi := &file_openshell_proto_msgTypes[147] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -10374,7 +10532,7 @@ func (x *RelayClose) ProtoReflect() protoreflect.Message { // Deprecated: Use RelayClose.ProtoReflect.Descriptor instead. func (*RelayClose) Descriptor() ([]byte, []int) { - return file_openshell_proto_rawDescGZIP(), []int{145} + return file_openshell_proto_rawDescGZIP(), []int{147} } func (x *RelayClose) GetChannelId() string { @@ -10408,7 +10566,7 @@ type L7RequestSample struct { func (x *L7RequestSample) Reset() { *x = L7RequestSample{} - mi := &file_openshell_proto_msgTypes[146] + mi := &file_openshell_proto_msgTypes[148] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -10420,7 +10578,7 @@ func (x *L7RequestSample) String() string { func (*L7RequestSample) ProtoMessage() {} func (x *L7RequestSample) ProtoReflect() protoreflect.Message { - mi := &file_openshell_proto_msgTypes[146] + mi := &file_openshell_proto_msgTypes[148] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -10433,7 +10591,7 @@ func (x *L7RequestSample) ProtoReflect() protoreflect.Message { // Deprecated: Use L7RequestSample.ProtoReflect.Descriptor instead. func (*L7RequestSample) Descriptor() ([]byte, []int) { - return file_openshell_proto_rawDescGZIP(), []int{146} + return file_openshell_proto_rawDescGZIP(), []int{148} } func (x *L7RequestSample) GetMethod() string { @@ -10507,7 +10665,7 @@ type DenialSummary struct { func (x *DenialSummary) Reset() { *x = DenialSummary{} - mi := &file_openshell_proto_msgTypes[147] + mi := &file_openshell_proto_msgTypes[149] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -10519,7 +10677,7 @@ func (x *DenialSummary) String() string { func (*DenialSummary) ProtoMessage() {} func (x *DenialSummary) ProtoReflect() protoreflect.Message { - mi := &file_openshell_proto_msgTypes[147] + mi := &file_openshell_proto_msgTypes[149] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -10532,7 +10690,7 @@ func (x *DenialSummary) ProtoReflect() protoreflect.Message { // Deprecated: Use DenialSummary.ProtoReflect.Descriptor instead. func (*DenialSummary) Descriptor() ([]byte, []int) { - return file_openshell_proto_rawDescGZIP(), []int{147} + return file_openshell_proto_rawDescGZIP(), []int{149} } func (x *DenialSummary) GetSandboxId() string { @@ -10667,7 +10825,7 @@ type DenialGroupCount struct { func (x *DenialGroupCount) Reset() { *x = DenialGroupCount{} - mi := &file_openshell_proto_msgTypes[148] + mi := &file_openshell_proto_msgTypes[150] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -10679,7 +10837,7 @@ func (x *DenialGroupCount) String() string { func (*DenialGroupCount) ProtoMessage() {} func (x *DenialGroupCount) ProtoReflect() protoreflect.Message { - mi := &file_openshell_proto_msgTypes[148] + mi := &file_openshell_proto_msgTypes[150] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -10692,7 +10850,7 @@ func (x *DenialGroupCount) ProtoReflect() protoreflect.Message { // Deprecated: Use DenialGroupCount.ProtoReflect.Descriptor instead. func (*DenialGroupCount) Descriptor() ([]byte, []int) { - return file_openshell_proto_rawDescGZIP(), []int{148} + return file_openshell_proto_rawDescGZIP(), []int{150} } func (x *DenialGroupCount) GetDenyGroup() string { @@ -10725,7 +10883,7 @@ type NetworkActivitySummary struct { func (x *NetworkActivitySummary) Reset() { *x = NetworkActivitySummary{} - mi := &file_openshell_proto_msgTypes[149] + mi := &file_openshell_proto_msgTypes[151] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -10737,7 +10895,7 @@ func (x *NetworkActivitySummary) String() string { func (*NetworkActivitySummary) ProtoMessage() {} func (x *NetworkActivitySummary) ProtoReflect() protoreflect.Message { - mi := &file_openshell_proto_msgTypes[149] + mi := &file_openshell_proto_msgTypes[151] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -10750,7 +10908,7 @@ func (x *NetworkActivitySummary) ProtoReflect() protoreflect.Message { // Deprecated: Use NetworkActivitySummary.ProtoReflect.Descriptor instead. func (*NetworkActivitySummary) Descriptor() ([]byte, []int) { - return file_openshell_proto_rawDescGZIP(), []int{149} + return file_openshell_proto_rawDescGZIP(), []int{151} } func (x *NetworkActivitySummary) GetNetworkActivityCount() uint32 { @@ -10838,7 +10996,7 @@ type PolicyChunk struct { func (x *PolicyChunk) Reset() { *x = PolicyChunk{} - mi := &file_openshell_proto_msgTypes[150] + mi := &file_openshell_proto_msgTypes[152] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -10850,7 +11008,7 @@ func (x *PolicyChunk) String() string { func (*PolicyChunk) ProtoMessage() {} func (x *PolicyChunk) ProtoReflect() protoreflect.Message { - mi := &file_openshell_proto_msgTypes[150] + mi := &file_openshell_proto_msgTypes[152] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -10863,7 +11021,7 @@ func (x *PolicyChunk) ProtoReflect() protoreflect.Message { // Deprecated: Use PolicyChunk.ProtoReflect.Descriptor instead. func (*PolicyChunk) Descriptor() ([]byte, []int) { - return file_openshell_proto_rawDescGZIP(), []int{150} + return file_openshell_proto_rawDescGZIP(), []int{152} } func (x *PolicyChunk) GetId() string { @@ -11051,7 +11209,7 @@ type DraftPolicyUpdate struct { func (x *DraftPolicyUpdate) Reset() { *x = DraftPolicyUpdate{} - mi := &file_openshell_proto_msgTypes[151] + mi := &file_openshell_proto_msgTypes[153] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -11063,7 +11221,7 @@ func (x *DraftPolicyUpdate) String() string { func (*DraftPolicyUpdate) ProtoMessage() {} func (x *DraftPolicyUpdate) ProtoReflect() protoreflect.Message { - mi := &file_openshell_proto_msgTypes[151] + mi := &file_openshell_proto_msgTypes[153] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -11076,7 +11234,7 @@ func (x *DraftPolicyUpdate) ProtoReflect() protoreflect.Message { // Deprecated: Use DraftPolicyUpdate.ProtoReflect.Descriptor instead. func (*DraftPolicyUpdate) Descriptor() ([]byte, []int) { - return file_openshell_proto_rawDescGZIP(), []int{151} + return file_openshell_proto_rawDescGZIP(), []int{153} } func (x *DraftPolicyUpdate) GetDraftVersion() uint64 { @@ -11134,7 +11292,7 @@ type SubmitPolicyAnalysisRequest struct { func (x *SubmitPolicyAnalysisRequest) Reset() { *x = SubmitPolicyAnalysisRequest{} - mi := &file_openshell_proto_msgTypes[152] + mi := &file_openshell_proto_msgTypes[154] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -11146,7 +11304,7 @@ func (x *SubmitPolicyAnalysisRequest) String() string { func (*SubmitPolicyAnalysisRequest) ProtoMessage() {} func (x *SubmitPolicyAnalysisRequest) ProtoReflect() protoreflect.Message { - mi := &file_openshell_proto_msgTypes[152] + mi := &file_openshell_proto_msgTypes[154] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -11159,7 +11317,7 @@ func (x *SubmitPolicyAnalysisRequest) ProtoReflect() protoreflect.Message { // Deprecated: Use SubmitPolicyAnalysisRequest.ProtoReflect.Descriptor instead. func (*SubmitPolicyAnalysisRequest) Descriptor() ([]byte, []int) { - return file_openshell_proto_rawDescGZIP(), []int{152} + return file_openshell_proto_rawDescGZIP(), []int{154} } func (x *SubmitPolicyAnalysisRequest) GetSummaries() []*DenialSummary { @@ -11222,7 +11380,7 @@ type SubmitPolicyAnalysisResponse struct { func (x *SubmitPolicyAnalysisResponse) Reset() { *x = SubmitPolicyAnalysisResponse{} - mi := &file_openshell_proto_msgTypes[153] + mi := &file_openshell_proto_msgTypes[155] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -11234,7 +11392,7 @@ func (x *SubmitPolicyAnalysisResponse) String() string { func (*SubmitPolicyAnalysisResponse) ProtoMessage() {} func (x *SubmitPolicyAnalysisResponse) ProtoReflect() protoreflect.Message { - mi := &file_openshell_proto_msgTypes[153] + mi := &file_openshell_proto_msgTypes[155] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -11247,7 +11405,7 @@ func (x *SubmitPolicyAnalysisResponse) ProtoReflect() protoreflect.Message { // Deprecated: Use SubmitPolicyAnalysisResponse.ProtoReflect.Descriptor instead. func (*SubmitPolicyAnalysisResponse) Descriptor() ([]byte, []int) { - return file_openshell_proto_rawDescGZIP(), []int{153} + return file_openshell_proto_rawDescGZIP(), []int{155} } func (x *SubmitPolicyAnalysisResponse) GetAcceptedChunks() uint32 { @@ -11293,7 +11451,7 @@ type GetDraftPolicyRequest struct { func (x *GetDraftPolicyRequest) Reset() { *x = GetDraftPolicyRequest{} - mi := &file_openshell_proto_msgTypes[154] + mi := &file_openshell_proto_msgTypes[156] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -11305,7 +11463,7 @@ func (x *GetDraftPolicyRequest) String() string { func (*GetDraftPolicyRequest) ProtoMessage() {} func (x *GetDraftPolicyRequest) ProtoReflect() protoreflect.Message { - mi := &file_openshell_proto_msgTypes[154] + mi := &file_openshell_proto_msgTypes[156] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -11318,7 +11476,7 @@ func (x *GetDraftPolicyRequest) ProtoReflect() protoreflect.Message { // Deprecated: Use GetDraftPolicyRequest.ProtoReflect.Descriptor instead. func (*GetDraftPolicyRequest) Descriptor() ([]byte, []int) { - return file_openshell_proto_rawDescGZIP(), []int{154} + return file_openshell_proto_rawDescGZIP(), []int{156} } func (x *GetDraftPolicyRequest) GetName() string { @@ -11358,7 +11516,7 @@ type GetDraftPolicyResponse struct { func (x *GetDraftPolicyResponse) Reset() { *x = GetDraftPolicyResponse{} - mi := &file_openshell_proto_msgTypes[155] + mi := &file_openshell_proto_msgTypes[157] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -11370,7 +11528,7 @@ func (x *GetDraftPolicyResponse) String() string { func (*GetDraftPolicyResponse) ProtoMessage() {} func (x *GetDraftPolicyResponse) ProtoReflect() protoreflect.Message { - mi := &file_openshell_proto_msgTypes[155] + mi := &file_openshell_proto_msgTypes[157] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -11383,7 +11541,7 @@ func (x *GetDraftPolicyResponse) ProtoReflect() protoreflect.Message { // Deprecated: Use GetDraftPolicyResponse.ProtoReflect.Descriptor instead. func (*GetDraftPolicyResponse) Descriptor() ([]byte, []int) { - return file_openshell_proto_rawDescGZIP(), []int{155} + return file_openshell_proto_rawDescGZIP(), []int{157} } func (x *GetDraftPolicyResponse) GetChunks() []*PolicyChunk { @@ -11432,7 +11590,7 @@ type ApproveDraftChunkRequest struct { func (x *ApproveDraftChunkRequest) Reset() { *x = ApproveDraftChunkRequest{} - mi := &file_openshell_proto_msgTypes[156] + mi := &file_openshell_proto_msgTypes[158] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -11444,7 +11602,7 @@ func (x *ApproveDraftChunkRequest) String() string { func (*ApproveDraftChunkRequest) ProtoMessage() {} func (x *ApproveDraftChunkRequest) ProtoReflect() protoreflect.Message { - mi := &file_openshell_proto_msgTypes[156] + mi := &file_openshell_proto_msgTypes[158] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -11457,7 +11615,7 @@ func (x *ApproveDraftChunkRequest) ProtoReflect() protoreflect.Message { // Deprecated: Use ApproveDraftChunkRequest.ProtoReflect.Descriptor instead. func (*ApproveDraftChunkRequest) Descriptor() ([]byte, []int) { - return file_openshell_proto_rawDescGZIP(), []int{156} + return file_openshell_proto_rawDescGZIP(), []int{158} } func (x *ApproveDraftChunkRequest) GetName() string { @@ -11500,7 +11658,7 @@ type ApproveDraftChunkResponse struct { func (x *ApproveDraftChunkResponse) Reset() { *x = ApproveDraftChunkResponse{} - mi := &file_openshell_proto_msgTypes[157] + mi := &file_openshell_proto_msgTypes[159] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -11512,7 +11670,7 @@ func (x *ApproveDraftChunkResponse) String() string { func (*ApproveDraftChunkResponse) ProtoMessage() {} func (x *ApproveDraftChunkResponse) ProtoReflect() protoreflect.Message { - mi := &file_openshell_proto_msgTypes[157] + mi := &file_openshell_proto_msgTypes[159] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -11525,7 +11683,7 @@ func (x *ApproveDraftChunkResponse) ProtoReflect() protoreflect.Message { // Deprecated: Use ApproveDraftChunkResponse.ProtoReflect.Descriptor instead. func (*ApproveDraftChunkResponse) Descriptor() ([]byte, []int) { - return file_openshell_proto_rawDescGZIP(), []int{157} + return file_openshell_proto_rawDescGZIP(), []int{159} } func (x *ApproveDraftChunkResponse) GetPolicyVersion() uint32 { @@ -11559,7 +11717,7 @@ type RejectDraftChunkRequest struct { func (x *RejectDraftChunkRequest) Reset() { *x = RejectDraftChunkRequest{} - mi := &file_openshell_proto_msgTypes[158] + mi := &file_openshell_proto_msgTypes[160] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -11571,7 +11729,7 @@ func (x *RejectDraftChunkRequest) String() string { func (*RejectDraftChunkRequest) ProtoMessage() {} func (x *RejectDraftChunkRequest) ProtoReflect() protoreflect.Message { - mi := &file_openshell_proto_msgTypes[158] + mi := &file_openshell_proto_msgTypes[160] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -11584,7 +11742,7 @@ func (x *RejectDraftChunkRequest) ProtoReflect() protoreflect.Message { // Deprecated: Use RejectDraftChunkRequest.ProtoReflect.Descriptor instead. func (*RejectDraftChunkRequest) Descriptor() ([]byte, []int) { - return file_openshell_proto_rawDescGZIP(), []int{158} + return file_openshell_proto_rawDescGZIP(), []int{160} } func (x *RejectDraftChunkRequest) GetName() string { @@ -11623,7 +11781,7 @@ type RejectDraftChunkResponse struct { func (x *RejectDraftChunkResponse) Reset() { *x = RejectDraftChunkResponse{} - mi := &file_openshell_proto_msgTypes[159] + mi := &file_openshell_proto_msgTypes[161] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -11635,7 +11793,7 @@ func (x *RejectDraftChunkResponse) String() string { func (*RejectDraftChunkResponse) ProtoMessage() {} func (x *RejectDraftChunkResponse) ProtoReflect() protoreflect.Message { - mi := &file_openshell_proto_msgTypes[159] + mi := &file_openshell_proto_msgTypes[161] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -11648,7 +11806,7 @@ func (x *RejectDraftChunkResponse) ProtoReflect() protoreflect.Message { // Deprecated: Use RejectDraftChunkResponse.ProtoReflect.Descriptor instead. func (*RejectDraftChunkResponse) Descriptor() ([]byte, []int) { - return file_openshell_proto_rawDescGZIP(), []int{159} + return file_openshell_proto_rawDescGZIP(), []int{161} } // Approve all pending chunks. @@ -11662,7 +11820,7 @@ type DraftChunkApproval struct { func (x *DraftChunkApproval) Reset() { *x = DraftChunkApproval{} - mi := &file_openshell_proto_msgTypes[160] + mi := &file_openshell_proto_msgTypes[162] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -11674,7 +11832,7 @@ func (x *DraftChunkApproval) String() string { func (*DraftChunkApproval) ProtoMessage() {} func (x *DraftChunkApproval) ProtoReflect() protoreflect.Message { - mi := &file_openshell_proto_msgTypes[160] + mi := &file_openshell_proto_msgTypes[162] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -11687,7 +11845,7 @@ func (x *DraftChunkApproval) ProtoReflect() protoreflect.Message { // Deprecated: Use DraftChunkApproval.ProtoReflect.Descriptor instead. func (*DraftChunkApproval) Descriptor() ([]byte, []int) { - return file_openshell_proto_rawDescGZIP(), []int{160} + return file_openshell_proto_rawDescGZIP(), []int{162} } func (x *DraftChunkApproval) GetChunkId() string { @@ -11721,7 +11879,7 @@ type ApproveAllDraftChunksRequest struct { func (x *ApproveAllDraftChunksRequest) Reset() { *x = ApproveAllDraftChunksRequest{} - mi := &file_openshell_proto_msgTypes[161] + mi := &file_openshell_proto_msgTypes[163] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -11733,7 +11891,7 @@ func (x *ApproveAllDraftChunksRequest) String() string { func (*ApproveAllDraftChunksRequest) ProtoMessage() {} func (x *ApproveAllDraftChunksRequest) ProtoReflect() protoreflect.Message { - mi := &file_openshell_proto_msgTypes[161] + mi := &file_openshell_proto_msgTypes[163] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -11746,7 +11904,7 @@ func (x *ApproveAllDraftChunksRequest) ProtoReflect() protoreflect.Message { // Deprecated: Use ApproveAllDraftChunksRequest.ProtoReflect.Descriptor instead. func (*ApproveAllDraftChunksRequest) Descriptor() ([]byte, []int) { - return file_openshell_proto_rawDescGZIP(), []int{161} + return file_openshell_proto_rawDescGZIP(), []int{163} } func (x *ApproveAllDraftChunksRequest) GetName() string { @@ -11794,7 +11952,7 @@ type ApproveAllDraftChunksResponse struct { func (x *ApproveAllDraftChunksResponse) Reset() { *x = ApproveAllDraftChunksResponse{} - mi := &file_openshell_proto_msgTypes[162] + mi := &file_openshell_proto_msgTypes[164] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -11806,7 +11964,7 @@ func (x *ApproveAllDraftChunksResponse) String() string { func (*ApproveAllDraftChunksResponse) ProtoMessage() {} func (x *ApproveAllDraftChunksResponse) ProtoReflect() protoreflect.Message { - mi := &file_openshell_proto_msgTypes[162] + mi := &file_openshell_proto_msgTypes[164] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -11819,7 +11977,7 @@ func (x *ApproveAllDraftChunksResponse) ProtoReflect() protoreflect.Message { // Deprecated: Use ApproveAllDraftChunksResponse.ProtoReflect.Descriptor instead. func (*ApproveAllDraftChunksResponse) Descriptor() ([]byte, []int) { - return file_openshell_proto_rawDescGZIP(), []int{162} + return file_openshell_proto_rawDescGZIP(), []int{164} } func (x *ApproveAllDraftChunksResponse) GetPolicyVersion() uint32 { @@ -11867,7 +12025,7 @@ type EditDraftChunkRequest struct { func (x *EditDraftChunkRequest) Reset() { *x = EditDraftChunkRequest{} - mi := &file_openshell_proto_msgTypes[163] + mi := &file_openshell_proto_msgTypes[165] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -11879,7 +12037,7 @@ func (x *EditDraftChunkRequest) String() string { func (*EditDraftChunkRequest) ProtoMessage() {} func (x *EditDraftChunkRequest) ProtoReflect() protoreflect.Message { - mi := &file_openshell_proto_msgTypes[163] + mi := &file_openshell_proto_msgTypes[165] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -11892,7 +12050,7 @@ func (x *EditDraftChunkRequest) ProtoReflect() protoreflect.Message { // Deprecated: Use EditDraftChunkRequest.ProtoReflect.Descriptor instead. func (*EditDraftChunkRequest) Descriptor() ([]byte, []int) { - return file_openshell_proto_rawDescGZIP(), []int{163} + return file_openshell_proto_rawDescGZIP(), []int{165} } func (x *EditDraftChunkRequest) GetName() string { @@ -11931,7 +12089,7 @@ type EditDraftChunkResponse struct { func (x *EditDraftChunkResponse) Reset() { *x = EditDraftChunkResponse{} - mi := &file_openshell_proto_msgTypes[164] + mi := &file_openshell_proto_msgTypes[166] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -11943,7 +12101,7 @@ func (x *EditDraftChunkResponse) String() string { func (*EditDraftChunkResponse) ProtoMessage() {} func (x *EditDraftChunkResponse) ProtoReflect() protoreflect.Message { - mi := &file_openshell_proto_msgTypes[164] + mi := &file_openshell_proto_msgTypes[166] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -11956,7 +12114,7 @@ func (x *EditDraftChunkResponse) ProtoReflect() protoreflect.Message { // Deprecated: Use EditDraftChunkResponse.ProtoReflect.Descriptor instead. func (*EditDraftChunkResponse) Descriptor() ([]byte, []int) { - return file_openshell_proto_rawDescGZIP(), []int{164} + return file_openshell_proto_rawDescGZIP(), []int{166} } // Reverse an approval (remove merged rule from active policy). @@ -11974,7 +12132,7 @@ type UndoDraftChunkRequest struct { func (x *UndoDraftChunkRequest) Reset() { *x = UndoDraftChunkRequest{} - mi := &file_openshell_proto_msgTypes[165] + mi := &file_openshell_proto_msgTypes[167] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -11986,7 +12144,7 @@ func (x *UndoDraftChunkRequest) String() string { func (*UndoDraftChunkRequest) ProtoMessage() {} func (x *UndoDraftChunkRequest) ProtoReflect() protoreflect.Message { - mi := &file_openshell_proto_msgTypes[165] + mi := &file_openshell_proto_msgTypes[167] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -11999,7 +12157,7 @@ func (x *UndoDraftChunkRequest) ProtoReflect() protoreflect.Message { // Deprecated: Use UndoDraftChunkRequest.ProtoReflect.Descriptor instead. func (*UndoDraftChunkRequest) Descriptor() ([]byte, []int) { - return file_openshell_proto_rawDescGZIP(), []int{165} + return file_openshell_proto_rawDescGZIP(), []int{167} } func (x *UndoDraftChunkRequest) GetName() string { @@ -12035,7 +12193,7 @@ type UndoDraftChunkResponse struct { func (x *UndoDraftChunkResponse) Reset() { *x = UndoDraftChunkResponse{} - mi := &file_openshell_proto_msgTypes[166] + mi := &file_openshell_proto_msgTypes[168] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -12047,7 +12205,7 @@ func (x *UndoDraftChunkResponse) String() string { func (*UndoDraftChunkResponse) ProtoMessage() {} func (x *UndoDraftChunkResponse) ProtoReflect() protoreflect.Message { - mi := &file_openshell_proto_msgTypes[166] + mi := &file_openshell_proto_msgTypes[168] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -12060,7 +12218,7 @@ func (x *UndoDraftChunkResponse) ProtoReflect() protoreflect.Message { // Deprecated: Use UndoDraftChunkResponse.ProtoReflect.Descriptor instead. func (*UndoDraftChunkResponse) Descriptor() ([]byte, []int) { - return file_openshell_proto_rawDescGZIP(), []int{166} + return file_openshell_proto_rawDescGZIP(), []int{168} } func (x *UndoDraftChunkResponse) GetPolicyVersion() uint32 { @@ -12090,7 +12248,7 @@ type ClearDraftChunksRequest struct { func (x *ClearDraftChunksRequest) Reset() { *x = ClearDraftChunksRequest{} - mi := &file_openshell_proto_msgTypes[167] + mi := &file_openshell_proto_msgTypes[169] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -12102,7 +12260,7 @@ func (x *ClearDraftChunksRequest) String() string { func (*ClearDraftChunksRequest) ProtoMessage() {} func (x *ClearDraftChunksRequest) ProtoReflect() protoreflect.Message { - mi := &file_openshell_proto_msgTypes[167] + mi := &file_openshell_proto_msgTypes[169] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -12115,7 +12273,7 @@ func (x *ClearDraftChunksRequest) ProtoReflect() protoreflect.Message { // Deprecated: Use ClearDraftChunksRequest.ProtoReflect.Descriptor instead. func (*ClearDraftChunksRequest) Descriptor() ([]byte, []int) { - return file_openshell_proto_rawDescGZIP(), []int{167} + return file_openshell_proto_rawDescGZIP(), []int{169} } func (x *ClearDraftChunksRequest) GetName() string { @@ -12142,7 +12300,7 @@ type ClearDraftChunksResponse struct { func (x *ClearDraftChunksResponse) Reset() { *x = ClearDraftChunksResponse{} - mi := &file_openshell_proto_msgTypes[168] + mi := &file_openshell_proto_msgTypes[170] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -12154,7 +12312,7 @@ func (x *ClearDraftChunksResponse) String() string { func (*ClearDraftChunksResponse) ProtoMessage() {} func (x *ClearDraftChunksResponse) ProtoReflect() protoreflect.Message { - mi := &file_openshell_proto_msgTypes[168] + mi := &file_openshell_proto_msgTypes[170] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -12167,7 +12325,7 @@ func (x *ClearDraftChunksResponse) ProtoReflect() protoreflect.Message { // Deprecated: Use ClearDraftChunksResponse.ProtoReflect.Descriptor instead. func (*ClearDraftChunksResponse) Descriptor() ([]byte, []int) { - return file_openshell_proto_rawDescGZIP(), []int{168} + return file_openshell_proto_rawDescGZIP(), []int{170} } func (x *ClearDraftChunksResponse) GetChunksCleared() uint32 { @@ -12190,7 +12348,7 @@ type GetDraftHistoryRequest struct { func (x *GetDraftHistoryRequest) Reset() { *x = GetDraftHistoryRequest{} - mi := &file_openshell_proto_msgTypes[169] + mi := &file_openshell_proto_msgTypes[171] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -12202,7 +12360,7 @@ func (x *GetDraftHistoryRequest) String() string { func (*GetDraftHistoryRequest) ProtoMessage() {} func (x *GetDraftHistoryRequest) ProtoReflect() protoreflect.Message { - mi := &file_openshell_proto_msgTypes[169] + mi := &file_openshell_proto_msgTypes[171] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -12215,7 +12373,7 @@ func (x *GetDraftHistoryRequest) ProtoReflect() protoreflect.Message { // Deprecated: Use GetDraftHistoryRequest.ProtoReflect.Descriptor instead. func (*GetDraftHistoryRequest) Descriptor() ([]byte, []int) { - return file_openshell_proto_rawDescGZIP(), []int{169} + return file_openshell_proto_rawDescGZIP(), []int{171} } func (x *GetDraftHistoryRequest) GetName() string { @@ -12249,7 +12407,7 @@ type DraftHistoryEntry struct { func (x *DraftHistoryEntry) Reset() { *x = DraftHistoryEntry{} - mi := &file_openshell_proto_msgTypes[170] + mi := &file_openshell_proto_msgTypes[172] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -12261,7 +12419,7 @@ func (x *DraftHistoryEntry) String() string { func (*DraftHistoryEntry) ProtoMessage() {} func (x *DraftHistoryEntry) ProtoReflect() protoreflect.Message { - mi := &file_openshell_proto_msgTypes[170] + mi := &file_openshell_proto_msgTypes[172] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -12274,7 +12432,7 @@ func (x *DraftHistoryEntry) ProtoReflect() protoreflect.Message { // Deprecated: Use DraftHistoryEntry.ProtoReflect.Descriptor instead. func (*DraftHistoryEntry) Descriptor() ([]byte, []int) { - return file_openshell_proto_rawDescGZIP(), []int{170} + return file_openshell_proto_rawDescGZIP(), []int{172} } func (x *DraftHistoryEntry) GetTimestampMs() int64 { @@ -12315,7 +12473,7 @@ type GetDraftHistoryResponse struct { func (x *GetDraftHistoryResponse) Reset() { *x = GetDraftHistoryResponse{} - mi := &file_openshell_proto_msgTypes[171] + mi := &file_openshell_proto_msgTypes[173] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -12327,7 +12485,7 @@ func (x *GetDraftHistoryResponse) String() string { func (*GetDraftHistoryResponse) ProtoMessage() {} func (x *GetDraftHistoryResponse) ProtoReflect() protoreflect.Message { - mi := &file_openshell_proto_msgTypes[171] + mi := &file_openshell_proto_msgTypes[173] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -12340,7 +12498,7 @@ func (x *GetDraftHistoryResponse) ProtoReflect() protoreflect.Message { // Deprecated: Use GetDraftHistoryResponse.ProtoReflect.Descriptor instead. func (*GetDraftHistoryResponse) Descriptor() ([]byte, []int) { - return file_openshell_proto_rawDescGZIP(), []int{171} + return file_openshell_proto_rawDescGZIP(), []int{173} } func (x *GetDraftHistoryResponse) GetEntries() []*DraftHistoryEntry { @@ -12369,7 +12527,7 @@ type PolicyRevisionPayload struct { func (x *PolicyRevisionPayload) Reset() { *x = PolicyRevisionPayload{} - mi := &file_openshell_proto_msgTypes[172] + mi := &file_openshell_proto_msgTypes[174] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -12381,7 +12539,7 @@ func (x *PolicyRevisionPayload) String() string { func (*PolicyRevisionPayload) ProtoMessage() {} func (x *PolicyRevisionPayload) ProtoReflect() protoreflect.Message { - mi := &file_openshell_proto_msgTypes[172] + mi := &file_openshell_proto_msgTypes[174] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -12394,7 +12552,7 @@ func (x *PolicyRevisionPayload) ProtoReflect() protoreflect.Message { // Deprecated: Use PolicyRevisionPayload.ProtoReflect.Descriptor instead. func (*PolicyRevisionPayload) Descriptor() ([]byte, []int) { - return file_openshell_proto_rawDescGZIP(), []int{172} + return file_openshell_proto_rawDescGZIP(), []int{174} } func (x *PolicyRevisionPayload) GetPolicy() *sandboxv1.SandboxPolicy { @@ -12473,7 +12631,7 @@ type DraftChunkPayload struct { func (x *DraftChunkPayload) Reset() { *x = DraftChunkPayload{} - mi := &file_openshell_proto_msgTypes[173] + mi := &file_openshell_proto_msgTypes[175] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -12485,7 +12643,7 @@ func (x *DraftChunkPayload) String() string { func (*DraftChunkPayload) ProtoMessage() {} func (x *DraftChunkPayload) ProtoReflect() protoreflect.Message { - mi := &file_openshell_proto_msgTypes[173] + mi := &file_openshell_proto_msgTypes[175] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -12498,7 +12656,7 @@ func (x *DraftChunkPayload) ProtoReflect() protoreflect.Message { // Deprecated: Use DraftChunkPayload.ProtoReflect.Descriptor instead. func (*DraftChunkPayload) Descriptor() ([]byte, []int) { - return file_openshell_proto_rawDescGZIP(), []int{173} + return file_openshell_proto_rawDescGZIP(), []int{175} } func (x *DraftChunkPayload) GetRuleName() string { @@ -12646,7 +12804,7 @@ type StoredPolicyRevision struct { func (x *StoredPolicyRevision) Reset() { *x = StoredPolicyRevision{} - mi := &file_openshell_proto_msgTypes[174] + mi := &file_openshell_proto_msgTypes[176] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -12658,7 +12816,7 @@ func (x *StoredPolicyRevision) String() string { func (*StoredPolicyRevision) ProtoMessage() {} func (x *StoredPolicyRevision) ProtoReflect() protoreflect.Message { - mi := &file_openshell_proto_msgTypes[174] + mi := &file_openshell_proto_msgTypes[176] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -12671,7 +12829,7 @@ func (x *StoredPolicyRevision) ProtoReflect() protoreflect.Message { // Deprecated: Use StoredPolicyRevision.ProtoReflect.Descriptor instead. func (*StoredPolicyRevision) Descriptor() ([]byte, []int) { - return file_openshell_proto_rawDescGZIP(), []int{174} + return file_openshell_proto_rawDescGZIP(), []int{176} } func (x *StoredPolicyRevision) GetId() string { @@ -12780,7 +12938,7 @@ type StoredDraftChunk struct { func (x *StoredDraftChunk) Reset() { *x = StoredDraftChunk{} - mi := &file_openshell_proto_msgTypes[175] + mi := &file_openshell_proto_msgTypes[177] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -12792,7 +12950,7 @@ func (x *StoredDraftChunk) String() string { func (*StoredDraftChunk) ProtoMessage() {} func (x *StoredDraftChunk) ProtoReflect() protoreflect.Message { - mi := &file_openshell_proto_msgTypes[175] + mi := &file_openshell_proto_msgTypes[177] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -12805,7 +12963,7 @@ func (x *StoredDraftChunk) ProtoReflect() protoreflect.Message { // Deprecated: Use StoredDraftChunk.ProtoReflect.Descriptor instead. func (*StoredDraftChunk) Descriptor() ([]byte, []int) { - return file_openshell_proto_rawDescGZIP(), []int{175} + return file_openshell_proto_rawDescGZIP(), []int{177} } func (x *StoredDraftChunk) GetId() string { @@ -12996,7 +13154,7 @@ type CreateWorkspaceRequest struct { func (x *CreateWorkspaceRequest) Reset() { *x = CreateWorkspaceRequest{} - mi := &file_openshell_proto_msgTypes[176] + mi := &file_openshell_proto_msgTypes[178] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -13008,7 +13166,7 @@ func (x *CreateWorkspaceRequest) String() string { func (*CreateWorkspaceRequest) ProtoMessage() {} func (x *CreateWorkspaceRequest) ProtoReflect() protoreflect.Message { - mi := &file_openshell_proto_msgTypes[176] + mi := &file_openshell_proto_msgTypes[178] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -13021,7 +13179,7 @@ func (x *CreateWorkspaceRequest) ProtoReflect() protoreflect.Message { // Deprecated: Use CreateWorkspaceRequest.ProtoReflect.Descriptor instead. func (*CreateWorkspaceRequest) Descriptor() ([]byte, []int) { - return file_openshell_proto_rawDescGZIP(), []int{176} + return file_openshell_proto_rawDescGZIP(), []int{178} } func (x *CreateWorkspaceRequest) GetName() string { @@ -13048,7 +13206,7 @@ type CreateWorkspaceResponse struct { func (x *CreateWorkspaceResponse) Reset() { *x = CreateWorkspaceResponse{} - mi := &file_openshell_proto_msgTypes[177] + mi := &file_openshell_proto_msgTypes[179] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -13060,7 +13218,7 @@ func (x *CreateWorkspaceResponse) String() string { func (*CreateWorkspaceResponse) ProtoMessage() {} func (x *CreateWorkspaceResponse) ProtoReflect() protoreflect.Message { - mi := &file_openshell_proto_msgTypes[177] + mi := &file_openshell_proto_msgTypes[179] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -13073,7 +13231,7 @@ func (x *CreateWorkspaceResponse) ProtoReflect() protoreflect.Message { // Deprecated: Use CreateWorkspaceResponse.ProtoReflect.Descriptor instead. func (*CreateWorkspaceResponse) Descriptor() ([]byte, []int) { - return file_openshell_proto_rawDescGZIP(), []int{177} + return file_openshell_proto_rawDescGZIP(), []int{179} } func (x *CreateWorkspaceResponse) GetWorkspace() *datamodelv1.Workspace { @@ -13094,7 +13252,7 @@ type GetWorkspaceRequest struct { func (x *GetWorkspaceRequest) Reset() { *x = GetWorkspaceRequest{} - mi := &file_openshell_proto_msgTypes[178] + mi := &file_openshell_proto_msgTypes[180] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -13106,7 +13264,7 @@ func (x *GetWorkspaceRequest) String() string { func (*GetWorkspaceRequest) ProtoMessage() {} func (x *GetWorkspaceRequest) ProtoReflect() protoreflect.Message { - mi := &file_openshell_proto_msgTypes[178] + mi := &file_openshell_proto_msgTypes[180] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -13119,7 +13277,7 @@ func (x *GetWorkspaceRequest) ProtoReflect() protoreflect.Message { // Deprecated: Use GetWorkspaceRequest.ProtoReflect.Descriptor instead. func (*GetWorkspaceRequest) Descriptor() ([]byte, []int) { - return file_openshell_proto_rawDescGZIP(), []int{178} + return file_openshell_proto_rawDescGZIP(), []int{180} } func (x *GetWorkspaceRequest) GetName() string { @@ -13139,7 +13297,7 @@ type GetWorkspaceResponse struct { func (x *GetWorkspaceResponse) Reset() { *x = GetWorkspaceResponse{} - mi := &file_openshell_proto_msgTypes[179] + mi := &file_openshell_proto_msgTypes[181] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -13151,7 +13309,7 @@ func (x *GetWorkspaceResponse) String() string { func (*GetWorkspaceResponse) ProtoMessage() {} func (x *GetWorkspaceResponse) ProtoReflect() protoreflect.Message { - mi := &file_openshell_proto_msgTypes[179] + mi := &file_openshell_proto_msgTypes[181] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -13164,7 +13322,7 @@ func (x *GetWorkspaceResponse) ProtoReflect() protoreflect.Message { // Deprecated: Use GetWorkspaceResponse.ProtoReflect.Descriptor instead. func (*GetWorkspaceResponse) Descriptor() ([]byte, []int) { - return file_openshell_proto_rawDescGZIP(), []int{179} + return file_openshell_proto_rawDescGZIP(), []int{181} } func (x *GetWorkspaceResponse) GetWorkspace() *datamodelv1.Workspace { @@ -13187,7 +13345,7 @@ type ListWorkspacesRequest struct { func (x *ListWorkspacesRequest) Reset() { *x = ListWorkspacesRequest{} - mi := &file_openshell_proto_msgTypes[180] + mi := &file_openshell_proto_msgTypes[182] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -13199,7 +13357,7 @@ func (x *ListWorkspacesRequest) String() string { func (*ListWorkspacesRequest) ProtoMessage() {} func (x *ListWorkspacesRequest) ProtoReflect() protoreflect.Message { - mi := &file_openshell_proto_msgTypes[180] + mi := &file_openshell_proto_msgTypes[182] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -13212,7 +13370,7 @@ func (x *ListWorkspacesRequest) ProtoReflect() protoreflect.Message { // Deprecated: Use ListWorkspacesRequest.ProtoReflect.Descriptor instead. func (*ListWorkspacesRequest) Descriptor() ([]byte, []int) { - return file_openshell_proto_rawDescGZIP(), []int{180} + return file_openshell_proto_rawDescGZIP(), []int{182} } func (x *ListWorkspacesRequest) GetLimit() uint32 { @@ -13246,7 +13404,7 @@ type ListWorkspacesResponse struct { func (x *ListWorkspacesResponse) Reset() { *x = ListWorkspacesResponse{} - mi := &file_openshell_proto_msgTypes[181] + mi := &file_openshell_proto_msgTypes[183] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -13258,7 +13416,7 @@ func (x *ListWorkspacesResponse) String() string { func (*ListWorkspacesResponse) ProtoMessage() {} func (x *ListWorkspacesResponse) ProtoReflect() protoreflect.Message { - mi := &file_openshell_proto_msgTypes[181] + mi := &file_openshell_proto_msgTypes[183] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -13271,7 +13429,7 @@ func (x *ListWorkspacesResponse) ProtoReflect() protoreflect.Message { // Deprecated: Use ListWorkspacesResponse.ProtoReflect.Descriptor instead. func (*ListWorkspacesResponse) Descriptor() ([]byte, []int) { - return file_openshell_proto_rawDescGZIP(), []int{181} + return file_openshell_proto_rawDescGZIP(), []int{183} } func (x *ListWorkspacesResponse) GetWorkspaces() []*datamodelv1.Workspace { @@ -13292,7 +13450,7 @@ type DeleteWorkspaceRequest struct { func (x *DeleteWorkspaceRequest) Reset() { *x = DeleteWorkspaceRequest{} - mi := &file_openshell_proto_msgTypes[182] + mi := &file_openshell_proto_msgTypes[184] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -13304,7 +13462,7 @@ func (x *DeleteWorkspaceRequest) String() string { func (*DeleteWorkspaceRequest) ProtoMessage() {} func (x *DeleteWorkspaceRequest) ProtoReflect() protoreflect.Message { - mi := &file_openshell_proto_msgTypes[182] + mi := &file_openshell_proto_msgTypes[184] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -13317,7 +13475,7 @@ func (x *DeleteWorkspaceRequest) ProtoReflect() protoreflect.Message { // Deprecated: Use DeleteWorkspaceRequest.ProtoReflect.Descriptor instead. func (*DeleteWorkspaceRequest) Descriptor() ([]byte, []int) { - return file_openshell_proto_rawDescGZIP(), []int{182} + return file_openshell_proto_rawDescGZIP(), []int{184} } func (x *DeleteWorkspaceRequest) GetName() string { @@ -13337,7 +13495,7 @@ type DeleteWorkspaceResponse struct { func (x *DeleteWorkspaceResponse) Reset() { *x = DeleteWorkspaceResponse{} - mi := &file_openshell_proto_msgTypes[183] + mi := &file_openshell_proto_msgTypes[185] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -13349,7 +13507,7 @@ func (x *DeleteWorkspaceResponse) String() string { func (*DeleteWorkspaceResponse) ProtoMessage() {} func (x *DeleteWorkspaceResponse) ProtoReflect() protoreflect.Message { - mi := &file_openshell_proto_msgTypes[183] + mi := &file_openshell_proto_msgTypes[185] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -13362,7 +13520,7 @@ func (x *DeleteWorkspaceResponse) ProtoReflect() protoreflect.Message { // Deprecated: Use DeleteWorkspaceResponse.ProtoReflect.Descriptor instead. func (*DeleteWorkspaceResponse) Descriptor() ([]byte, []int) { - return file_openshell_proto_rawDescGZIP(), []int{183} + return file_openshell_proto_rawDescGZIP(), []int{185} } func (x *DeleteWorkspaceResponse) GetDeleted() bool { @@ -13386,7 +13544,7 @@ type WorkspaceMember struct { func (x *WorkspaceMember) Reset() { *x = WorkspaceMember{} - mi := &file_openshell_proto_msgTypes[184] + mi := &file_openshell_proto_msgTypes[186] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -13398,7 +13556,7 @@ func (x *WorkspaceMember) String() string { func (*WorkspaceMember) ProtoMessage() {} func (x *WorkspaceMember) ProtoReflect() protoreflect.Message { - mi := &file_openshell_proto_msgTypes[184] + mi := &file_openshell_proto_msgTypes[186] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -13411,7 +13569,7 @@ func (x *WorkspaceMember) ProtoReflect() protoreflect.Message { // Deprecated: Use WorkspaceMember.ProtoReflect.Descriptor instead. func (*WorkspaceMember) Descriptor() ([]byte, []int) { - return file_openshell_proto_rawDescGZIP(), []int{184} + return file_openshell_proto_rawDescGZIP(), []int{186} } func (x *WorkspaceMember) GetMetadata() *datamodelv1.ObjectMeta { @@ -13450,7 +13608,7 @@ type AddWorkspaceMemberRequest struct { func (x *AddWorkspaceMemberRequest) Reset() { *x = AddWorkspaceMemberRequest{} - mi := &file_openshell_proto_msgTypes[185] + mi := &file_openshell_proto_msgTypes[187] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -13462,7 +13620,7 @@ func (x *AddWorkspaceMemberRequest) String() string { func (*AddWorkspaceMemberRequest) ProtoMessage() {} func (x *AddWorkspaceMemberRequest) ProtoReflect() protoreflect.Message { - mi := &file_openshell_proto_msgTypes[185] + mi := &file_openshell_proto_msgTypes[187] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -13475,7 +13633,7 @@ func (x *AddWorkspaceMemberRequest) ProtoReflect() protoreflect.Message { // Deprecated: Use AddWorkspaceMemberRequest.ProtoReflect.Descriptor instead. func (*AddWorkspaceMemberRequest) Descriptor() ([]byte, []int) { - return file_openshell_proto_rawDescGZIP(), []int{185} + return file_openshell_proto_rawDescGZIP(), []int{187} } func (x *AddWorkspaceMemberRequest) GetWorkspace() string { @@ -13509,7 +13667,7 @@ type AddWorkspaceMemberResponse struct { func (x *AddWorkspaceMemberResponse) Reset() { *x = AddWorkspaceMemberResponse{} - mi := &file_openshell_proto_msgTypes[186] + mi := &file_openshell_proto_msgTypes[188] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -13521,7 +13679,7 @@ func (x *AddWorkspaceMemberResponse) String() string { func (*AddWorkspaceMemberResponse) ProtoMessage() {} func (x *AddWorkspaceMemberResponse) ProtoReflect() protoreflect.Message { - mi := &file_openshell_proto_msgTypes[186] + mi := &file_openshell_proto_msgTypes[188] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -13534,7 +13692,7 @@ func (x *AddWorkspaceMemberResponse) ProtoReflect() protoreflect.Message { // Deprecated: Use AddWorkspaceMemberResponse.ProtoReflect.Descriptor instead. func (*AddWorkspaceMemberResponse) Descriptor() ([]byte, []int) { - return file_openshell_proto_rawDescGZIP(), []int{186} + return file_openshell_proto_rawDescGZIP(), []int{188} } func (x *AddWorkspaceMemberResponse) GetMember() *WorkspaceMember { @@ -13557,7 +13715,7 @@ type RemoveWorkspaceMemberRequest struct { func (x *RemoveWorkspaceMemberRequest) Reset() { *x = RemoveWorkspaceMemberRequest{} - mi := &file_openshell_proto_msgTypes[187] + mi := &file_openshell_proto_msgTypes[189] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -13569,7 +13727,7 @@ func (x *RemoveWorkspaceMemberRequest) String() string { func (*RemoveWorkspaceMemberRequest) ProtoMessage() {} func (x *RemoveWorkspaceMemberRequest) ProtoReflect() protoreflect.Message { - mi := &file_openshell_proto_msgTypes[187] + mi := &file_openshell_proto_msgTypes[189] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -13582,7 +13740,7 @@ func (x *RemoveWorkspaceMemberRequest) ProtoReflect() protoreflect.Message { // Deprecated: Use RemoveWorkspaceMemberRequest.ProtoReflect.Descriptor instead. func (*RemoveWorkspaceMemberRequest) Descriptor() ([]byte, []int) { - return file_openshell_proto_rawDescGZIP(), []int{187} + return file_openshell_proto_rawDescGZIP(), []int{189} } func (x *RemoveWorkspaceMemberRequest) GetWorkspace() string { @@ -13609,7 +13767,7 @@ type RemoveWorkspaceMemberResponse struct { func (x *RemoveWorkspaceMemberResponse) Reset() { *x = RemoveWorkspaceMemberResponse{} - mi := &file_openshell_proto_msgTypes[188] + mi := &file_openshell_proto_msgTypes[190] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -13621,7 +13779,7 @@ func (x *RemoveWorkspaceMemberResponse) String() string { func (*RemoveWorkspaceMemberResponse) ProtoMessage() {} func (x *RemoveWorkspaceMemberResponse) ProtoReflect() protoreflect.Message { - mi := &file_openshell_proto_msgTypes[188] + mi := &file_openshell_proto_msgTypes[190] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -13634,7 +13792,7 @@ func (x *RemoveWorkspaceMemberResponse) ProtoReflect() protoreflect.Message { // Deprecated: Use RemoveWorkspaceMemberResponse.ProtoReflect.Descriptor instead. func (*RemoveWorkspaceMemberResponse) Descriptor() ([]byte, []int) { - return file_openshell_proto_rawDescGZIP(), []int{188} + return file_openshell_proto_rawDescGZIP(), []int{190} } func (x *RemoveWorkspaceMemberResponse) GetRemoved() bool { @@ -13657,7 +13815,7 @@ type ListWorkspaceMembersRequest struct { func (x *ListWorkspaceMembersRequest) Reset() { *x = ListWorkspaceMembersRequest{} - mi := &file_openshell_proto_msgTypes[189] + mi := &file_openshell_proto_msgTypes[191] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -13669,7 +13827,7 @@ func (x *ListWorkspaceMembersRequest) String() string { func (*ListWorkspaceMembersRequest) ProtoMessage() {} func (x *ListWorkspaceMembersRequest) ProtoReflect() protoreflect.Message { - mi := &file_openshell_proto_msgTypes[189] + mi := &file_openshell_proto_msgTypes[191] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -13682,7 +13840,7 @@ func (x *ListWorkspaceMembersRequest) ProtoReflect() protoreflect.Message { // Deprecated: Use ListWorkspaceMembersRequest.ProtoReflect.Descriptor instead. func (*ListWorkspaceMembersRequest) Descriptor() ([]byte, []int) { - return file_openshell_proto_rawDescGZIP(), []int{189} + return file_openshell_proto_rawDescGZIP(), []int{191} } func (x *ListWorkspaceMembersRequest) GetWorkspace() string { @@ -13716,7 +13874,7 @@ type ListWorkspaceMembersResponse struct { func (x *ListWorkspaceMembersResponse) Reset() { *x = ListWorkspaceMembersResponse{} - mi := &file_openshell_proto_msgTypes[190] + mi := &file_openshell_proto_msgTypes[192] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -13728,7 +13886,7 @@ func (x *ListWorkspaceMembersResponse) String() string { func (*ListWorkspaceMembersResponse) ProtoMessage() {} func (x *ListWorkspaceMembersResponse) ProtoReflect() protoreflect.Message { - mi := &file_openshell_proto_msgTypes[190] + mi := &file_openshell_proto_msgTypes[192] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -13741,7 +13899,7 @@ func (x *ListWorkspaceMembersResponse) ProtoReflect() protoreflect.Message { // Deprecated: Use ListWorkspaceMembersResponse.ProtoReflect.Descriptor instead. func (*ListWorkspaceMembersResponse) Descriptor() ([]byte, []int) { - return file_openshell_proto_rawDescGZIP(), []int{190} + return file_openshell_proto_rawDescGZIP(), []int{192} } func (x *ListWorkspaceMembersResponse) GetMembers() []*WorkspaceMember { @@ -13769,7 +13927,7 @@ type ExtensionServiceCredential struct { func (x *ExtensionServiceCredential) Reset() { *x = ExtensionServiceCredential{} - mi := &file_openshell_proto_msgTypes[191] + mi := &file_openshell_proto_msgTypes[193] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -13781,7 +13939,7 @@ func (x *ExtensionServiceCredential) String() string { func (*ExtensionServiceCredential) ProtoMessage() {} func (x *ExtensionServiceCredential) ProtoReflect() protoreflect.Message { - mi := &file_openshell_proto_msgTypes[191] + mi := &file_openshell_proto_msgTypes[193] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -13794,7 +13952,7 @@ func (x *ExtensionServiceCredential) ProtoReflect() protoreflect.Message { // Deprecated: Use ExtensionServiceCredential.ProtoReflect.Descriptor instead. func (*ExtensionServiceCredential) Descriptor() ([]byte, []int) { - return file_openshell_proto_rawDescGZIP(), []int{191} + return file_openshell_proto_rawDescGZIP(), []int{193} } func (x *ExtensionServiceCredential) GetServiceName() string { @@ -14542,12 +14700,13 @@ const file_openshell_proto_rawDesc = "" + "relay_open\x18\x04 \x01(\v2\x17.openshell.v1.RelayOpenH\x00R\trelayOpen\x12;\n" + "\vrelay_close\x18\x05 \x01(\v2\x18.openshell.v1.RelayCloseH\x00R\n" + "relayCloseB\t\n" + - "\apayload\"Q\n" + + "\apayload\"|\n" + "\x0fSupervisorHello\x12\x1d\n" + "\n" + "sandbox_id\x18\x01 \x01(\tR\tsandboxId\x12\x1f\n" + "\vinstance_id\x18\x02 \x01(\tR\n" + - "instanceId\"h\n" + + "instanceId\x12)\n" + + "\x10connection_epoch\x18\x03 \x01(\x04R\x0fconnectionEpoch\"h\n" + "\x0fSessionAccepted\x12\x1d\n" + "\n" + "session_id\x18\x01 \x01(\tR\tsessionId\x126\n" + @@ -14582,6 +14741,16 @@ const file_openshell_proto_rawDesc = "" + "RelayFrame\x12-\n" + "\x04init\x18\x01 \x01(\v2\x17.openshell.v1.RelayInitH\x00R\x04init\x12\x14\n" + "\x04data\x18\x02 \x01(\fH\x00R\x04dataB\t\n" + + "\apayload\"\x98\x01\n" + + "\rPeerRelayInit\x12\x1d\n" + + "\n" + + "sandbox_id\x18\x01 \x01(\tR\tsandboxId\x126\n" + + "\n" + + "relay_open\x18\x02 \x01(\v2\x17.openshell.v1.RelayOpenR\trelayOpen\x120\n" + + "\x14requester_replica_id\x18\x03 \x01(\tR\x12requesterReplicaId\"d\n" + + "\x0ePeerRelayFrame\x121\n" + + "\x04init\x18\x01 \x01(\v2\x1b.openshell.v1.PeerRelayInitH\x00R\x04init\x12\x14\n" + + "\x04data\x18\x02 \x01(\fH\x00R\x04dataB\t\n" + "\apayload\"`\n" + "\x0fRelayOpenResult\x12\x1d\n" + "\n" + @@ -14930,7 +15099,7 @@ const file_openshell_proto_rawDesc = "" + "\rWorkspaceRole\x12\x1e\n" + "\x1aWORKSPACE_ROLE_UNSPECIFIED\x10\x00\x12\x17\n" + "\x13WORKSPACE_ROLE_USER\x10\x01\x12\x18\n" + - "\x14WORKSPACE_ROLE_ADMIN\x10\x022\xacF\n" + + "\x14WORKSPACE_ROLE_ADMIN\x10\x022\x85G\n" + "\tOpenShell\x12Z\n" + "\x06Health\x12\x1b.openshell.v1.HealthRequest\x1a\x1c.openshell.v1.HealthResponse\"\x15\x82\xb5\x18\x11\n" + "\x0funauthenticated\x12i\n" + @@ -15032,7 +15201,10 @@ const file_openshell_proto_rawDesc = "" + "\x15ReportMainProcessExit\x12*.openshell.v1.ReportMainProcessExitRequest\x1a+.openshell.v1.ReportMainProcessExitResponse\"\r\x82\xb5\x18\t\n" + "\asandbox\x12T\n" + "\vRelayStream\x12\x18.openshell.v1.RelayFrame\x1a\x18.openshell.v1.RelayFrame\"\r\x82\xb5\x18\t\n" + - "\asandbox(\x010\x01\x12w\n" + + "\asandbox(\x010\x01\x12W\n" + + "\tPeerRelay\x12\x1c.openshell.v1.PeerRelayFrame\x1a\x1c.openshell.v1.PeerRelayFrame\"\n" + + "\x82\xb5\x18\x06\n" + + "\x04peer(\x010\x01\x12w\n" + "\fWatchSandbox\x12!.openshell.v1.WatchSandboxRequest\x1a .openshell.v1.SandboxStreamEvent\" \x82\xb5\x18\x1c\n" + "\x06bearer\x12\x04user\"\fsandbox:read0\x01\x12|\n" + "\x14SubmitPolicyAnalysis\x12).openshell.v1.SubmitPolicyAnalysisRequest\x1a*.openshell.v1.SubmitPolicyAnalysisResponse\"\r\x82\xb5\x18\t\n" + @@ -15085,7 +15257,7 @@ func file_openshell_proto_rawDescGZIP() []byte { } var file_openshell_proto_enumTypes = make([]protoimpl.EnumInfo, 7) -var file_openshell_proto_msgTypes = make([]protoimpl.MessageInfo, 217) +var file_openshell_proto_msgTypes = make([]protoimpl.MessageInfo, 219) var file_openshell_proto_goTypes = []any{ (SandboxPhase)(0), // 0: openshell.v1.SandboxPhase (ProviderCredentialTokenGrantType)(0), // 1: openshell.v1.ProviderCredentialTokenGrantType @@ -15238,130 +15410,132 @@ var file_openshell_proto_goTypes = []any{ (*TcpRelayTarget)(nil), // 148: openshell.v1.TcpRelayTarget (*RelayInit)(nil), // 149: openshell.v1.RelayInit (*RelayFrame)(nil), // 150: openshell.v1.RelayFrame - (*RelayOpenResult)(nil), // 151: openshell.v1.RelayOpenResult - (*RelayClose)(nil), // 152: openshell.v1.RelayClose - (*L7RequestSample)(nil), // 153: openshell.v1.L7RequestSample - (*DenialSummary)(nil), // 154: openshell.v1.DenialSummary - (*DenialGroupCount)(nil), // 155: openshell.v1.DenialGroupCount - (*NetworkActivitySummary)(nil), // 156: openshell.v1.NetworkActivitySummary - (*PolicyChunk)(nil), // 157: openshell.v1.PolicyChunk - (*DraftPolicyUpdate)(nil), // 158: openshell.v1.DraftPolicyUpdate - (*SubmitPolicyAnalysisRequest)(nil), // 159: openshell.v1.SubmitPolicyAnalysisRequest - (*SubmitPolicyAnalysisResponse)(nil), // 160: openshell.v1.SubmitPolicyAnalysisResponse - (*GetDraftPolicyRequest)(nil), // 161: openshell.v1.GetDraftPolicyRequest - (*GetDraftPolicyResponse)(nil), // 162: openshell.v1.GetDraftPolicyResponse - (*ApproveDraftChunkRequest)(nil), // 163: openshell.v1.ApproveDraftChunkRequest - (*ApproveDraftChunkResponse)(nil), // 164: openshell.v1.ApproveDraftChunkResponse - (*RejectDraftChunkRequest)(nil), // 165: openshell.v1.RejectDraftChunkRequest - (*RejectDraftChunkResponse)(nil), // 166: openshell.v1.RejectDraftChunkResponse - (*DraftChunkApproval)(nil), // 167: openshell.v1.DraftChunkApproval - (*ApproveAllDraftChunksRequest)(nil), // 168: openshell.v1.ApproveAllDraftChunksRequest - (*ApproveAllDraftChunksResponse)(nil), // 169: openshell.v1.ApproveAllDraftChunksResponse - (*EditDraftChunkRequest)(nil), // 170: openshell.v1.EditDraftChunkRequest - (*EditDraftChunkResponse)(nil), // 171: openshell.v1.EditDraftChunkResponse - (*UndoDraftChunkRequest)(nil), // 172: openshell.v1.UndoDraftChunkRequest - (*UndoDraftChunkResponse)(nil), // 173: openshell.v1.UndoDraftChunkResponse - (*ClearDraftChunksRequest)(nil), // 174: openshell.v1.ClearDraftChunksRequest - (*ClearDraftChunksResponse)(nil), // 175: openshell.v1.ClearDraftChunksResponse - (*GetDraftHistoryRequest)(nil), // 176: openshell.v1.GetDraftHistoryRequest - (*DraftHistoryEntry)(nil), // 177: openshell.v1.DraftHistoryEntry - (*GetDraftHistoryResponse)(nil), // 178: openshell.v1.GetDraftHistoryResponse - (*PolicyRevisionPayload)(nil), // 179: openshell.v1.PolicyRevisionPayload - (*DraftChunkPayload)(nil), // 180: openshell.v1.DraftChunkPayload - (*StoredPolicyRevision)(nil), // 181: openshell.v1.StoredPolicyRevision - (*StoredDraftChunk)(nil), // 182: openshell.v1.StoredDraftChunk - (*CreateWorkspaceRequest)(nil), // 183: openshell.v1.CreateWorkspaceRequest - (*CreateWorkspaceResponse)(nil), // 184: openshell.v1.CreateWorkspaceResponse - (*GetWorkspaceRequest)(nil), // 185: openshell.v1.GetWorkspaceRequest - (*GetWorkspaceResponse)(nil), // 186: openshell.v1.GetWorkspaceResponse - (*ListWorkspacesRequest)(nil), // 187: openshell.v1.ListWorkspacesRequest - (*ListWorkspacesResponse)(nil), // 188: openshell.v1.ListWorkspacesResponse - (*DeleteWorkspaceRequest)(nil), // 189: openshell.v1.DeleteWorkspaceRequest - (*DeleteWorkspaceResponse)(nil), // 190: openshell.v1.DeleteWorkspaceResponse - (*WorkspaceMember)(nil), // 191: openshell.v1.WorkspaceMember - (*AddWorkspaceMemberRequest)(nil), // 192: openshell.v1.AddWorkspaceMemberRequest - (*AddWorkspaceMemberResponse)(nil), // 193: openshell.v1.AddWorkspaceMemberResponse - (*RemoveWorkspaceMemberRequest)(nil), // 194: openshell.v1.RemoveWorkspaceMemberRequest - (*RemoveWorkspaceMemberResponse)(nil), // 195: openshell.v1.RemoveWorkspaceMemberResponse - (*ListWorkspaceMembersRequest)(nil), // 196: openshell.v1.ListWorkspaceMembersRequest - (*ListWorkspaceMembersResponse)(nil), // 197: openshell.v1.ListWorkspaceMembersResponse - (*ExtensionServiceCredential)(nil), // 198: openshell.v1.ExtensionServiceCredential - nil, // 199: openshell.v1.SandboxSpec.EnvironmentEntry - nil, // 200: openshell.v1.SandboxTemplate.LabelsEntry - nil, // 201: openshell.v1.SandboxTemplate.AnnotationsEntry - nil, // 202: openshell.v1.SandboxTemplate.EnvironmentEntry - nil, // 203: openshell.v1.PlatformEvent.MetadataEntry - nil, // 204: openshell.v1.CreateSandboxRequest.LabelsEntry - nil, // 205: openshell.v1.CreateSandboxRequest.AnnotationsEntry - nil, // 206: openshell.v1.ExecSandboxRequest.EnvironmentEntry - nil, // 207: openshell.v1.SandboxLogLine.FieldsEntry - nil, // 208: openshell.v1.UpdateProviderRequest.CredentialExpiresAtMsEntry - nil, // 209: openshell.v1.StoredProviderCredentialRefreshState.MaterialEntry - nil, // 210: openshell.v1.StoredProviderCredentialRefreshState.AdditionalOutputKeysEntry - nil, // 211: openshell.v1.StoredProviderCredentialRefreshState.SecretMaterialHandlesEntry - nil, // 212: openshell.v1.ConfigureProviderRefreshRequest.MaterialEntry - nil, // 213: openshell.v1.ProviderProfile.AnnotationsEntry - nil, // 214: openshell.v1.GetSandboxProviderEnvironmentResponse.EnvironmentEntry - nil, // 215: openshell.v1.GetSandboxProviderEnvironmentResponse.CredentialExpiresAtMsEntry - nil, // 216: openshell.v1.GetSandboxProviderEnvironmentResponse.DynamicCredentialsEntry - nil, // 217: openshell.v1.GetSandboxProviderEnvironmentResponse.StaticCredentialBindingsEntry - nil, // 218: openshell.v1.UpdateConfigRequest.AnnotationsEntry - nil, // 219: openshell.v1.UpdateConfigResponse.AnnotationsEntry - nil, // 220: openshell.v1.SandboxPolicyRevision.ProvenanceEntry - nil, // 221: openshell.v1.PolicyRevisionPayload.ProvenanceEntry - nil, // 222: openshell.v1.StoredPolicyRevision.ProvenanceEntry - nil, // 223: openshell.v1.CreateWorkspaceRequest.LabelsEntry - (*datamodelv1.ObjectMeta)(nil), // 224: openshell.datamodel.v1.ObjectMeta - (*sandboxv1.SandboxPolicy)(nil), // 225: openshell.sandbox.v1.SandboxPolicy - (*structpb.Struct)(nil), // 226: google.protobuf.Struct - (*datamodelv1.Provider)(nil), // 227: openshell.datamodel.v1.Provider - (*datamodelv1.CredentialHandle)(nil), // 228: openshell.datamodel.v1.CredentialHandle - (*sandboxv1.NetworkEndpoint)(nil), // 229: openshell.sandbox.v1.NetworkEndpoint - (*sandboxv1.NetworkBinary)(nil), // 230: openshell.sandbox.v1.NetworkBinary - (*sandboxv1.SettingValue)(nil), // 231: openshell.sandbox.v1.SettingValue - (*sandboxv1.NetworkPolicyRule)(nil), // 232: openshell.sandbox.v1.NetworkPolicyRule - (*sandboxv1.L7DenyRule)(nil), // 233: openshell.sandbox.v1.L7DenyRule - (*sandboxv1.L7Rule)(nil), // 234: openshell.sandbox.v1.L7Rule - (*datamodelv1.Workspace)(nil), // 235: openshell.datamodel.v1.Workspace - (*sandboxv1.GetSandboxConfigRequest)(nil), // 236: openshell.sandbox.v1.GetSandboxConfigRequest - (*sandboxv1.GetGatewayConfigRequest)(nil), // 237: openshell.sandbox.v1.GetGatewayConfigRequest - (*sandboxv1.GetSandboxConfigResponse)(nil), // 238: openshell.sandbox.v1.GetSandboxConfigResponse - (*sandboxv1.GetGatewayConfigResponse)(nil), // 239: openshell.sandbox.v1.GetGatewayConfigResponse + (*PeerRelayInit)(nil), // 151: openshell.v1.PeerRelayInit + (*PeerRelayFrame)(nil), // 152: openshell.v1.PeerRelayFrame + (*RelayOpenResult)(nil), // 153: openshell.v1.RelayOpenResult + (*RelayClose)(nil), // 154: openshell.v1.RelayClose + (*L7RequestSample)(nil), // 155: openshell.v1.L7RequestSample + (*DenialSummary)(nil), // 156: openshell.v1.DenialSummary + (*DenialGroupCount)(nil), // 157: openshell.v1.DenialGroupCount + (*NetworkActivitySummary)(nil), // 158: openshell.v1.NetworkActivitySummary + (*PolicyChunk)(nil), // 159: openshell.v1.PolicyChunk + (*DraftPolicyUpdate)(nil), // 160: openshell.v1.DraftPolicyUpdate + (*SubmitPolicyAnalysisRequest)(nil), // 161: openshell.v1.SubmitPolicyAnalysisRequest + (*SubmitPolicyAnalysisResponse)(nil), // 162: openshell.v1.SubmitPolicyAnalysisResponse + (*GetDraftPolicyRequest)(nil), // 163: openshell.v1.GetDraftPolicyRequest + (*GetDraftPolicyResponse)(nil), // 164: openshell.v1.GetDraftPolicyResponse + (*ApproveDraftChunkRequest)(nil), // 165: openshell.v1.ApproveDraftChunkRequest + (*ApproveDraftChunkResponse)(nil), // 166: openshell.v1.ApproveDraftChunkResponse + (*RejectDraftChunkRequest)(nil), // 167: openshell.v1.RejectDraftChunkRequest + (*RejectDraftChunkResponse)(nil), // 168: openshell.v1.RejectDraftChunkResponse + (*DraftChunkApproval)(nil), // 169: openshell.v1.DraftChunkApproval + (*ApproveAllDraftChunksRequest)(nil), // 170: openshell.v1.ApproveAllDraftChunksRequest + (*ApproveAllDraftChunksResponse)(nil), // 171: openshell.v1.ApproveAllDraftChunksResponse + (*EditDraftChunkRequest)(nil), // 172: openshell.v1.EditDraftChunkRequest + (*EditDraftChunkResponse)(nil), // 173: openshell.v1.EditDraftChunkResponse + (*UndoDraftChunkRequest)(nil), // 174: openshell.v1.UndoDraftChunkRequest + (*UndoDraftChunkResponse)(nil), // 175: openshell.v1.UndoDraftChunkResponse + (*ClearDraftChunksRequest)(nil), // 176: openshell.v1.ClearDraftChunksRequest + (*ClearDraftChunksResponse)(nil), // 177: openshell.v1.ClearDraftChunksResponse + (*GetDraftHistoryRequest)(nil), // 178: openshell.v1.GetDraftHistoryRequest + (*DraftHistoryEntry)(nil), // 179: openshell.v1.DraftHistoryEntry + (*GetDraftHistoryResponse)(nil), // 180: openshell.v1.GetDraftHistoryResponse + (*PolicyRevisionPayload)(nil), // 181: openshell.v1.PolicyRevisionPayload + (*DraftChunkPayload)(nil), // 182: openshell.v1.DraftChunkPayload + (*StoredPolicyRevision)(nil), // 183: openshell.v1.StoredPolicyRevision + (*StoredDraftChunk)(nil), // 184: openshell.v1.StoredDraftChunk + (*CreateWorkspaceRequest)(nil), // 185: openshell.v1.CreateWorkspaceRequest + (*CreateWorkspaceResponse)(nil), // 186: openshell.v1.CreateWorkspaceResponse + (*GetWorkspaceRequest)(nil), // 187: openshell.v1.GetWorkspaceRequest + (*GetWorkspaceResponse)(nil), // 188: openshell.v1.GetWorkspaceResponse + (*ListWorkspacesRequest)(nil), // 189: openshell.v1.ListWorkspacesRequest + (*ListWorkspacesResponse)(nil), // 190: openshell.v1.ListWorkspacesResponse + (*DeleteWorkspaceRequest)(nil), // 191: openshell.v1.DeleteWorkspaceRequest + (*DeleteWorkspaceResponse)(nil), // 192: openshell.v1.DeleteWorkspaceResponse + (*WorkspaceMember)(nil), // 193: openshell.v1.WorkspaceMember + (*AddWorkspaceMemberRequest)(nil), // 194: openshell.v1.AddWorkspaceMemberRequest + (*AddWorkspaceMemberResponse)(nil), // 195: openshell.v1.AddWorkspaceMemberResponse + (*RemoveWorkspaceMemberRequest)(nil), // 196: openshell.v1.RemoveWorkspaceMemberRequest + (*RemoveWorkspaceMemberResponse)(nil), // 197: openshell.v1.RemoveWorkspaceMemberResponse + (*ListWorkspaceMembersRequest)(nil), // 198: openshell.v1.ListWorkspaceMembersRequest + (*ListWorkspaceMembersResponse)(nil), // 199: openshell.v1.ListWorkspaceMembersResponse + (*ExtensionServiceCredential)(nil), // 200: openshell.v1.ExtensionServiceCredential + nil, // 201: openshell.v1.SandboxSpec.EnvironmentEntry + nil, // 202: openshell.v1.SandboxTemplate.LabelsEntry + nil, // 203: openshell.v1.SandboxTemplate.AnnotationsEntry + nil, // 204: openshell.v1.SandboxTemplate.EnvironmentEntry + nil, // 205: openshell.v1.PlatformEvent.MetadataEntry + nil, // 206: openshell.v1.CreateSandboxRequest.LabelsEntry + nil, // 207: openshell.v1.CreateSandboxRequest.AnnotationsEntry + nil, // 208: openshell.v1.ExecSandboxRequest.EnvironmentEntry + nil, // 209: openshell.v1.SandboxLogLine.FieldsEntry + nil, // 210: openshell.v1.UpdateProviderRequest.CredentialExpiresAtMsEntry + nil, // 211: openshell.v1.StoredProviderCredentialRefreshState.MaterialEntry + nil, // 212: openshell.v1.StoredProviderCredentialRefreshState.AdditionalOutputKeysEntry + nil, // 213: openshell.v1.StoredProviderCredentialRefreshState.SecretMaterialHandlesEntry + nil, // 214: openshell.v1.ConfigureProviderRefreshRequest.MaterialEntry + nil, // 215: openshell.v1.ProviderProfile.AnnotationsEntry + nil, // 216: openshell.v1.GetSandboxProviderEnvironmentResponse.EnvironmentEntry + nil, // 217: openshell.v1.GetSandboxProviderEnvironmentResponse.CredentialExpiresAtMsEntry + nil, // 218: openshell.v1.GetSandboxProviderEnvironmentResponse.DynamicCredentialsEntry + nil, // 219: openshell.v1.GetSandboxProviderEnvironmentResponse.StaticCredentialBindingsEntry + nil, // 220: openshell.v1.UpdateConfigRequest.AnnotationsEntry + nil, // 221: openshell.v1.UpdateConfigResponse.AnnotationsEntry + nil, // 222: openshell.v1.SandboxPolicyRevision.ProvenanceEntry + nil, // 223: openshell.v1.PolicyRevisionPayload.ProvenanceEntry + nil, // 224: openshell.v1.StoredPolicyRevision.ProvenanceEntry + nil, // 225: openshell.v1.CreateWorkspaceRequest.LabelsEntry + (*datamodelv1.ObjectMeta)(nil), // 226: openshell.datamodel.v1.ObjectMeta + (*sandboxv1.SandboxPolicy)(nil), // 227: openshell.sandbox.v1.SandboxPolicy + (*structpb.Struct)(nil), // 228: google.protobuf.Struct + (*datamodelv1.Provider)(nil), // 229: openshell.datamodel.v1.Provider + (*datamodelv1.CredentialHandle)(nil), // 230: openshell.datamodel.v1.CredentialHandle + (*sandboxv1.NetworkEndpoint)(nil), // 231: openshell.sandbox.v1.NetworkEndpoint + (*sandboxv1.NetworkBinary)(nil), // 232: openshell.sandbox.v1.NetworkBinary + (*sandboxv1.SettingValue)(nil), // 233: openshell.sandbox.v1.SettingValue + (*sandboxv1.NetworkPolicyRule)(nil), // 234: openshell.sandbox.v1.NetworkPolicyRule + (*sandboxv1.L7DenyRule)(nil), // 235: openshell.sandbox.v1.L7DenyRule + (*sandboxv1.L7Rule)(nil), // 236: openshell.sandbox.v1.L7Rule + (*datamodelv1.Workspace)(nil), // 237: openshell.datamodel.v1.Workspace + (*sandboxv1.GetSandboxConfigRequest)(nil), // 238: openshell.sandbox.v1.GetSandboxConfigRequest + (*sandboxv1.GetGatewayConfigRequest)(nil), // 239: openshell.sandbox.v1.GetGatewayConfigRequest + (*sandboxv1.GetSandboxConfigResponse)(nil), // 240: openshell.sandbox.v1.GetSandboxConfigResponse + (*sandboxv1.GetGatewayConfigResponse)(nil), // 241: openshell.sandbox.v1.GetGatewayConfigResponse } var file_openshell_proto_depIdxs = []int32{ - 198, // 0: openshell.v1.RefreshSandboxTokenResponse.extension_credentials:type_name -> openshell.v1.ExtensionServiceCredential + 200, // 0: openshell.v1.RefreshSandboxTokenResponse.extension_credentials:type_name -> openshell.v1.ExtensionServiceCredential 5, // 1: openshell.v1.HealthResponse.status:type_name -> openshell.v1.ServiceStatus 5, // 2: openshell.v1.GetGatewayInfoResponse.status:type_name -> openshell.v1.ServiceStatus 17, // 3: openshell.v1.GetGatewayInfoResponse.compute_drivers:type_name -> openshell.v1.ComputeDriverInfo 18, // 4: openshell.v1.ComputeDriverInfo.capabilities:type_name -> openshell.v1.ComputeDriverCapabilities - 224, // 5: openshell.v1.Sandbox.metadata:type_name -> openshell.datamodel.v1.ObjectMeta + 226, // 5: openshell.v1.Sandbox.metadata:type_name -> openshell.datamodel.v1.ObjectMeta 20, // 6: openshell.v1.Sandbox.spec:type_name -> openshell.v1.SandboxSpec 24, // 7: openshell.v1.Sandbox.status:type_name -> openshell.v1.SandboxStatus - 199, // 8: openshell.v1.SandboxSpec.environment:type_name -> openshell.v1.SandboxSpec.EnvironmentEntry + 201, // 8: openshell.v1.SandboxSpec.environment:type_name -> openshell.v1.SandboxSpec.EnvironmentEntry 23, // 9: openshell.v1.SandboxSpec.template:type_name -> openshell.v1.SandboxTemplate - 225, // 10: openshell.v1.SandboxSpec.policy:type_name -> openshell.sandbox.v1.SandboxPolicy + 227, // 10: openshell.v1.SandboxSpec.policy:type_name -> openshell.sandbox.v1.SandboxPolicy 21, // 11: openshell.v1.SandboxSpec.resource_requirements:type_name -> openshell.v1.ResourceRequirements 22, // 12: openshell.v1.ResourceRequirements.gpu:type_name -> openshell.v1.GpuResourceRequirements - 200, // 13: openshell.v1.SandboxTemplate.labels:type_name -> openshell.v1.SandboxTemplate.LabelsEntry - 201, // 14: openshell.v1.SandboxTemplate.annotations:type_name -> openshell.v1.SandboxTemplate.AnnotationsEntry - 202, // 15: openshell.v1.SandboxTemplate.environment:type_name -> openshell.v1.SandboxTemplate.EnvironmentEntry - 226, // 16: openshell.v1.SandboxTemplate.resources:type_name -> google.protobuf.Struct - 226, // 17: openshell.v1.SandboxTemplate.driver_config:type_name -> google.protobuf.Struct + 202, // 13: openshell.v1.SandboxTemplate.labels:type_name -> openshell.v1.SandboxTemplate.LabelsEntry + 203, // 14: openshell.v1.SandboxTemplate.annotations:type_name -> openshell.v1.SandboxTemplate.AnnotationsEntry + 204, // 15: openshell.v1.SandboxTemplate.environment:type_name -> openshell.v1.SandboxTemplate.EnvironmentEntry + 228, // 16: openshell.v1.SandboxTemplate.resources:type_name -> google.protobuf.Struct + 228, // 17: openshell.v1.SandboxTemplate.driver_config:type_name -> google.protobuf.Struct 25, // 18: openshell.v1.SandboxStatus.conditions:type_name -> openshell.v1.SandboxCondition 0, // 19: openshell.v1.SandboxStatus.phase:type_name -> openshell.v1.SandboxPhase - 203, // 20: openshell.v1.PlatformEvent.metadata:type_name -> openshell.v1.PlatformEvent.MetadataEntry + 205, // 20: openshell.v1.PlatformEvent.metadata:type_name -> openshell.v1.PlatformEvent.MetadataEntry 20, // 21: openshell.v1.CreateSandboxRequest.spec:type_name -> openshell.v1.SandboxSpec - 204, // 22: openshell.v1.CreateSandboxRequest.labels:type_name -> openshell.v1.CreateSandboxRequest.LabelsEntry - 205, // 23: openshell.v1.CreateSandboxRequest.annotations:type_name -> openshell.v1.CreateSandboxRequest.AnnotationsEntry + 206, // 22: openshell.v1.CreateSandboxRequest.labels:type_name -> openshell.v1.CreateSandboxRequest.LabelsEntry + 207, // 23: openshell.v1.CreateSandboxRequest.annotations:type_name -> openshell.v1.CreateSandboxRequest.AnnotationsEntry 19, // 24: openshell.v1.SandboxResponse.sandbox:type_name -> openshell.v1.Sandbox 19, // 25: openshell.v1.ListSandboxesResponse.sandboxes:type_name -> openshell.v1.Sandbox - 227, // 26: openshell.v1.ListSandboxProvidersResponse.providers:type_name -> openshell.datamodel.v1.Provider + 229, // 26: openshell.v1.ListSandboxProvidersResponse.providers:type_name -> openshell.datamodel.v1.Provider 19, // 27: openshell.v1.AttachSandboxProviderResponse.sandbox:type_name -> openshell.v1.Sandbox 19, // 28: openshell.v1.DetachSandboxProviderResponse.sandbox:type_name -> openshell.v1.Sandbox 51, // 29: openshell.v1.ListServicesResponse.services:type_name -> openshell.v1.ServiceEndpointResponse - 224, // 30: openshell.v1.ServiceEndpoint.metadata:type_name -> openshell.datamodel.v1.ObjectMeta + 226, // 30: openshell.v1.ServiceEndpoint.metadata:type_name -> openshell.datamodel.v1.ObjectMeta 50, // 31: openshell.v1.ServiceEndpointResponse.endpoint:type_name -> openshell.v1.ServiceEndpoint - 206, // 32: openshell.v1.ExecSandboxRequest.environment:type_name -> openshell.v1.ExecSandboxRequest.EnvironmentEntry + 208, // 32: openshell.v1.ExecSandboxRequest.environment:type_name -> openshell.v1.ExecSandboxRequest.EnvironmentEntry 55, // 33: openshell.v1.ExecSandboxEvent.stdout:type_name -> openshell.v1.ExecSandboxStdout 56, // 34: openshell.v1.ExecSandboxEvent.stderr:type_name -> openshell.v1.ExecSandboxStderr 57, // 35: openshell.v1.ExecSandboxEvent.exit:type_name -> openshell.v1.ExecSandboxExit @@ -15370,18 +15544,18 @@ var file_openshell_proto_depIdxs = []int32{ 59, // 38: openshell.v1.TcpForwardFrame.init:type_name -> openshell.v1.TcpForwardInit 54, // 39: openshell.v1.ExecSandboxInput.start:type_name -> openshell.v1.ExecSandboxRequest 62, // 40: openshell.v1.ExecSandboxInput.resize:type_name -> openshell.v1.ExecSandboxWindowResize - 224, // 41: openshell.v1.SshSession.metadata:type_name -> openshell.datamodel.v1.ObjectMeta + 226, // 41: openshell.v1.SshSession.metadata:type_name -> openshell.datamodel.v1.ObjectMeta 19, // 42: openshell.v1.SandboxStreamEvent.sandbox:type_name -> openshell.v1.Sandbox 66, // 43: openshell.v1.SandboxStreamEvent.log:type_name -> openshell.v1.SandboxLogLine 26, // 44: openshell.v1.SandboxStreamEvent.event:type_name -> openshell.v1.PlatformEvent 67, // 45: openshell.v1.SandboxStreamEvent.warning:type_name -> openshell.v1.SandboxStreamWarning - 158, // 46: openshell.v1.SandboxStreamEvent.draft_policy_update:type_name -> openshell.v1.DraftPolicyUpdate - 207, // 47: openshell.v1.SandboxLogLine.fields:type_name -> openshell.v1.SandboxLogLine.FieldsEntry - 227, // 48: openshell.v1.CreateProviderRequest.provider:type_name -> openshell.datamodel.v1.Provider - 227, // 49: openshell.v1.UpdateProviderRequest.provider:type_name -> openshell.datamodel.v1.Provider - 208, // 50: openshell.v1.UpdateProviderRequest.credential_expires_at_ms:type_name -> openshell.v1.UpdateProviderRequest.CredentialExpiresAtMsEntry - 227, // 51: openshell.v1.ProviderResponse.provider:type_name -> openshell.datamodel.v1.Provider - 227, // 52: openshell.v1.ListProvidersResponse.providers:type_name -> openshell.datamodel.v1.Provider + 160, // 46: openshell.v1.SandboxStreamEvent.draft_policy_update:type_name -> openshell.v1.DraftPolicyUpdate + 209, // 47: openshell.v1.SandboxLogLine.fields:type_name -> openshell.v1.SandboxLogLine.FieldsEntry + 229, // 48: openshell.v1.CreateProviderRequest.provider:type_name -> openshell.datamodel.v1.Provider + 229, // 49: openshell.v1.UpdateProviderRequest.provider:type_name -> openshell.datamodel.v1.Provider + 210, // 50: openshell.v1.UpdateProviderRequest.credential_expires_at_ms:type_name -> openshell.v1.UpdateProviderRequest.CredentialExpiresAtMsEntry + 229, // 51: openshell.v1.ProviderResponse.provider:type_name -> openshell.datamodel.v1.Provider + 229, // 52: openshell.v1.ListProvidersResponse.providers:type_name -> openshell.datamodel.v1.Provider 98, // 53: openshell.v1.ProviderProfileImportItem.profile:type_name -> openshell.v1.ProviderProfile 79, // 54: openshell.v1.ProviderCredentialTokenGrant.audience_overrides:type_name -> openshell.v1.ProviderCredentialTokenGrantAudienceOverride 1, // 55: openshell.v1.ProviderCredentialTokenGrant.grant_type:type_name -> openshell.v1.ProviderCredentialTokenGrantType @@ -15392,25 +15566,25 @@ var file_openshell_proto_depIdxs = []int32{ 83, // 60: openshell.v1.ProviderCredentialRefresh.material:type_name -> openshell.v1.ProviderCredentialRefreshMaterial 84, // 61: openshell.v1.ProviderCredentialRefresh.additional_outputs:type_name -> openshell.v1.ProviderCredentialRefreshOutput 2, // 62: openshell.v1.ProviderCredentialRefreshStatus.strategy:type_name -> openshell.v1.ProviderCredentialRefreshStrategy - 224, // 63: openshell.v1.StoredProviderCredentialRefreshState.metadata:type_name -> openshell.datamodel.v1.ObjectMeta + 226, // 63: openshell.v1.StoredProviderCredentialRefreshState.metadata:type_name -> openshell.datamodel.v1.ObjectMeta 2, // 64: openshell.v1.StoredProviderCredentialRefreshState.strategy:type_name -> openshell.v1.ProviderCredentialRefreshStrategy - 209, // 65: openshell.v1.StoredProviderCredentialRefreshState.material:type_name -> openshell.v1.StoredProviderCredentialRefreshState.MaterialEntry - 210, // 66: openshell.v1.StoredProviderCredentialRefreshState.additional_output_keys:type_name -> openshell.v1.StoredProviderCredentialRefreshState.AdditionalOutputKeysEntry - 211, // 67: openshell.v1.StoredProviderCredentialRefreshState.secret_material_handles:type_name -> openshell.v1.StoredProviderCredentialRefreshState.SecretMaterialHandlesEntry + 211, // 65: openshell.v1.StoredProviderCredentialRefreshState.material:type_name -> openshell.v1.StoredProviderCredentialRefreshState.MaterialEntry + 212, // 66: openshell.v1.StoredProviderCredentialRefreshState.additional_output_keys:type_name -> openshell.v1.StoredProviderCredentialRefreshState.AdditionalOutputKeysEntry + 213, // 67: openshell.v1.StoredProviderCredentialRefreshState.secret_material_handles:type_name -> openshell.v1.StoredProviderCredentialRefreshState.SecretMaterialHandlesEntry 89, // 68: openshell.v1.StoredProviderCredentialRefreshState.pending_secret_deletions:type_name -> openshell.v1.StoredRefreshMaterialDeletion - 228, // 69: openshell.v1.StoredRefreshMaterialDeletion.handle:type_name -> openshell.datamodel.v1.CredentialHandle + 230, // 69: openshell.v1.StoredRefreshMaterialDeletion.handle:type_name -> openshell.datamodel.v1.CredentialHandle 86, // 70: openshell.v1.GetProviderRefreshStatusResponse.credentials:type_name -> openshell.v1.ProviderCredentialRefreshStatus 2, // 71: openshell.v1.ConfigureProviderRefreshRequest.strategy:type_name -> openshell.v1.ProviderCredentialRefreshStrategy - 212, // 72: openshell.v1.ConfigureProviderRefreshRequest.material:type_name -> openshell.v1.ConfigureProviderRefreshRequest.MaterialEntry + 214, // 72: openshell.v1.ConfigureProviderRefreshRequest.material:type_name -> openshell.v1.ConfigureProviderRefreshRequest.MaterialEntry 86, // 73: openshell.v1.ConfigureProviderRefreshResponse.status:type_name -> openshell.v1.ProviderCredentialRefreshStatus 86, // 74: openshell.v1.RotateProviderCredentialResponse.status:type_name -> openshell.v1.ProviderCredentialRefreshStatus 3, // 75: openshell.v1.ProviderProfile.category:type_name -> openshell.v1.ProviderProfileCategory 82, // 76: openshell.v1.ProviderProfile.credentials:type_name -> openshell.v1.ProviderProfileCredential - 229, // 77: openshell.v1.ProviderProfile.endpoints:type_name -> openshell.sandbox.v1.NetworkEndpoint - 230, // 78: openshell.v1.ProviderProfile.binaries:type_name -> openshell.sandbox.v1.NetworkBinary + 231, // 77: openshell.v1.ProviderProfile.endpoints:type_name -> openshell.sandbox.v1.NetworkEndpoint + 232, // 78: openshell.v1.ProviderProfile.binaries:type_name -> openshell.sandbox.v1.NetworkBinary 87, // 79: openshell.v1.ProviderProfile.discovery:type_name -> openshell.v1.ProviderProfileDiscovery - 213, // 80: openshell.v1.ProviderProfile.annotations:type_name -> openshell.v1.ProviderProfile.AnnotationsEntry - 224, // 81: openshell.v1.StoredProviderProfile.metadata:type_name -> openshell.datamodel.v1.ObjectMeta + 215, // 80: openshell.v1.ProviderProfile.annotations:type_name -> openshell.v1.ProviderProfile.AnnotationsEntry + 226, // 81: openshell.v1.StoredProviderProfile.metadata:type_name -> openshell.datamodel.v1.ObjectMeta 98, // 82: openshell.v1.StoredProviderProfile.profile:type_name -> openshell.v1.ProviderProfile 98, // 83: openshell.v1.ProviderProfileResponse.profile:type_name -> openshell.v1.ProviderProfile 98, // 84: openshell.v1.ListProviderProfilesResponse.profiles:type_name -> openshell.v1.ProviderProfile @@ -15423,217 +15597,221 @@ var file_openshell_proto_depIdxs = []int32{ 77, // 91: openshell.v1.LintProviderProfilesRequest.profiles:type_name -> openshell.v1.ProviderProfileImportItem 78, // 92: openshell.v1.LintProviderProfilesResponse.diagnostics:type_name -> openshell.v1.ProviderProfileDiagnostic 112, // 93: openshell.v1.StaticCredentialBinding.endpoints:type_name -> openshell.v1.StaticCredentialEndpointBinding - 214, // 94: openshell.v1.GetSandboxProviderEnvironmentResponse.environment:type_name -> openshell.v1.GetSandboxProviderEnvironmentResponse.EnvironmentEntry - 215, // 95: openshell.v1.GetSandboxProviderEnvironmentResponse.credential_expires_at_ms:type_name -> openshell.v1.GetSandboxProviderEnvironmentResponse.CredentialExpiresAtMsEntry - 216, // 96: openshell.v1.GetSandboxProviderEnvironmentResponse.dynamic_credentials:type_name -> openshell.v1.GetSandboxProviderEnvironmentResponse.DynamicCredentialsEntry - 217, // 97: openshell.v1.GetSandboxProviderEnvironmentResponse.static_credential_bindings:type_name -> openshell.v1.GetSandboxProviderEnvironmentResponse.StaticCredentialBindingsEntry - 225, // 98: openshell.v1.UpdateConfigRequest.policy:type_name -> openshell.sandbox.v1.SandboxPolicy - 231, // 99: openshell.v1.UpdateConfigRequest.setting_value:type_name -> openshell.sandbox.v1.SettingValue + 216, // 94: openshell.v1.GetSandboxProviderEnvironmentResponse.environment:type_name -> openshell.v1.GetSandboxProviderEnvironmentResponse.EnvironmentEntry + 217, // 95: openshell.v1.GetSandboxProviderEnvironmentResponse.credential_expires_at_ms:type_name -> openshell.v1.GetSandboxProviderEnvironmentResponse.CredentialExpiresAtMsEntry + 218, // 96: openshell.v1.GetSandboxProviderEnvironmentResponse.dynamic_credentials:type_name -> openshell.v1.GetSandboxProviderEnvironmentResponse.DynamicCredentialsEntry + 219, // 97: openshell.v1.GetSandboxProviderEnvironmentResponse.static_credential_bindings:type_name -> openshell.v1.GetSandboxProviderEnvironmentResponse.StaticCredentialBindingsEntry + 227, // 98: openshell.v1.UpdateConfigRequest.policy:type_name -> openshell.sandbox.v1.SandboxPolicy + 233, // 99: openshell.v1.UpdateConfigRequest.setting_value:type_name -> openshell.sandbox.v1.SettingValue 118, // 100: openshell.v1.UpdateConfigRequest.merge_operations:type_name -> openshell.v1.PolicyMergeOperation - 218, // 101: openshell.v1.UpdateConfigRequest.annotations:type_name -> openshell.v1.UpdateConfigRequest.AnnotationsEntry + 220, // 101: openshell.v1.UpdateConfigRequest.annotations:type_name -> openshell.v1.UpdateConfigRequest.AnnotationsEntry 119, // 102: openshell.v1.PolicyMergeOperation.add_rule:type_name -> openshell.v1.AddNetworkRule 120, // 103: openshell.v1.PolicyMergeOperation.remove_endpoint:type_name -> openshell.v1.RemoveNetworkEndpoint 121, // 104: openshell.v1.PolicyMergeOperation.remove_rule:type_name -> openshell.v1.RemoveNetworkRule 122, // 105: openshell.v1.PolicyMergeOperation.add_deny_rules:type_name -> openshell.v1.AddDenyRules 123, // 106: openshell.v1.PolicyMergeOperation.add_allow_rules:type_name -> openshell.v1.AddAllowRules 124, // 107: openshell.v1.PolicyMergeOperation.remove_binary:type_name -> openshell.v1.RemoveNetworkBinary - 232, // 108: openshell.v1.AddNetworkRule.rule:type_name -> openshell.sandbox.v1.NetworkPolicyRule - 233, // 109: openshell.v1.AddDenyRules.deny_rules:type_name -> openshell.sandbox.v1.L7DenyRule - 234, // 110: openshell.v1.AddAllowRules.rules:type_name -> openshell.sandbox.v1.L7Rule - 219, // 111: openshell.v1.UpdateConfigResponse.annotations:type_name -> openshell.v1.UpdateConfigResponse.AnnotationsEntry + 234, // 108: openshell.v1.AddNetworkRule.rule:type_name -> openshell.sandbox.v1.NetworkPolicyRule + 235, // 109: openshell.v1.AddDenyRules.deny_rules:type_name -> openshell.sandbox.v1.L7DenyRule + 236, // 110: openshell.v1.AddAllowRules.rules:type_name -> openshell.sandbox.v1.L7Rule + 221, // 111: openshell.v1.UpdateConfigResponse.annotations:type_name -> openshell.v1.UpdateConfigResponse.AnnotationsEntry 132, // 112: openshell.v1.GetSandboxPolicyStatusResponse.revision:type_name -> openshell.v1.SandboxPolicyRevision 132, // 113: openshell.v1.ListSandboxPoliciesResponse.revisions:type_name -> openshell.v1.SandboxPolicyRevision 4, // 114: openshell.v1.ReportPolicyStatusRequest.status:type_name -> openshell.v1.PolicyStatus 4, // 115: openshell.v1.SandboxPolicyRevision.status:type_name -> openshell.v1.PolicyStatus - 225, // 116: openshell.v1.SandboxPolicyRevision.policy:type_name -> openshell.sandbox.v1.SandboxPolicy - 220, // 117: openshell.v1.SandboxPolicyRevision.provenance:type_name -> openshell.v1.SandboxPolicyRevision.ProvenanceEntry + 227, // 116: openshell.v1.SandboxPolicyRevision.policy:type_name -> openshell.sandbox.v1.SandboxPolicy + 222, // 117: openshell.v1.SandboxPolicyRevision.provenance:type_name -> openshell.v1.SandboxPolicyRevision.ProvenanceEntry 66, // 118: openshell.v1.PushSandboxLogsRequest.logs:type_name -> openshell.v1.SandboxLogLine 66, // 119: openshell.v1.GetSandboxLogsResponse.logs:type_name -> openshell.v1.SandboxLogLine 139, // 120: openshell.v1.SupervisorMessage.hello:type_name -> openshell.v1.SupervisorHello 142, // 121: openshell.v1.SupervisorMessage.heartbeat:type_name -> openshell.v1.SupervisorHeartbeat - 151, // 122: openshell.v1.SupervisorMessage.relay_open_result:type_name -> openshell.v1.RelayOpenResult - 152, // 123: openshell.v1.SupervisorMessage.relay_close:type_name -> openshell.v1.RelayClose + 153, // 122: openshell.v1.SupervisorMessage.relay_open_result:type_name -> openshell.v1.RelayOpenResult + 154, // 123: openshell.v1.SupervisorMessage.relay_close:type_name -> openshell.v1.RelayClose 140, // 124: openshell.v1.GatewayMessage.session_accepted:type_name -> openshell.v1.SessionAccepted 141, // 125: openshell.v1.GatewayMessage.session_rejected:type_name -> openshell.v1.SessionRejected 143, // 126: openshell.v1.GatewayMessage.heartbeat:type_name -> openshell.v1.GatewayHeartbeat 146, // 127: openshell.v1.GatewayMessage.relay_open:type_name -> openshell.v1.RelayOpen - 152, // 128: openshell.v1.GatewayMessage.relay_close:type_name -> openshell.v1.RelayClose + 154, // 128: openshell.v1.GatewayMessage.relay_close:type_name -> openshell.v1.RelayClose 147, // 129: openshell.v1.RelayOpen.ssh:type_name -> openshell.v1.SshRelayTarget 148, // 130: openshell.v1.RelayOpen.tcp:type_name -> openshell.v1.TcpRelayTarget 149, // 131: openshell.v1.RelayFrame.init:type_name -> openshell.v1.RelayInit - 153, // 132: openshell.v1.DenialSummary.l7_request_samples:type_name -> openshell.v1.L7RequestSample - 155, // 133: openshell.v1.NetworkActivitySummary.denials_by_group:type_name -> openshell.v1.DenialGroupCount - 232, // 134: openshell.v1.PolicyChunk.proposed_rule:type_name -> openshell.sandbox.v1.NetworkPolicyRule - 225, // 135: openshell.v1.PolicyChunk.current_effective_policy:type_name -> openshell.sandbox.v1.SandboxPolicy - 225, // 136: openshell.v1.PolicyChunk.candidate_effective_policy:type_name -> openshell.sandbox.v1.SandboxPolicy - 154, // 137: openshell.v1.SubmitPolicyAnalysisRequest.summaries:type_name -> openshell.v1.DenialSummary - 157, // 138: openshell.v1.SubmitPolicyAnalysisRequest.proposed_chunks:type_name -> openshell.v1.PolicyChunk - 156, // 139: openshell.v1.SubmitPolicyAnalysisRequest.network_activity_summaries:type_name -> openshell.v1.NetworkActivitySummary - 157, // 140: openshell.v1.GetDraftPolicyResponse.chunks:type_name -> openshell.v1.PolicyChunk - 167, // 141: openshell.v1.ApproveAllDraftChunksRequest.approvals:type_name -> openshell.v1.DraftChunkApproval - 232, // 142: openshell.v1.EditDraftChunkRequest.proposed_rule:type_name -> openshell.sandbox.v1.NetworkPolicyRule - 177, // 143: openshell.v1.GetDraftHistoryResponse.entries:type_name -> openshell.v1.DraftHistoryEntry - 225, // 144: openshell.v1.PolicyRevisionPayload.policy:type_name -> openshell.sandbox.v1.SandboxPolicy - 221, // 145: openshell.v1.PolicyRevisionPayload.provenance:type_name -> openshell.v1.PolicyRevisionPayload.ProvenanceEntry - 232, // 146: openshell.v1.DraftChunkPayload.proposed_rule:type_name -> openshell.sandbox.v1.NetworkPolicyRule - 225, // 147: openshell.v1.DraftChunkPayload.current_effective_policy:type_name -> openshell.sandbox.v1.SandboxPolicy - 225, // 148: openshell.v1.DraftChunkPayload.candidate_effective_policy:type_name -> openshell.sandbox.v1.SandboxPolicy - 222, // 149: openshell.v1.StoredPolicyRevision.provenance:type_name -> openshell.v1.StoredPolicyRevision.ProvenanceEntry - 225, // 150: openshell.v1.StoredDraftChunk.current_effective_policy:type_name -> openshell.sandbox.v1.SandboxPolicy - 225, // 151: openshell.v1.StoredDraftChunk.candidate_effective_policy:type_name -> openshell.sandbox.v1.SandboxPolicy - 223, // 152: openshell.v1.CreateWorkspaceRequest.labels:type_name -> openshell.v1.CreateWorkspaceRequest.LabelsEntry - 235, // 153: openshell.v1.CreateWorkspaceResponse.workspace:type_name -> openshell.datamodel.v1.Workspace - 235, // 154: openshell.v1.GetWorkspaceResponse.workspace:type_name -> openshell.datamodel.v1.Workspace - 235, // 155: openshell.v1.ListWorkspacesResponse.workspaces:type_name -> openshell.datamodel.v1.Workspace - 224, // 156: openshell.v1.WorkspaceMember.metadata:type_name -> openshell.datamodel.v1.ObjectMeta - 6, // 157: openshell.v1.WorkspaceMember.role:type_name -> openshell.v1.WorkspaceRole - 6, // 158: openshell.v1.AddWorkspaceMemberRequest.role:type_name -> openshell.v1.WorkspaceRole - 191, // 159: openshell.v1.AddWorkspaceMemberResponse.member:type_name -> openshell.v1.WorkspaceMember - 191, // 160: openshell.v1.ListWorkspaceMembersResponse.members:type_name -> openshell.v1.WorkspaceMember - 228, // 161: openshell.v1.StoredProviderCredentialRefreshState.SecretMaterialHandlesEntry.value:type_name -> openshell.datamodel.v1.CredentialHandle - 82, // 162: openshell.v1.GetSandboxProviderEnvironmentResponse.DynamicCredentialsEntry.value:type_name -> openshell.v1.ProviderProfileCredential - 113, // 163: openshell.v1.GetSandboxProviderEnvironmentResponse.StaticCredentialBindingsEntry.value:type_name -> openshell.v1.StaticCredentialBinding - 11, // 164: openshell.v1.OpenShell.Health:input_type -> openshell.v1.HealthRequest - 13, // 165: openshell.v1.OpenShell.GetCurrentUser:input_type -> openshell.v1.GetCurrentUserRequest - 15, // 166: openshell.v1.OpenShell.GetGatewayInfo:input_type -> openshell.v1.GetGatewayInfoRequest - 27, // 167: openshell.v1.OpenShell.CreateSandbox:input_type -> openshell.v1.CreateSandboxRequest - 28, // 168: openshell.v1.OpenShell.GetSandbox:input_type -> openshell.v1.GetSandboxRequest - 29, // 169: openshell.v1.OpenShell.ListSandboxes:input_type -> openshell.v1.ListSandboxesRequest - 30, // 170: openshell.v1.OpenShell.ListSandboxProviders:input_type -> openshell.v1.ListSandboxProvidersRequest - 31, // 171: openshell.v1.OpenShell.AttachSandboxProvider:input_type -> openshell.v1.AttachSandboxProviderRequest - 32, // 172: openshell.v1.OpenShell.DetachSandboxProvider:input_type -> openshell.v1.DetachSandboxProviderRequest - 33, // 173: openshell.v1.OpenShell.DeleteSandbox:input_type -> openshell.v1.DeleteSandboxRequest - 34, // 174: openshell.v1.OpenShell.StopSandbox:input_type -> openshell.v1.StopSandboxRequest - 35, // 175: openshell.v1.OpenShell.StartSandbox:input_type -> openshell.v1.StartSandboxRequest - 42, // 176: openshell.v1.OpenShell.CreateSshSession:input_type -> openshell.v1.CreateSshSessionRequest - 44, // 177: openshell.v1.OpenShell.ExposeService:input_type -> openshell.v1.ExposeServiceRequest - 45, // 178: openshell.v1.OpenShell.GetService:input_type -> openshell.v1.GetServiceRequest - 46, // 179: openshell.v1.OpenShell.ListServices:input_type -> openshell.v1.ListServicesRequest - 48, // 180: openshell.v1.OpenShell.DeleteService:input_type -> openshell.v1.DeleteServiceRequest - 52, // 181: openshell.v1.OpenShell.RevokeSshSession:input_type -> openshell.v1.RevokeSshSessionRequest - 54, // 182: openshell.v1.OpenShell.ExecSandbox:input_type -> openshell.v1.ExecSandboxRequest - 60, // 183: openshell.v1.OpenShell.ForwardTcp:input_type -> openshell.v1.TcpForwardFrame - 61, // 184: openshell.v1.OpenShell.ExecSandboxInteractive:input_type -> openshell.v1.ExecSandboxInput - 68, // 185: openshell.v1.OpenShell.CreateProvider:input_type -> openshell.v1.CreateProviderRequest - 69, // 186: openshell.v1.OpenShell.GetProvider:input_type -> openshell.v1.GetProviderRequest - 70, // 187: openshell.v1.OpenShell.ListProviders:input_type -> openshell.v1.ListProvidersRequest - 75, // 188: openshell.v1.OpenShell.ListProviderProfiles:input_type -> openshell.v1.ListProviderProfilesRequest - 76, // 189: openshell.v1.OpenShell.GetProviderProfile:input_type -> openshell.v1.GetProviderProfileRequest - 102, // 190: openshell.v1.OpenShell.ImportProviderProfiles:input_type -> openshell.v1.ImportProviderProfilesRequest - 104, // 191: openshell.v1.OpenShell.UpdateProviderProfiles:input_type -> openshell.v1.UpdateProviderProfilesRequest - 106, // 192: openshell.v1.OpenShell.LintProviderProfiles:input_type -> openshell.v1.LintProviderProfilesRequest - 71, // 193: openshell.v1.OpenShell.UpdateProvider:input_type -> openshell.v1.UpdateProviderRequest - 90, // 194: openshell.v1.OpenShell.GetProviderRefreshStatus:input_type -> openshell.v1.GetProviderRefreshStatusRequest - 92, // 195: openshell.v1.OpenShell.ConfigureProviderRefresh:input_type -> openshell.v1.ConfigureProviderRefreshRequest - 94, // 196: openshell.v1.OpenShell.RotateProviderCredential:input_type -> openshell.v1.RotateProviderCredentialRequest - 96, // 197: openshell.v1.OpenShell.DeleteProviderRefresh:input_type -> openshell.v1.DeleteProviderRefreshRequest - 72, // 198: openshell.v1.OpenShell.DeleteProvider:input_type -> openshell.v1.DeleteProviderRequest - 109, // 199: openshell.v1.OpenShell.DeleteProviderProfile:input_type -> openshell.v1.DeleteProviderProfileRequest - 236, // 200: openshell.v1.OpenShell.GetSandboxConfig:input_type -> openshell.sandbox.v1.GetSandboxConfigRequest - 237, // 201: openshell.v1.OpenShell.GetGatewayConfig:input_type -> openshell.sandbox.v1.GetGatewayConfigRequest - 117, // 202: openshell.v1.OpenShell.UpdateConfig:input_type -> openshell.v1.UpdateConfigRequest - 126, // 203: openshell.v1.OpenShell.GetSandboxPolicyStatus:input_type -> openshell.v1.GetSandboxPolicyStatusRequest - 128, // 204: openshell.v1.OpenShell.ListSandboxPolicies:input_type -> openshell.v1.ListSandboxPoliciesRequest - 130, // 205: openshell.v1.OpenShell.ReportPolicyStatus:input_type -> openshell.v1.ReportPolicyStatusRequest - 111, // 206: openshell.v1.OpenShell.GetSandboxProviderEnvironment:input_type -> openshell.v1.GetSandboxProviderEnvironmentRequest - 115, // 207: openshell.v1.OpenShell.ExchangeProviderSubjectToken:input_type -> openshell.v1.ExchangeProviderSubjectTokenRequest - 133, // 208: openshell.v1.OpenShell.GetSandboxLogs:input_type -> openshell.v1.GetSandboxLogsRequest - 134, // 209: openshell.v1.OpenShell.PushSandboxLogs:input_type -> openshell.v1.PushSandboxLogsRequest - 137, // 210: openshell.v1.OpenShell.ConnectSupervisor:input_type -> openshell.v1.SupervisorMessage - 144, // 211: openshell.v1.OpenShell.ReportMainProcessExit:input_type -> openshell.v1.ReportMainProcessExitRequest - 150, // 212: openshell.v1.OpenShell.RelayStream:input_type -> openshell.v1.RelayFrame - 64, // 213: openshell.v1.OpenShell.WatchSandbox:input_type -> openshell.v1.WatchSandboxRequest - 159, // 214: openshell.v1.OpenShell.SubmitPolicyAnalysis:input_type -> openshell.v1.SubmitPolicyAnalysisRequest - 161, // 215: openshell.v1.OpenShell.GetDraftPolicy:input_type -> openshell.v1.GetDraftPolicyRequest - 163, // 216: openshell.v1.OpenShell.ApproveDraftChunk:input_type -> openshell.v1.ApproveDraftChunkRequest - 165, // 217: openshell.v1.OpenShell.RejectDraftChunk:input_type -> openshell.v1.RejectDraftChunkRequest - 168, // 218: openshell.v1.OpenShell.ApproveAllDraftChunks:input_type -> openshell.v1.ApproveAllDraftChunksRequest - 170, // 219: openshell.v1.OpenShell.EditDraftChunk:input_type -> openshell.v1.EditDraftChunkRequest - 172, // 220: openshell.v1.OpenShell.UndoDraftChunk:input_type -> openshell.v1.UndoDraftChunkRequest - 174, // 221: openshell.v1.OpenShell.ClearDraftChunks:input_type -> openshell.v1.ClearDraftChunksRequest - 176, // 222: openshell.v1.OpenShell.GetDraftHistory:input_type -> openshell.v1.GetDraftHistoryRequest - 7, // 223: openshell.v1.OpenShell.IssueSandboxToken:input_type -> openshell.v1.IssueSandboxTokenRequest - 9, // 224: openshell.v1.OpenShell.RefreshSandboxToken:input_type -> openshell.v1.RefreshSandboxTokenRequest - 183, // 225: openshell.v1.OpenShell.CreateWorkspace:input_type -> openshell.v1.CreateWorkspaceRequest - 185, // 226: openshell.v1.OpenShell.GetWorkspace:input_type -> openshell.v1.GetWorkspaceRequest - 187, // 227: openshell.v1.OpenShell.ListWorkspaces:input_type -> openshell.v1.ListWorkspacesRequest - 189, // 228: openshell.v1.OpenShell.DeleteWorkspace:input_type -> openshell.v1.DeleteWorkspaceRequest - 192, // 229: openshell.v1.OpenShell.AddWorkspaceMember:input_type -> openshell.v1.AddWorkspaceMemberRequest - 194, // 230: openshell.v1.OpenShell.RemoveWorkspaceMember:input_type -> openshell.v1.RemoveWorkspaceMemberRequest - 196, // 231: openshell.v1.OpenShell.ListWorkspaceMembers:input_type -> openshell.v1.ListWorkspaceMembersRequest - 12, // 232: openshell.v1.OpenShell.Health:output_type -> openshell.v1.HealthResponse - 14, // 233: openshell.v1.OpenShell.GetCurrentUser:output_type -> openshell.v1.GetCurrentUserResponse - 16, // 234: openshell.v1.OpenShell.GetGatewayInfo:output_type -> openshell.v1.GetGatewayInfoResponse - 36, // 235: openshell.v1.OpenShell.CreateSandbox:output_type -> openshell.v1.SandboxResponse - 36, // 236: openshell.v1.OpenShell.GetSandbox:output_type -> openshell.v1.SandboxResponse - 37, // 237: openshell.v1.OpenShell.ListSandboxes:output_type -> openshell.v1.ListSandboxesResponse - 38, // 238: openshell.v1.OpenShell.ListSandboxProviders:output_type -> openshell.v1.ListSandboxProvidersResponse - 39, // 239: openshell.v1.OpenShell.AttachSandboxProvider:output_type -> openshell.v1.AttachSandboxProviderResponse - 40, // 240: openshell.v1.OpenShell.DetachSandboxProvider:output_type -> openshell.v1.DetachSandboxProviderResponse - 41, // 241: openshell.v1.OpenShell.DeleteSandbox:output_type -> openshell.v1.DeleteSandboxResponse - 36, // 242: openshell.v1.OpenShell.StopSandbox:output_type -> openshell.v1.SandboxResponse - 36, // 243: openshell.v1.OpenShell.StartSandbox:output_type -> openshell.v1.SandboxResponse - 43, // 244: openshell.v1.OpenShell.CreateSshSession:output_type -> openshell.v1.CreateSshSessionResponse - 51, // 245: openshell.v1.OpenShell.ExposeService:output_type -> openshell.v1.ServiceEndpointResponse - 51, // 246: openshell.v1.OpenShell.GetService:output_type -> openshell.v1.ServiceEndpointResponse - 47, // 247: openshell.v1.OpenShell.ListServices:output_type -> openshell.v1.ListServicesResponse - 49, // 248: openshell.v1.OpenShell.DeleteService:output_type -> openshell.v1.DeleteServiceResponse - 53, // 249: openshell.v1.OpenShell.RevokeSshSession:output_type -> openshell.v1.RevokeSshSessionResponse - 58, // 250: openshell.v1.OpenShell.ExecSandbox:output_type -> openshell.v1.ExecSandboxEvent - 60, // 251: openshell.v1.OpenShell.ForwardTcp:output_type -> openshell.v1.TcpForwardFrame - 58, // 252: openshell.v1.OpenShell.ExecSandboxInteractive:output_type -> openshell.v1.ExecSandboxEvent - 73, // 253: openshell.v1.OpenShell.CreateProvider:output_type -> openshell.v1.ProviderResponse - 73, // 254: openshell.v1.OpenShell.GetProvider:output_type -> openshell.v1.ProviderResponse - 74, // 255: openshell.v1.OpenShell.ListProviders:output_type -> openshell.v1.ListProvidersResponse - 101, // 256: openshell.v1.OpenShell.ListProviderProfiles:output_type -> openshell.v1.ListProviderProfilesResponse - 100, // 257: openshell.v1.OpenShell.GetProviderProfile:output_type -> openshell.v1.ProviderProfileResponse - 103, // 258: openshell.v1.OpenShell.ImportProviderProfiles:output_type -> openshell.v1.ImportProviderProfilesResponse - 105, // 259: openshell.v1.OpenShell.UpdateProviderProfiles:output_type -> openshell.v1.UpdateProviderProfilesResponse - 107, // 260: openshell.v1.OpenShell.LintProviderProfiles:output_type -> openshell.v1.LintProviderProfilesResponse - 73, // 261: openshell.v1.OpenShell.UpdateProvider:output_type -> openshell.v1.ProviderResponse - 91, // 262: openshell.v1.OpenShell.GetProviderRefreshStatus:output_type -> openshell.v1.GetProviderRefreshStatusResponse - 93, // 263: openshell.v1.OpenShell.ConfigureProviderRefresh:output_type -> openshell.v1.ConfigureProviderRefreshResponse - 95, // 264: openshell.v1.OpenShell.RotateProviderCredential:output_type -> openshell.v1.RotateProviderCredentialResponse - 97, // 265: openshell.v1.OpenShell.DeleteProviderRefresh:output_type -> openshell.v1.DeleteProviderRefreshResponse - 108, // 266: openshell.v1.OpenShell.DeleteProvider:output_type -> openshell.v1.DeleteProviderResponse - 110, // 267: openshell.v1.OpenShell.DeleteProviderProfile:output_type -> openshell.v1.DeleteProviderProfileResponse - 238, // 268: openshell.v1.OpenShell.GetSandboxConfig:output_type -> openshell.sandbox.v1.GetSandboxConfigResponse - 239, // 269: openshell.v1.OpenShell.GetGatewayConfig:output_type -> openshell.sandbox.v1.GetGatewayConfigResponse - 125, // 270: openshell.v1.OpenShell.UpdateConfig:output_type -> openshell.v1.UpdateConfigResponse - 127, // 271: openshell.v1.OpenShell.GetSandboxPolicyStatus:output_type -> openshell.v1.GetSandboxPolicyStatusResponse - 129, // 272: openshell.v1.OpenShell.ListSandboxPolicies:output_type -> openshell.v1.ListSandboxPoliciesResponse - 131, // 273: openshell.v1.OpenShell.ReportPolicyStatus:output_type -> openshell.v1.ReportPolicyStatusResponse - 114, // 274: openshell.v1.OpenShell.GetSandboxProviderEnvironment:output_type -> openshell.v1.GetSandboxProviderEnvironmentResponse - 116, // 275: openshell.v1.OpenShell.ExchangeProviderSubjectToken:output_type -> openshell.v1.ExchangeProviderSubjectTokenResponse - 136, // 276: openshell.v1.OpenShell.GetSandboxLogs:output_type -> openshell.v1.GetSandboxLogsResponse - 135, // 277: openshell.v1.OpenShell.PushSandboxLogs:output_type -> openshell.v1.PushSandboxLogsResponse - 138, // 278: openshell.v1.OpenShell.ConnectSupervisor:output_type -> openshell.v1.GatewayMessage - 145, // 279: openshell.v1.OpenShell.ReportMainProcessExit:output_type -> openshell.v1.ReportMainProcessExitResponse - 150, // 280: openshell.v1.OpenShell.RelayStream:output_type -> openshell.v1.RelayFrame - 65, // 281: openshell.v1.OpenShell.WatchSandbox:output_type -> openshell.v1.SandboxStreamEvent - 160, // 282: openshell.v1.OpenShell.SubmitPolicyAnalysis:output_type -> openshell.v1.SubmitPolicyAnalysisResponse - 162, // 283: openshell.v1.OpenShell.GetDraftPolicy:output_type -> openshell.v1.GetDraftPolicyResponse - 164, // 284: openshell.v1.OpenShell.ApproveDraftChunk:output_type -> openshell.v1.ApproveDraftChunkResponse - 166, // 285: openshell.v1.OpenShell.RejectDraftChunk:output_type -> openshell.v1.RejectDraftChunkResponse - 169, // 286: openshell.v1.OpenShell.ApproveAllDraftChunks:output_type -> openshell.v1.ApproveAllDraftChunksResponse - 171, // 287: openshell.v1.OpenShell.EditDraftChunk:output_type -> openshell.v1.EditDraftChunkResponse - 173, // 288: openshell.v1.OpenShell.UndoDraftChunk:output_type -> openshell.v1.UndoDraftChunkResponse - 175, // 289: openshell.v1.OpenShell.ClearDraftChunks:output_type -> openshell.v1.ClearDraftChunksResponse - 178, // 290: openshell.v1.OpenShell.GetDraftHistory:output_type -> openshell.v1.GetDraftHistoryResponse - 8, // 291: openshell.v1.OpenShell.IssueSandboxToken:output_type -> openshell.v1.IssueSandboxTokenResponse - 10, // 292: openshell.v1.OpenShell.RefreshSandboxToken:output_type -> openshell.v1.RefreshSandboxTokenResponse - 184, // 293: openshell.v1.OpenShell.CreateWorkspace:output_type -> openshell.v1.CreateWorkspaceResponse - 186, // 294: openshell.v1.OpenShell.GetWorkspace:output_type -> openshell.v1.GetWorkspaceResponse - 188, // 295: openshell.v1.OpenShell.ListWorkspaces:output_type -> openshell.v1.ListWorkspacesResponse - 190, // 296: openshell.v1.OpenShell.DeleteWorkspace:output_type -> openshell.v1.DeleteWorkspaceResponse - 193, // 297: openshell.v1.OpenShell.AddWorkspaceMember:output_type -> openshell.v1.AddWorkspaceMemberResponse - 195, // 298: openshell.v1.OpenShell.RemoveWorkspaceMember:output_type -> openshell.v1.RemoveWorkspaceMemberResponse - 197, // 299: openshell.v1.OpenShell.ListWorkspaceMembers:output_type -> openshell.v1.ListWorkspaceMembersResponse - 232, // [232:300] is the sub-list for method output_type - 164, // [164:232] is the sub-list for method input_type - 164, // [164:164] is the sub-list for extension type_name - 164, // [164:164] is the sub-list for extension extendee - 0, // [0:164] is the sub-list for field type_name + 146, // 132: openshell.v1.PeerRelayInit.relay_open:type_name -> openshell.v1.RelayOpen + 151, // 133: openshell.v1.PeerRelayFrame.init:type_name -> openshell.v1.PeerRelayInit + 155, // 134: openshell.v1.DenialSummary.l7_request_samples:type_name -> openshell.v1.L7RequestSample + 157, // 135: openshell.v1.NetworkActivitySummary.denials_by_group:type_name -> openshell.v1.DenialGroupCount + 234, // 136: openshell.v1.PolicyChunk.proposed_rule:type_name -> openshell.sandbox.v1.NetworkPolicyRule + 227, // 137: openshell.v1.PolicyChunk.current_effective_policy:type_name -> openshell.sandbox.v1.SandboxPolicy + 227, // 138: openshell.v1.PolicyChunk.candidate_effective_policy:type_name -> openshell.sandbox.v1.SandboxPolicy + 156, // 139: openshell.v1.SubmitPolicyAnalysisRequest.summaries:type_name -> openshell.v1.DenialSummary + 159, // 140: openshell.v1.SubmitPolicyAnalysisRequest.proposed_chunks:type_name -> openshell.v1.PolicyChunk + 158, // 141: openshell.v1.SubmitPolicyAnalysisRequest.network_activity_summaries:type_name -> openshell.v1.NetworkActivitySummary + 159, // 142: openshell.v1.GetDraftPolicyResponse.chunks:type_name -> openshell.v1.PolicyChunk + 169, // 143: openshell.v1.ApproveAllDraftChunksRequest.approvals:type_name -> openshell.v1.DraftChunkApproval + 234, // 144: openshell.v1.EditDraftChunkRequest.proposed_rule:type_name -> openshell.sandbox.v1.NetworkPolicyRule + 179, // 145: openshell.v1.GetDraftHistoryResponse.entries:type_name -> openshell.v1.DraftHistoryEntry + 227, // 146: openshell.v1.PolicyRevisionPayload.policy:type_name -> openshell.sandbox.v1.SandboxPolicy + 223, // 147: openshell.v1.PolicyRevisionPayload.provenance:type_name -> openshell.v1.PolicyRevisionPayload.ProvenanceEntry + 234, // 148: openshell.v1.DraftChunkPayload.proposed_rule:type_name -> openshell.sandbox.v1.NetworkPolicyRule + 227, // 149: openshell.v1.DraftChunkPayload.current_effective_policy:type_name -> openshell.sandbox.v1.SandboxPolicy + 227, // 150: openshell.v1.DraftChunkPayload.candidate_effective_policy:type_name -> openshell.sandbox.v1.SandboxPolicy + 224, // 151: openshell.v1.StoredPolicyRevision.provenance:type_name -> openshell.v1.StoredPolicyRevision.ProvenanceEntry + 227, // 152: openshell.v1.StoredDraftChunk.current_effective_policy:type_name -> openshell.sandbox.v1.SandboxPolicy + 227, // 153: openshell.v1.StoredDraftChunk.candidate_effective_policy:type_name -> openshell.sandbox.v1.SandboxPolicy + 225, // 154: openshell.v1.CreateWorkspaceRequest.labels:type_name -> openshell.v1.CreateWorkspaceRequest.LabelsEntry + 237, // 155: openshell.v1.CreateWorkspaceResponse.workspace:type_name -> openshell.datamodel.v1.Workspace + 237, // 156: openshell.v1.GetWorkspaceResponse.workspace:type_name -> openshell.datamodel.v1.Workspace + 237, // 157: openshell.v1.ListWorkspacesResponse.workspaces:type_name -> openshell.datamodel.v1.Workspace + 226, // 158: openshell.v1.WorkspaceMember.metadata:type_name -> openshell.datamodel.v1.ObjectMeta + 6, // 159: openshell.v1.WorkspaceMember.role:type_name -> openshell.v1.WorkspaceRole + 6, // 160: openshell.v1.AddWorkspaceMemberRequest.role:type_name -> openshell.v1.WorkspaceRole + 193, // 161: openshell.v1.AddWorkspaceMemberResponse.member:type_name -> openshell.v1.WorkspaceMember + 193, // 162: openshell.v1.ListWorkspaceMembersResponse.members:type_name -> openshell.v1.WorkspaceMember + 230, // 163: openshell.v1.StoredProviderCredentialRefreshState.SecretMaterialHandlesEntry.value:type_name -> openshell.datamodel.v1.CredentialHandle + 82, // 164: openshell.v1.GetSandboxProviderEnvironmentResponse.DynamicCredentialsEntry.value:type_name -> openshell.v1.ProviderProfileCredential + 113, // 165: openshell.v1.GetSandboxProviderEnvironmentResponse.StaticCredentialBindingsEntry.value:type_name -> openshell.v1.StaticCredentialBinding + 11, // 166: openshell.v1.OpenShell.Health:input_type -> openshell.v1.HealthRequest + 13, // 167: openshell.v1.OpenShell.GetCurrentUser:input_type -> openshell.v1.GetCurrentUserRequest + 15, // 168: openshell.v1.OpenShell.GetGatewayInfo:input_type -> openshell.v1.GetGatewayInfoRequest + 27, // 169: openshell.v1.OpenShell.CreateSandbox:input_type -> openshell.v1.CreateSandboxRequest + 28, // 170: openshell.v1.OpenShell.GetSandbox:input_type -> openshell.v1.GetSandboxRequest + 29, // 171: openshell.v1.OpenShell.ListSandboxes:input_type -> openshell.v1.ListSandboxesRequest + 30, // 172: openshell.v1.OpenShell.ListSandboxProviders:input_type -> openshell.v1.ListSandboxProvidersRequest + 31, // 173: openshell.v1.OpenShell.AttachSandboxProvider:input_type -> openshell.v1.AttachSandboxProviderRequest + 32, // 174: openshell.v1.OpenShell.DetachSandboxProvider:input_type -> openshell.v1.DetachSandboxProviderRequest + 33, // 175: openshell.v1.OpenShell.DeleteSandbox:input_type -> openshell.v1.DeleteSandboxRequest + 34, // 176: openshell.v1.OpenShell.StopSandbox:input_type -> openshell.v1.StopSandboxRequest + 35, // 177: openshell.v1.OpenShell.StartSandbox:input_type -> openshell.v1.StartSandboxRequest + 42, // 178: openshell.v1.OpenShell.CreateSshSession:input_type -> openshell.v1.CreateSshSessionRequest + 44, // 179: openshell.v1.OpenShell.ExposeService:input_type -> openshell.v1.ExposeServiceRequest + 45, // 180: openshell.v1.OpenShell.GetService:input_type -> openshell.v1.GetServiceRequest + 46, // 181: openshell.v1.OpenShell.ListServices:input_type -> openshell.v1.ListServicesRequest + 48, // 182: openshell.v1.OpenShell.DeleteService:input_type -> openshell.v1.DeleteServiceRequest + 52, // 183: openshell.v1.OpenShell.RevokeSshSession:input_type -> openshell.v1.RevokeSshSessionRequest + 54, // 184: openshell.v1.OpenShell.ExecSandbox:input_type -> openshell.v1.ExecSandboxRequest + 60, // 185: openshell.v1.OpenShell.ForwardTcp:input_type -> openshell.v1.TcpForwardFrame + 61, // 186: openshell.v1.OpenShell.ExecSandboxInteractive:input_type -> openshell.v1.ExecSandboxInput + 68, // 187: openshell.v1.OpenShell.CreateProvider:input_type -> openshell.v1.CreateProviderRequest + 69, // 188: openshell.v1.OpenShell.GetProvider:input_type -> openshell.v1.GetProviderRequest + 70, // 189: openshell.v1.OpenShell.ListProviders:input_type -> openshell.v1.ListProvidersRequest + 75, // 190: openshell.v1.OpenShell.ListProviderProfiles:input_type -> openshell.v1.ListProviderProfilesRequest + 76, // 191: openshell.v1.OpenShell.GetProviderProfile:input_type -> openshell.v1.GetProviderProfileRequest + 102, // 192: openshell.v1.OpenShell.ImportProviderProfiles:input_type -> openshell.v1.ImportProviderProfilesRequest + 104, // 193: openshell.v1.OpenShell.UpdateProviderProfiles:input_type -> openshell.v1.UpdateProviderProfilesRequest + 106, // 194: openshell.v1.OpenShell.LintProviderProfiles:input_type -> openshell.v1.LintProviderProfilesRequest + 71, // 195: openshell.v1.OpenShell.UpdateProvider:input_type -> openshell.v1.UpdateProviderRequest + 90, // 196: openshell.v1.OpenShell.GetProviderRefreshStatus:input_type -> openshell.v1.GetProviderRefreshStatusRequest + 92, // 197: openshell.v1.OpenShell.ConfigureProviderRefresh:input_type -> openshell.v1.ConfigureProviderRefreshRequest + 94, // 198: openshell.v1.OpenShell.RotateProviderCredential:input_type -> openshell.v1.RotateProviderCredentialRequest + 96, // 199: openshell.v1.OpenShell.DeleteProviderRefresh:input_type -> openshell.v1.DeleteProviderRefreshRequest + 72, // 200: openshell.v1.OpenShell.DeleteProvider:input_type -> openshell.v1.DeleteProviderRequest + 109, // 201: openshell.v1.OpenShell.DeleteProviderProfile:input_type -> openshell.v1.DeleteProviderProfileRequest + 238, // 202: openshell.v1.OpenShell.GetSandboxConfig:input_type -> openshell.sandbox.v1.GetSandboxConfigRequest + 239, // 203: openshell.v1.OpenShell.GetGatewayConfig:input_type -> openshell.sandbox.v1.GetGatewayConfigRequest + 117, // 204: openshell.v1.OpenShell.UpdateConfig:input_type -> openshell.v1.UpdateConfigRequest + 126, // 205: openshell.v1.OpenShell.GetSandboxPolicyStatus:input_type -> openshell.v1.GetSandboxPolicyStatusRequest + 128, // 206: openshell.v1.OpenShell.ListSandboxPolicies:input_type -> openshell.v1.ListSandboxPoliciesRequest + 130, // 207: openshell.v1.OpenShell.ReportPolicyStatus:input_type -> openshell.v1.ReportPolicyStatusRequest + 111, // 208: openshell.v1.OpenShell.GetSandboxProviderEnvironment:input_type -> openshell.v1.GetSandboxProviderEnvironmentRequest + 115, // 209: openshell.v1.OpenShell.ExchangeProviderSubjectToken:input_type -> openshell.v1.ExchangeProviderSubjectTokenRequest + 133, // 210: openshell.v1.OpenShell.GetSandboxLogs:input_type -> openshell.v1.GetSandboxLogsRequest + 134, // 211: openshell.v1.OpenShell.PushSandboxLogs:input_type -> openshell.v1.PushSandboxLogsRequest + 137, // 212: openshell.v1.OpenShell.ConnectSupervisor:input_type -> openshell.v1.SupervisorMessage + 144, // 213: openshell.v1.OpenShell.ReportMainProcessExit:input_type -> openshell.v1.ReportMainProcessExitRequest + 150, // 214: openshell.v1.OpenShell.RelayStream:input_type -> openshell.v1.RelayFrame + 152, // 215: openshell.v1.OpenShell.PeerRelay:input_type -> openshell.v1.PeerRelayFrame + 64, // 216: openshell.v1.OpenShell.WatchSandbox:input_type -> openshell.v1.WatchSandboxRequest + 161, // 217: openshell.v1.OpenShell.SubmitPolicyAnalysis:input_type -> openshell.v1.SubmitPolicyAnalysisRequest + 163, // 218: openshell.v1.OpenShell.GetDraftPolicy:input_type -> openshell.v1.GetDraftPolicyRequest + 165, // 219: openshell.v1.OpenShell.ApproveDraftChunk:input_type -> openshell.v1.ApproveDraftChunkRequest + 167, // 220: openshell.v1.OpenShell.RejectDraftChunk:input_type -> openshell.v1.RejectDraftChunkRequest + 170, // 221: openshell.v1.OpenShell.ApproveAllDraftChunks:input_type -> openshell.v1.ApproveAllDraftChunksRequest + 172, // 222: openshell.v1.OpenShell.EditDraftChunk:input_type -> openshell.v1.EditDraftChunkRequest + 174, // 223: openshell.v1.OpenShell.UndoDraftChunk:input_type -> openshell.v1.UndoDraftChunkRequest + 176, // 224: openshell.v1.OpenShell.ClearDraftChunks:input_type -> openshell.v1.ClearDraftChunksRequest + 178, // 225: openshell.v1.OpenShell.GetDraftHistory:input_type -> openshell.v1.GetDraftHistoryRequest + 7, // 226: openshell.v1.OpenShell.IssueSandboxToken:input_type -> openshell.v1.IssueSandboxTokenRequest + 9, // 227: openshell.v1.OpenShell.RefreshSandboxToken:input_type -> openshell.v1.RefreshSandboxTokenRequest + 185, // 228: openshell.v1.OpenShell.CreateWorkspace:input_type -> openshell.v1.CreateWorkspaceRequest + 187, // 229: openshell.v1.OpenShell.GetWorkspace:input_type -> openshell.v1.GetWorkspaceRequest + 189, // 230: openshell.v1.OpenShell.ListWorkspaces:input_type -> openshell.v1.ListWorkspacesRequest + 191, // 231: openshell.v1.OpenShell.DeleteWorkspace:input_type -> openshell.v1.DeleteWorkspaceRequest + 194, // 232: openshell.v1.OpenShell.AddWorkspaceMember:input_type -> openshell.v1.AddWorkspaceMemberRequest + 196, // 233: openshell.v1.OpenShell.RemoveWorkspaceMember:input_type -> openshell.v1.RemoveWorkspaceMemberRequest + 198, // 234: openshell.v1.OpenShell.ListWorkspaceMembers:input_type -> openshell.v1.ListWorkspaceMembersRequest + 12, // 235: openshell.v1.OpenShell.Health:output_type -> openshell.v1.HealthResponse + 14, // 236: openshell.v1.OpenShell.GetCurrentUser:output_type -> openshell.v1.GetCurrentUserResponse + 16, // 237: openshell.v1.OpenShell.GetGatewayInfo:output_type -> openshell.v1.GetGatewayInfoResponse + 36, // 238: openshell.v1.OpenShell.CreateSandbox:output_type -> openshell.v1.SandboxResponse + 36, // 239: openshell.v1.OpenShell.GetSandbox:output_type -> openshell.v1.SandboxResponse + 37, // 240: openshell.v1.OpenShell.ListSandboxes:output_type -> openshell.v1.ListSandboxesResponse + 38, // 241: openshell.v1.OpenShell.ListSandboxProviders:output_type -> openshell.v1.ListSandboxProvidersResponse + 39, // 242: openshell.v1.OpenShell.AttachSandboxProvider:output_type -> openshell.v1.AttachSandboxProviderResponse + 40, // 243: openshell.v1.OpenShell.DetachSandboxProvider:output_type -> openshell.v1.DetachSandboxProviderResponse + 41, // 244: openshell.v1.OpenShell.DeleteSandbox:output_type -> openshell.v1.DeleteSandboxResponse + 36, // 245: openshell.v1.OpenShell.StopSandbox:output_type -> openshell.v1.SandboxResponse + 36, // 246: openshell.v1.OpenShell.StartSandbox:output_type -> openshell.v1.SandboxResponse + 43, // 247: openshell.v1.OpenShell.CreateSshSession:output_type -> openshell.v1.CreateSshSessionResponse + 51, // 248: openshell.v1.OpenShell.ExposeService:output_type -> openshell.v1.ServiceEndpointResponse + 51, // 249: openshell.v1.OpenShell.GetService:output_type -> openshell.v1.ServiceEndpointResponse + 47, // 250: openshell.v1.OpenShell.ListServices:output_type -> openshell.v1.ListServicesResponse + 49, // 251: openshell.v1.OpenShell.DeleteService:output_type -> openshell.v1.DeleteServiceResponse + 53, // 252: openshell.v1.OpenShell.RevokeSshSession:output_type -> openshell.v1.RevokeSshSessionResponse + 58, // 253: openshell.v1.OpenShell.ExecSandbox:output_type -> openshell.v1.ExecSandboxEvent + 60, // 254: openshell.v1.OpenShell.ForwardTcp:output_type -> openshell.v1.TcpForwardFrame + 58, // 255: openshell.v1.OpenShell.ExecSandboxInteractive:output_type -> openshell.v1.ExecSandboxEvent + 73, // 256: openshell.v1.OpenShell.CreateProvider:output_type -> openshell.v1.ProviderResponse + 73, // 257: openshell.v1.OpenShell.GetProvider:output_type -> openshell.v1.ProviderResponse + 74, // 258: openshell.v1.OpenShell.ListProviders:output_type -> openshell.v1.ListProvidersResponse + 101, // 259: openshell.v1.OpenShell.ListProviderProfiles:output_type -> openshell.v1.ListProviderProfilesResponse + 100, // 260: openshell.v1.OpenShell.GetProviderProfile:output_type -> openshell.v1.ProviderProfileResponse + 103, // 261: openshell.v1.OpenShell.ImportProviderProfiles:output_type -> openshell.v1.ImportProviderProfilesResponse + 105, // 262: openshell.v1.OpenShell.UpdateProviderProfiles:output_type -> openshell.v1.UpdateProviderProfilesResponse + 107, // 263: openshell.v1.OpenShell.LintProviderProfiles:output_type -> openshell.v1.LintProviderProfilesResponse + 73, // 264: openshell.v1.OpenShell.UpdateProvider:output_type -> openshell.v1.ProviderResponse + 91, // 265: openshell.v1.OpenShell.GetProviderRefreshStatus:output_type -> openshell.v1.GetProviderRefreshStatusResponse + 93, // 266: openshell.v1.OpenShell.ConfigureProviderRefresh:output_type -> openshell.v1.ConfigureProviderRefreshResponse + 95, // 267: openshell.v1.OpenShell.RotateProviderCredential:output_type -> openshell.v1.RotateProviderCredentialResponse + 97, // 268: openshell.v1.OpenShell.DeleteProviderRefresh:output_type -> openshell.v1.DeleteProviderRefreshResponse + 108, // 269: openshell.v1.OpenShell.DeleteProvider:output_type -> openshell.v1.DeleteProviderResponse + 110, // 270: openshell.v1.OpenShell.DeleteProviderProfile:output_type -> openshell.v1.DeleteProviderProfileResponse + 240, // 271: openshell.v1.OpenShell.GetSandboxConfig:output_type -> openshell.sandbox.v1.GetSandboxConfigResponse + 241, // 272: openshell.v1.OpenShell.GetGatewayConfig:output_type -> openshell.sandbox.v1.GetGatewayConfigResponse + 125, // 273: openshell.v1.OpenShell.UpdateConfig:output_type -> openshell.v1.UpdateConfigResponse + 127, // 274: openshell.v1.OpenShell.GetSandboxPolicyStatus:output_type -> openshell.v1.GetSandboxPolicyStatusResponse + 129, // 275: openshell.v1.OpenShell.ListSandboxPolicies:output_type -> openshell.v1.ListSandboxPoliciesResponse + 131, // 276: openshell.v1.OpenShell.ReportPolicyStatus:output_type -> openshell.v1.ReportPolicyStatusResponse + 114, // 277: openshell.v1.OpenShell.GetSandboxProviderEnvironment:output_type -> openshell.v1.GetSandboxProviderEnvironmentResponse + 116, // 278: openshell.v1.OpenShell.ExchangeProviderSubjectToken:output_type -> openshell.v1.ExchangeProviderSubjectTokenResponse + 136, // 279: openshell.v1.OpenShell.GetSandboxLogs:output_type -> openshell.v1.GetSandboxLogsResponse + 135, // 280: openshell.v1.OpenShell.PushSandboxLogs:output_type -> openshell.v1.PushSandboxLogsResponse + 138, // 281: openshell.v1.OpenShell.ConnectSupervisor:output_type -> openshell.v1.GatewayMessage + 145, // 282: openshell.v1.OpenShell.ReportMainProcessExit:output_type -> openshell.v1.ReportMainProcessExitResponse + 150, // 283: openshell.v1.OpenShell.RelayStream:output_type -> openshell.v1.RelayFrame + 152, // 284: openshell.v1.OpenShell.PeerRelay:output_type -> openshell.v1.PeerRelayFrame + 65, // 285: openshell.v1.OpenShell.WatchSandbox:output_type -> openshell.v1.SandboxStreamEvent + 162, // 286: openshell.v1.OpenShell.SubmitPolicyAnalysis:output_type -> openshell.v1.SubmitPolicyAnalysisResponse + 164, // 287: openshell.v1.OpenShell.GetDraftPolicy:output_type -> openshell.v1.GetDraftPolicyResponse + 166, // 288: openshell.v1.OpenShell.ApproveDraftChunk:output_type -> openshell.v1.ApproveDraftChunkResponse + 168, // 289: openshell.v1.OpenShell.RejectDraftChunk:output_type -> openshell.v1.RejectDraftChunkResponse + 171, // 290: openshell.v1.OpenShell.ApproveAllDraftChunks:output_type -> openshell.v1.ApproveAllDraftChunksResponse + 173, // 291: openshell.v1.OpenShell.EditDraftChunk:output_type -> openshell.v1.EditDraftChunkResponse + 175, // 292: openshell.v1.OpenShell.UndoDraftChunk:output_type -> openshell.v1.UndoDraftChunkResponse + 177, // 293: openshell.v1.OpenShell.ClearDraftChunks:output_type -> openshell.v1.ClearDraftChunksResponse + 180, // 294: openshell.v1.OpenShell.GetDraftHistory:output_type -> openshell.v1.GetDraftHistoryResponse + 8, // 295: openshell.v1.OpenShell.IssueSandboxToken:output_type -> openshell.v1.IssueSandboxTokenResponse + 10, // 296: openshell.v1.OpenShell.RefreshSandboxToken:output_type -> openshell.v1.RefreshSandboxTokenResponse + 186, // 297: openshell.v1.OpenShell.CreateWorkspace:output_type -> openshell.v1.CreateWorkspaceResponse + 188, // 298: openshell.v1.OpenShell.GetWorkspace:output_type -> openshell.v1.GetWorkspaceResponse + 190, // 299: openshell.v1.OpenShell.ListWorkspaces:output_type -> openshell.v1.ListWorkspacesResponse + 192, // 300: openshell.v1.OpenShell.DeleteWorkspace:output_type -> openshell.v1.DeleteWorkspaceResponse + 195, // 301: openshell.v1.OpenShell.AddWorkspaceMember:output_type -> openshell.v1.AddWorkspaceMemberResponse + 197, // 302: openshell.v1.OpenShell.RemoveWorkspaceMember:output_type -> openshell.v1.RemoveWorkspaceMemberResponse + 199, // 303: openshell.v1.OpenShell.ListWorkspaceMembers:output_type -> openshell.v1.ListWorkspaceMembersResponse + 235, // [235:304] is the sub-list for method output_type + 166, // [166:235] is the sub-list for method input_type + 166, // [166:166] is the sub-list for extension type_name + 166, // [166:166] is the sub-list for extension extendee + 0, // [0:166] is the sub-list for field type_name } func init() { file_openshell_proto_init() } @@ -15699,15 +15877,19 @@ func file_openshell_proto_init() { (*RelayFrame_Init)(nil), (*RelayFrame_Data)(nil), } - file_openshell_proto_msgTypes[174].OneofWrappers = []any{} - file_openshell_proto_msgTypes[175].OneofWrappers = []any{} + file_openshell_proto_msgTypes[145].OneofWrappers = []any{ + (*PeerRelayFrame_Init)(nil), + (*PeerRelayFrame_Data)(nil), + } + file_openshell_proto_msgTypes[176].OneofWrappers = []any{} + file_openshell_proto_msgTypes[177].OneofWrappers = []any{} type x struct{} out := protoimpl.TypeBuilder{ File: protoimpl.DescBuilder{ GoPackagePath: reflect.TypeOf(x{}).PkgPath(), RawDescriptor: unsafe.Slice(unsafe.StringData(file_openshell_proto_rawDesc), len(file_openshell_proto_rawDesc)), NumEnums: 7, - NumMessages: 217, + NumMessages: 219, NumExtensions: 0, NumServices: 1, }, diff --git a/sdk/go/proto/openshellv1/openshell_grpc.pb.go b/sdk/go/proto/openshellv1/openshell_grpc.pb.go index 663c09aed5..fdfa20fd76 100644 --- a/sdk/go/proto/openshellv1/openshell_grpc.pb.go +++ b/sdk/go/proto/openshellv1/openshell_grpc.pb.go @@ -72,6 +72,7 @@ const ( OpenShell_ConnectSupervisor_FullMethodName = "/openshell.v1.OpenShell/ConnectSupervisor" OpenShell_ReportMainProcessExit_FullMethodName = "/openshell.v1.OpenShell/ReportMainProcessExit" OpenShell_RelayStream_FullMethodName = "/openshell.v1.OpenShell/RelayStream" + OpenShell_PeerRelay_FullMethodName = "/openshell.v1.OpenShell/PeerRelay" OpenShell_WatchSandbox_FullMethodName = "/openshell.v1.OpenShell/WatchSandbox" OpenShell_SubmitPolicyAnalysis_FullMethodName = "/openshell.v1.OpenShell/SubmitPolicyAnalysis" OpenShell_GetDraftPolicy_FullMethodName = "/openshell.v1.OpenShell/GetDraftPolicy" @@ -230,6 +231,14 @@ type OpenShellClient interface { // This rides the same TCP+TLS+HTTP/2 connection as ConnectSupervisor — // no new TLS handshake, no reverse HTTP CONNECT. RelayStream(ctx context.Context, opts ...grpc.CallOption) (grpc.BidiStreamingClient[RelayFrame, RelayFrame], error) + // Internal gateway-to-gateway relay forwarding. + // + // A gateway replica that receives a user request for a sandbox whose + // supervisor session is owned by a different replica opens this stream to the + // owner. The first frame carries PeerRelayInit; subsequent frames carry raw + // bytes in either direction. This RPC is authenticated as a gateway peer, not + // as a user or sandbox supervisor. + PeerRelay(ctx context.Context, opts ...grpc.CallOption) (grpc.BidiStreamingClient[PeerRelayFrame, PeerRelayFrame], error) // Watch a sandbox and stream updates. // // This stream can include: @@ -806,9 +815,22 @@ func (c *openShellClient) RelayStream(ctx context.Context, opts ...grpc.CallOpti // This type alias is provided for backwards compatibility with existing code that references the prior non-generic stream type by name. type OpenShell_RelayStreamClient = grpc.BidiStreamingClient[RelayFrame, RelayFrame] +func (c *openShellClient) PeerRelay(ctx context.Context, opts ...grpc.CallOption) (grpc.BidiStreamingClient[PeerRelayFrame, PeerRelayFrame], error) { + cOpts := append([]grpc.CallOption{grpc.StaticMethod()}, opts...) + stream, err := c.cc.NewStream(ctx, &OpenShell_ServiceDesc.Streams[6], OpenShell_PeerRelay_FullMethodName, cOpts...) + if err != nil { + return nil, err + } + x := &grpc.GenericClientStream[PeerRelayFrame, PeerRelayFrame]{ClientStream: stream} + return x, nil +} + +// This type alias is provided for backwards compatibility with existing code that references the prior non-generic stream type by name. +type OpenShell_PeerRelayClient = grpc.BidiStreamingClient[PeerRelayFrame, PeerRelayFrame] + func (c *openShellClient) WatchSandbox(ctx context.Context, in *WatchSandboxRequest, opts ...grpc.CallOption) (grpc.ServerStreamingClient[SandboxStreamEvent], error) { cOpts := append([]grpc.CallOption{grpc.StaticMethod()}, opts...) - stream, err := c.cc.NewStream(ctx, &OpenShell_ServiceDesc.Streams[6], OpenShell_WatchSandbox_FullMethodName, cOpts...) + stream, err := c.cc.NewStream(ctx, &OpenShell_ServiceDesc.Streams[7], OpenShell_WatchSandbox_FullMethodName, cOpts...) if err != nil { return nil, err } @@ -1142,6 +1164,14 @@ type OpenShellServer interface { // This rides the same TCP+TLS+HTTP/2 connection as ConnectSupervisor — // no new TLS handshake, no reverse HTTP CONNECT. RelayStream(grpc.BidiStreamingServer[RelayFrame, RelayFrame]) error + // Internal gateway-to-gateway relay forwarding. + // + // A gateway replica that receives a user request for a sandbox whose + // supervisor session is owned by a different replica opens this stream to the + // owner. The first frame carries PeerRelayInit; subsequent frames carry raw + // bytes in either direction. This RPC is authenticated as a gateway peer, not + // as a user or sandbox supervisor. + PeerRelay(grpc.BidiStreamingServer[PeerRelayFrame, PeerRelayFrame]) error // Watch a sandbox and stream updates. // // This stream can include: @@ -1351,6 +1381,9 @@ func (UnimplementedOpenShellServer) ReportMainProcessExit(context.Context, *Repo func (UnimplementedOpenShellServer) RelayStream(grpc.BidiStreamingServer[RelayFrame, RelayFrame]) error { return status.Error(codes.Unimplemented, "method RelayStream not implemented") } +func (UnimplementedOpenShellServer) PeerRelay(grpc.BidiStreamingServer[PeerRelayFrame, PeerRelayFrame]) error { + return status.Error(codes.Unimplemented, "method PeerRelay not implemented") +} func (UnimplementedOpenShellServer) WatchSandbox(*WatchSandboxRequest, grpc.ServerStreamingServer[SandboxStreamEvent]) error { return status.Error(codes.Unimplemented, "method WatchSandbox not implemented") } @@ -2249,6 +2282,13 @@ func _OpenShell_RelayStream_Handler(srv interface{}, stream grpc.ServerStream) e // This type alias is provided for backwards compatibility with existing code that references the prior non-generic stream type by name. type OpenShell_RelayStreamServer = grpc.BidiStreamingServer[RelayFrame, RelayFrame] +func _OpenShell_PeerRelay_Handler(srv interface{}, stream grpc.ServerStream) error { + return srv.(OpenShellServer).PeerRelay(&grpc.GenericServerStream[PeerRelayFrame, PeerRelayFrame]{ServerStream: stream}) +} + +// This type alias is provided for backwards compatibility with existing code that references the prior non-generic stream type by name. +type OpenShell_PeerRelayServer = grpc.BidiStreamingServer[PeerRelayFrame, PeerRelayFrame] + func _OpenShell_WatchSandbox_Handler(srv interface{}, stream grpc.ServerStream) error { m := new(WatchSandboxRequest) if err := stream.RecvMsg(m); err != nil { @@ -2871,6 +2911,12 @@ var OpenShell_ServiceDesc = grpc.ServiceDesc{ ServerStreams: true, ClientStreams: true, }, + { + StreamName: "PeerRelay", + Handler: _OpenShell_PeerRelay_Handler, + ServerStreams: true, + ClientStreams: true, + }, { StreamName: "WatchSandbox", Handler: _OpenShell_WatchSandbox_Handler, diff --git a/tasks/test.toml b/tasks/test.toml index a796ea67b4..643171fe78 100644 --- a/tasks/test.toml +++ b/tasks/test.toml @@ -165,9 +165,14 @@ description = "Run Kubernetes e2e with all database backend scenarios (SQLite an env = { OPENSHELL_E2E_KUBE_DB_SCENARIOS = "1" } run = "e2e/rust/e2e-kubernetes.sh" +["e2e:kubernetes:ha-rebalancing"] +description = "Run the full Kubernetes e2e suite through Envoy against two gateway replicas and external PostgreSQL" +env = { OPENSHELL_E2E_KUBE_EXTERNAL_POSTGRES_SECRET = "openshell-ha-pg", OPENSHELL_E2E_KUBE_EXTRA_VALUES = "deploy/helm/openshell/ci/values-high-availability.yaml", OPENSHELL_E2E_KUBE_FEATURES = "e2e,e2e-host-gateway,e2e-kubernetes,e2e-kubernetes-ha", OPENSHELL_E2E_KUBE_USE_ENVOY = "1" } +run = "e2e/rust/e2e-kubernetes.sh" + ["e2e:kubernetes:credential-drivers"] description = "Run Kubernetes e2e for provider credential storage backed by Kubernetes Secrets and Vault" -env = { OPENSHELL_E2E_CREDENTIAL_DRIVERS = "1", OPENSHELL_E2E_KUBE_TEST = "credential_drivers", OPENSHELL_E2E_KUBERNETES_FEATURES = "e2e,e2e-kubernetes,e2e-kubernetes-credential-drivers" } +env = { OPENSHELL_E2E_CREDENTIAL_DRIVERS = "1", OPENSHELL_E2E_KUBE_TEST = "credential_drivers", OPENSHELL_E2E_KUBE_FEATURES = "e2e,e2e-kubernetes,e2e-kubernetes-credential-drivers" } run = "e2e/rust/e2e-kubernetes.sh" ["e2e:kubernetes:workspace-managed"]