feat(agent-server): sanitized product-analytics telemetry with split consent policy

openhands-agent-server/openhands/agent_server/README.md

@@ -55,6 +55,12 @@ The server can be configured using environment variables or a JSON configuration
 | `SESSION_API_KEY` | API key for authentication (optional) | None |
 | `OH_SECRET_KEY` | Secret key for encrypting sensitive data (LLM API keys, secrets) in stored conversations. **Required for persistence across restarts.** | None |
 | `OH_ALLOW_CORS_ORIGIN_REGEX` | Regular expression for additional allowed CORS origins. Use `https?://.+` to allow any HTTP(S) origin while echoing the concrete origin. | None |
+| `OH_TELEMETRY_MODE` | Product-analytics policy: `disabled`, `local_opt_in`, or `cloud_locked`. See [Telemetry](#telemetry). | `disabled` |
+| `OH_TELEMETRY_POSTHOG_API_KEY` | PostHog project API key. Without it telemetry stays inactive in any mode. | None |
+| `OH_TELEMETRY_POSTHOG_HOST` | PostHog ingestion host. | `https://us.i.posthog.com` |
+| `OH_TELEMETRY_SALT` | Key used to pseudonymize conversation ids. Falls back to `OH_SECRET_KEY`, then to a per-process random salt. | None |
+| `DO_NOT_TRACK` | Set to `1` to force telemetry off, **overriding every mode including `cloud_locked`**. | Unset |
+| `OH_TELEMETRY_DISABLED` | Project-specific alias for `DO_NOT_TRACK`. | Unset |
 
 ### Configuration File
 
@@ -90,6 +96,101 @@ Create a JSON configuration file (default: `workspace/openhands_agent_server_con
 
 **Note**: Directory configuration (`working_dir`) will be handled at the conversation level rather than globally. These directories are specified when starting a conversation through the API.
 
+### Telemetry
+
+The agent server can emit a small set of **product-analytics** events to PostHog.
+It is **disabled by default** and does nothing unless a deployment opts in.
+
+#### This is not the same as the other two observability features
+
+| Feature | What it collects | Where it goes |
+|---|---|---|
+| **Telemetry** (this section) | Allowlisted lifecycle/failure events. Never prompts, messages, file contents, paths, secrets, request/response bodies, or tracebacks. | PostHog, when configured |
+| **LLM completion logging** (`log_completions`) | Full prompts, responses, and raw provider payloads — deliberately high fidelity for debugging. | Local disk only |
+| **Laminar / OpenTelemetry tracing** | Distributed traces and spans for latency analysis. | Your OTel/Laminar backend |
+
+Nothing from completion logging or tracing is ever forwarded to telemetry.
+
+#### Modes
+
+`OH_TELEMETRY_MODE` selects the deployment's policy:
+
+- **`disabled`** (default) — never emits. This is what library and headless
+  consumers get, and requires no PostHog dependency.
+- **`local_opt_in`** — emits only after the user explicitly consents through
+  `PUT /api/telemetry/consent`. Nothing is buffered before consent, so granting
+  it later cannot retroactively ship earlier activity. Revoking stops delivery
+  immediately and **discards** anything already queued.
+- **`cloud_locked`** — always emits; user consent is recorded but does not
+  change delivery. Intended for hosted deployments where analytics is part of
+  the service.
+
+`DO_NOT_TRACK=1` (or `OH_TELEMETRY_DISABLED=1`) forces telemetry off in **all**
+modes, including `cloud_locked`. This is an operator-level break-glass, not a
+user-level setting.
+
+#### Consent API
+
+```
+GET  /api/telemetry/consent
+PUT  /api/telemetry/consent   {"consent": "granted" | "denied" | "unset"}
+```
+
+Both return the resolved state, including `effective_enabled` and `is_locked`
+so a UI can render "managed by your administrator" without special-casing an
+error response. Consent is stored as a typed field in persisted settings
+(schema v3) — deliberately **not** inside the opaque `misc_settings` container,
+which the server documents as never interpreting.
+
+#### What is sent
+
+Events: `conversation_started`, `conversation_finished`, `conversation_failed`,
+`conversation_error`, `request_failed`, `server_started`, `server_stopped` — all
+prefixed `agent_server.`.
+
+Properties are limited to:
+
+- **Envelope** — `schema_version`, `source`.
+- **Release / runtime** — `server_version`, `sdk_version`, `tools_version`,
+  `build_git_sha`, `build_git_ref`, `python_version`, `platform`,
+  `deployment_mode`, `deferred_init`.
+- **Conversation shape** — `conversation_ref`, `llm_model_family`, `agent_kind`,
+  `tool_count`, `is_fork`, `has_agent_profile`, `workspace_kind`,
+  `confirmation_policy`.
+- **Outcome (bucketed)** — `terminal_status`, `duration_bucket`,
+  `event_count_bucket`, `total_tokens_bucket`, `cost_bucket`.
+- **Failure** — `error_class`, `error_category`, `error_fingerprint`,
+  `error_origin_module`, `error_origin_lineno`, `is_first_party`,
+  `is_terminal`, `tool_name`, `error_id`.
+- **Request failure** — `route_template` (the parametrised route, never the
+  concrete path), `method`, `status_code`.
+
+Magnitudes are bucketed rather than exact, because a raw count joined with a
+timestamp is a re-identification vector. `conversation_ref` is a keyed digest,
+never the raw conversation UUID. Exception *messages* and tracebacks are never
+read at all — failures are grouped by a fingerprint computed from the exception
+type and first-party module/line pairs.
+
+#### Identity
+
+Events use the deployment-supplied `user_id` verbatim as the PostHog
+`distinct_id`, so they attach to the person your product already identified.
+The exporter only ever calls `capture()` — never `identify()`, `alias()`, or
+`group_identify()` — so it cannot create a duplicate identity or irreversibly
+merge two. Without a `user_id`, an in-memory `anon:<hex>` id is used that resets
+on restart and is flagged so PostHog creates no person profile.
+
+#### Installation
+
+The PostHog client is an optional extra:
+
+```bash
+pip install 'openhands-agent-server[posthog]'
+```
+
+Without it, telemetry logs one warning and stays inactive; the server starts
+normally.
+
 ### Secret Encryption
 
 The server encrypts sensitive data (such as LLM API keys and conversation secrets) when storing conversations to disk. To enable this encryption and ensure secrets persist across server restarts, you **must** set the `OH_SECRET_KEY` environment variable.

openhands-agent-server/openhands/agent_server/api.py

@@ -61,6 +61,20 @@
 from openhands.agent_server.skills_router import skills_router
 from openhands.agent_server.sockets import sockets_router
 from openhands.agent_server.sub_agents_router import sub_agents_router
+from openhands.agent_server.telemetry import (
+    build_telemetry_sink,
+    emit_server_started,
+    emit_server_stopped,
+    get_event_factory,
+    get_telemetry_sink,
+    shutdown_telemetry_sink,
+    telemetry_router,
+)
+from openhands.agent_server.telemetry.models import (
+    REQUEST_FAILED,
+    RequestFailedProperties,
+)
+from openhands.agent_server.telemetry.sanitizer import normalize_exception
 from openhands.agent_server.tool_preload_service import get_tool_preload_service
 from openhands.agent_server.tool_router import tool_router
 from openhands.agent_server.vscode_router import vscode_router
@@ -133,6 +147,13 @@ async def api_lifespan(api: FastAPI) -> AsyncIterator[None]:
 
         config: Config = api.state.config
         deferred = config.deferred_init
+
+        # Deferred pods boot with telemetry disabled and are rebuilt by
+        # InitService, so they emit `server_started` there instead.
+        api.state.telemetry_sink = await build_telemetry_sink(config)
+        if not deferred:
+            emit_server_started()
+
         vscode_service = get_vscode_service()
         desktop_service = get_desktop_service()
         tool_preload_service = get_tool_preload_service()
@@ -265,12 +286,63 @@ async def stop_tool_preload_service():
 
                 await stop_stateless_services()
     finally:
+        # Outer finally so a startup failure cannot leak the drain task, and
+        # after `async with service` so terminal events are still accepted.
+        emit_server_stopped()
+        await shutdown_telemetry_sink()
+
         if tmux_tmpdir_was_defaulted and os.environ.get("TMUX_TMPDIR") == str(
             tmux_tmpdir
         ):
             os.environ.pop("TMUX_TMPDIR", None)
 
 
+def _emit_request_failed(request: Request, exc: Exception, error_id: str) -> None:
+    """Report an unhandled 5xx as a sanitized diagnostic event.
+
+    Sends the *route template* (``/api/conversations/{conversation_id}``)
+    rather than ``request.url.path``, which embeds real identifiers. Fully
+    defensive: an error in the telemetry path must not replace the 500 the
+    caller is already getting with a different failure.
+    """
+    try:
+        sink = get_telemetry_sink()
+        if not sink.enabled:
+            return
+
+        factory = get_event_factory()
+        if factory is None:
+            return
+
+        route = request.scope.get("route")
+        route_template = getattr(route, "path", None)
+        if not isinstance(route_template, str) or not route_template:
+            # Unmatched route: reporting the raw path could leak identifiers.
+            route_template = "/unmatched"
+
+        method = request.method.upper()
+        if method not in ("GET", "POST", "PUT", "PATCH", "DELETE", "HEAD", "OPTIONS"):
+            return
+
+        fingerprint = normalize_exception(exc)
+        sink.emit(
+            factory.build(
+                REQUEST_FAILED,
+                RequestFailedProperties(
+                    route_template=route_template,
+                    method=method,  # type: ignore[arg-type]
+                    status_code=500,
+                    error_class=fingerprint.error_class,
+                    error_category=fingerprint.error_category,
+                    error_fingerprint=fingerprint.error_fingerprint,
+                    error_id=error_id,
+                ),
+            )
+        )
+    except Exception:
+        logger.debug("Could not emit request_failed telemetry", exc_info=True)
+
+
 def _get_root_path(config: Config) -> str:
     root_path = ""
     if config.web_url:
@@ -356,6 +428,7 @@ def _add_api_routes(app: FastAPI) -> None:
     api_router.include_router(llm_router)
     api_router.include_router(mcp_router)
     api_router.include_router(settings_router)
+    api_router.include_router(telemetry_router)
     api_router.include_router(workspaces_router)
     api_router.include_router(profiles_router)
     api_router.include_router(agent_profiles_router)
@@ -528,6 +601,7 @@ async def _unhandled_exception_handler(
             error_id,
             exc_info=(type(exc), exc, exc.__traceback__),
         )
+        _emit_request_failed(request, exc, error_id)
         return JSONResponse(status_code=500, content=content)
 
     @api.exception_handler(HTTPException)

openhands-agent-server/openhands/agent_server/config.py

@@ -2,7 +2,7 @@
 import logging
 import os
 from pathlib import Path
-from typing import Any, ClassVar
+from typing import Any, ClassVar, Literal
 
 from pydantic import BaseModel, ConfigDict, Field, SecretStr
 
@@ -111,6 +111,75 @@ class WebhookSpec(BaseModel):
     )
 
 
+class TelemetrySpec(BaseModel):
+    """Deployment-supplied product-analytics policy and transport settings.
+
+    This is the *deployment's* half of the telemetry policy; the user's half is
+    ``PersistedSettings.telemetry_consent``. Splitting them means a hosted
+    deployment can require telemetry without that requirement being defeatable
+    by editing a settings file, while a local installation stays strictly
+    opt-in.
+
+    Note that consent is deliberately **not** stored in the opaque
+    ``misc_settings`` container, which the agent-server documents as never
+    interpreting.
+    """
+
+    mode: Literal["cloud_locked", "local_opt_in", "disabled"] = Field(
+        default="disabled",
+        description=(
+            "Telemetry policy for this deployment. 'disabled' (the default) "
+            "never emits, and is what library and headless consumers get. "
+            "'local_opt_in' emits only after explicit consent via "
+            "PUT /api/telemetry/consent. 'cloud_locked' always emits and "
+            "ignores consent. The DO_NOT_TRACK / OH_TELEMETRY_DISABLED "
+            "environment variables override all three."
+        ),
+    )
+    posthog_api_key: SecretStr | None = Field(
+        default=None,
+        description=(
+            "PostHog project API key. When unset, no exporter is constructed "
+            "and telemetry stays a no-op regardless of mode."
+        ),
+    )
+    posthog_host: str = Field(
+        default="https://us.i.posthog.com",
+        description="PostHog ingestion host.",
+    )
+    salt: SecretStr | None = Field(
+        default=None,
+        description=(
+            "Key used to pseudonymize conversation identifiers. Falls back to "
+            "a per-process random salt, which keeps pseudonyms stable within a "
+            "run but unlinkable across runs."
+        ),
+    )
+    max_queue_size: int = Field(
+        default=1000,
+        ge=1,
+        description=(
+            "Upper bound on buffered diagnostic events. The queue is bounded "
+            "on ingest: past this many events the oldest are dropped, so a "
+            "failing exporter cannot grow memory without limit."
+        ),
+    )
+    event_buffer_size: int = Field(
+        default=20, ge=1, description="Maximum events per delivery batch."
+    )
+    flush_delay: float = Field(
+        default=30.0,
+        gt=0,
+        description="Seconds to wait before flushing a partial batch.",
+    )
+    num_retries: int = Field(
+        default=2, ge=0, description="Retries before a batch is dropped."
+    )
+    retry_delay: float = Field(
+        default=5.0, ge=0, description="Base seconds between delivery retries."
+    )
+
+
 class Config(BaseModel):
     """
     Immutable configuration for a server running in local mode.
@@ -270,6 +339,13 @@ class Config(BaseModel):
             "to expire before the first renewal, effectively making it one-shot."
         ),
     )
+    telemetry: TelemetrySpec = Field(
+        default_factory=TelemetrySpec,
+        description=(
+            "Product-analytics policy. Disabled by default; see TelemetrySpec. "
+            "Distinct from LLM completion logging and from Laminar/OTel tracing."
+        ),
+    )
     model_config: ClassVar[ConfigDict] = {"frozen": True}
 
     @property