feat(terminal): add /workspaces/:id/terminal/diagnose endpoint

[HongmingWang-Rabbit]

Summary

Adds GET /workspaces/:id/terminal/diagnose — a non-interactive HTTP/JSON sibling of /terminal that runs the same EIC + ssh stages and reports per-stage status. Built specifically because the canvas terminal currently fails silently ("Session ended" with no error frame on hongmingwang's hermes workspace) and there's no remote-readable signal of which stage broke — the ssh client's stderr lives only in the workspace-server's stdout on the tenant CP EC2.

The endpoint splits the failure modes that all look identical from the user's side:

• IAM broke → first_failure=\"send-ssh-public-key\", error contains AccessDeniedException
• Tunnel/SG broke → first_failure=\"wait-for-port\", detail contains the aws ec2-instance-connect open-tunnel stderr
• sshd auth broke → first_failure=\"ssh-probe\", detail contains Permission denied
• Shell or network broke → first_failure=\"ssh-probe\", detail contains Connection refused or timeout

Stages mirrored from handleRemoteConnect

1. ssh-keygen — ephemeral session keypair
2. send-ssh-public-key — AWS EIC API push (IAM-gated)
3. pick-free-port — local port for the tunnel
4. open-tunnel — aws ec2-instance-connect open-tunnel subprocess
5. wait-for-port — confirm tunnel actually binds (folds tunnel stderr into Detail when it doesn't)
6. ssh-probe — non-interactive `ssh ... 'echo MOLECULE_TERMINAL_PROBE_OK'` that proves auth + bash + round-trip via CombinedOutput

Local Docker workspaces get a smaller container-found + container-running probe, same response shape so callers don't need to branch.

Auth

Rides on existing WorkspaceAuth middleware. Operators with the tenant ADMIN_TOKEN (fetched via /cp/admin/orgs/:slug/admin-token) can probe any workspace without per-workspace token — same admin path the canvas dashboard uses.

Response shape

Always HTTP 200 (success or stage failure both in the JSON body — no need to branch on status):

```json
{
"workspace_id": "...",
"instance_id": "i-...",
"remote": true,
"ok": false,
"first_failure": "ssh-probe",
"steps": [
{"name": "ssh-keygen", "ok": true, "duration_ms": 12},
{"name": "send-ssh-public-key", "ok": true, "duration_ms": 234},
{"name": "pick-free-port", "ok": true, "duration_ms": 1, "detail": "port=12345"},
{"name": "open-tunnel", "ok": true, "duration_ms": 456},
{"name": "wait-for-port", "ok": true, "duration_ms": 789},
{"name": "ssh-probe", "ok": false, "duration_ms": 11000,
"error": "exit status 255",
"detail": "Permission denied (publickey)."}
]
}
```

Test plan

• `go test ./internal/handlers/ -run "TestHandleDiagnose|TestDiagnoseRemote" -v -count=1` — 3 new tests pass
• `go test ./internal/handlers/ -count=1` — full suite passes (3.9s, no regressions)
• `go vet ./...` clean
• `go build ./...` clean
• After merge: deploy to hongmingwang tenant + curl `/workspaces/{hermes-id}/terminal/diagnose` to pin the actual hermes failure stage

New tests

• `TestHandleDiagnose_RoutesToRemote` — instance_id present → remote path; `first_failure="send-ssh-public-key"` when stubbed to return AWS error
• `TestHandleDiagnose_RoutesToLocal` — empty instance_id → local path; `first_failure="docker-available"` with nil docker client
• `TestDiagnoseRemote_StopsAtSSHProbe` — full success through wait-for-port, probe stub returns "Permission denied"; confirms `first_failure="ssh-probe"` and the probe step's Detail/Error carries ssh stderr verbatim (the whole point of the endpoint)

Tests stub `sendSSHPublicKey`, `openTunnelCmd`, `sshProbeCmd` via existing package-level vars (same pattern as `TestSSHCommandCmd_*`) so the suite stays hermetic — no AWS calls, no network.

Related

• Resolves task fix(security): #190 — gate POST /templates/import behind AdminAuth #200
• Supports task fix(ci): set AGENT_TOOLSDIRECTORY on python-lint job (self-hosted runner) #193 (root cause: workspace EC2 sshd unresponsive — without this endpoint we couldn't pin the failure stage from outside the tenant CP EC2)
• Complements PR fix(terminal): cap ssh handshake at 10s so hung sshd surfaces fast #2434 (ConnectTimeout=10 symptom fix shipped 2026-04-30)

🤖 Generated with Claude Code