Skip to content

docs(fleet): add remote access runbook - #1152

Merged
POWERFULMOVES merged 5 commits into
mainfrom
codex/agnote4482-fleet-remote-access-runbook
Mar 28, 2026
Merged

POWERFULMOVES merged 5 commits into
mainfrom
codex/agnote4482-fleet-remote-access-runbook

Conversation

@POWERFULMOVES

@POWERFULMOVES POWERFULMOVES commented Mar 28, 2026

Copy link
Copy Markdown
Owner

Summary

  • add a canonical fleet remote-access runbook covering Tailscale ACLs, RustDesk, CHIT enrollment, and z890 Codex/Claude shared ownership
  • document the TAILSCALE_API_KEY admin-secret contract and the Known Roads rule for translating raw rebuild manifests to make-target bring-up
  • scrub the examples so the PR avoids live tailnet identifiers, device IDs, IP addresses, and personal device names

Testing

  • git diff --check
  • diff-only privacy scan for IPv4 literals, device IDs, and email-like strings on added lines

Reviewer Notes

  • This is docs-only.
  • Live tailnet inventory should stay in the Tailscale admin console, local ignored exports, or Cipher; committed docs now use placeholders and role-based descriptions.
  • The current operational blocker is still documented: KVM2 watcher publish needs one NATS broker exposed on a Tailscale-reachable interface beyond the repo-default localhost-only bind.

Summary by CodeRabbit

  • New Features

    • Added fleet remote access control plane combining Tailscale ACL enforcement and RustDesk relay for secure remote operations.
    • Introduced automated stale node cleanup procedures via Tailscale Admin API.
    • Added fleet audit watcher with NATS event publishing for KVM2 infrastructure monitoring.
  • Documentation

    • Comprehensive operational runbooks for fleet remote access, RustDesk deployment, and node hygiene.
    • Updated reference guides with Tailscale-to-RustDesk workflow mappings and remote access procedures.
  • Chores

    • Added Tailscale API credential configuration options to environment setup.

@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

@coderabbitai

coderabbitai Bot commented Mar 28, 2026

Copy link
Copy Markdown
Contributor

Caution

Review failed

The head commit changed during the review from 37dcbd5 to 1084e81.

📝 Walkthrough

Walkthrough

This PR adds comprehensive documentation for fleet remote access infrastructure, establishing canonical runbooks and operational procedures for Tailscale ACL enforcement, RustDesk transport, and KVM2 watcher integration with NATS publishing. Additionally, it expands environment variable support for Tailscale API key credential handling across shared configuration.

Changes

Cohort / File(s) Summary
Fleet Remote Access Runbooks
pmoves/docs/operations/FLEET_REMOTE_ACCESS_RUNBOOK.md, pmoves/docs/operations/RUSTDESK_SELF_HOSTED.md
New comprehensive operator runbook (184 lines) defining end-to-end fleet control stack with Tailscale ACLs, RustDesk transport, secret separation, and Known Roads make targets. Enhanced existing RustDesk doc with KVM2 watcher setup, NATS CLI installation, and localhost-only 4222 constraints for audit event publishing.
Agent Notes & Coordination
pmoves/docs/AGENTS/AGNOTE4482PHI.t1.md, pmoves/docs/AGENTS/CODEX_CLAUDE_PARITY_MAP.md, pmoves/docs/AGENTS/CODEX_ECOSYSTEM_TRAVERSAL.md, pmoves/docs/AGENTS/CODEX_OPERATOR_HOME.md
Updated agent notes with "z890 Infra Context Pack" definitions, fleet remote access parity mappings, traversal heuristics for VPN/RDP tasks, and control-plane lane rules. Added timeline entries for 2026-03-28 with fleet documentation claims and z890 infra sync release. Included fleet checks commands (Tailscale status, health endpoints).
Plans & Documentation Index
pmoves/docs/PMOVES.AI PLANS/PMOVES_TAILSCALE_STATUS.md, pmoves/docs/PMOVES.AI PLANS/README_DOCS_INDEX.md
Updated Tailscale status document replacing placeholder text with implemented artifacts: captured ACL policy, runbooks, enrollment generator, watcher script, and systemd service. Added navigation links to new Fleet Remote Access and RustDesk Self-Hosted runbooks.
Secrets & Configuration Onboarding
.claude/CLAUDE.md, docs/SECRETS_ONBOARDING.md, pmoves/env.shared.example
Added Fleet Remote Access documentation section clarifying credential-role splitting (TAILSCALE_AUTHKEY vs TAILSCALE_API_KEY vs CHIT_PASSPHRASE) and Known Roads rebuild guidance. Updated SECRETS_ONBOARDING with TAILSCALE_API_KEY high-privilege storage constraints. Extended env.shared.example with TAILSCALE_API_KEY and TAILSCALE_API_KEY_FILE variables.
Node Hygiene & Maintenance
pmoves/docs/TAILSCALE_NODE_HYGIENE.md
Refreshed stale-node table with revised node classes and timings. Expanded cleanup procedures to include Admin API deletion alongside console removal with curl examples. Added guidance to exclude device IDs and live hostnames from committed docs (use Cipher instead). Replaced concrete node list reference with abstract node-class table.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~22 minutes

Possibly related PRs

Suggested labels

codex


🐰 A fleet takes shape with Tailscale's grace,
RustDesk relays guide the race,
KVM2 watchers keep their eyes,
NATS events reach toward the skies,
Runbooks written—operations rise!

🚥 Pre-merge checks | ✅ 2 | ❌ 1

❌ Failed checks (1 inconclusive)

Check name Status Explanation Resolution
Description check ❓ Inconclusive The PR description covers the key changes and testing performed, but is missing the structured sections (Summary bullets, Testing commands, Required Checks, Review Coordination) specified in the template. Restructure the description to follow the template format: use bullet points under Summary, provide formatted Testing section with commands, and include the Required Checks and Review Coordination checkboxes.
✅ Passed checks (2 passed)
Check name Status Explanation
Title check ✅ Passed The title 'docs(fleet): add remote access runbook' clearly and concisely summarizes the main change: adding fleet remote access documentation.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch codex/agnote4482-fleet-remote-access-runbook

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@kilo-code-bot

kilo-code-bot Bot commented Mar 28, 2026

Copy link
Copy Markdown
Contributor

Code Review Roast 🔥

Verdict: No Issues Found | Recommendation: Merge

Oh wait, this PR is still clean. I came back expecting to find something — anything — to roast in the incremental diff, and instead the author went and fixed all four review findings in two surgical commits. That's annoying. How am I supposed to maintain my curmudgeon credentials?

The Authorization: Bearer-u "${TAILSCALE_API_KEY}:" fix across both TAILSCALE_NODE_HYGIENE.md and FLEET_REMOTE_ACCESS_RUNBOOK.md is the right call — Tailscale access tokens use HTTP Basic auth, and the previous examples would have returned 401 Unauthorized. That was the critical finding, now properly addressed with both the corrected curl flags and an explicit auth-method note.

The new nats-subjects.md and services-catalog.md entries for the Fleet Audit Watcher are thorough and consistent with existing entries — three NATS subjects (fleet.device.registered.v1, fleet.audit.connection.v1, fleet.audit.heartbeat.v1), proper direction/purpose/reliability fields, and the services-catalog entry even documents the systemd health probe and the NATS reachability caveat. The Evidence: annotations added to PMOVES_TAILSCALE_STATUS.md address the minor finding about traceability.

The RUSTDESK_SELF_HOSTED.md update documenting the default NATS_URL value is a nice touch — operators can now see what they're overriding instead of guessing.

📊 Overall: Like watching someone actually read the review feedback and apply it correctly. Deeply unsettling. Ship it.

Files Reviewed (14 files, 2 incremental)

Original review (12 files):

  • .claude/CLAUDE.md — Fleet remote access section + Known Roads rule
  • docs/SECRETS_ONBOARDING.md — TAILSCALE_API_KEY secret contract
  • pmoves/docs/AGENTS/AGNOTE4482PHI.t1.md — z890 infra context pack
  • pmoves/docs/AGENTS/CODEX_CLAUDE_PARITY_MAP.md — Fleet API parity table
  • pmoves/docs/AGENTS/CODEX_ECOSYSTEM_TRAVERSAL.md — Fleet remote access traversal entry
  • pmoves/docs/AGENTS/CODEX_OPERATOR_HOME.md — Fleet remote access control plane
  • pmoves/docs/PMOVES.AI PLANS/PMOVES_TAILSCALE_STATUS.md — Status with evidence links
  • pmoves/docs/PMOVES.AI PLANS/README_DOCS_INDEX.md — Index entries for new runbooks
  • pmoves/docs/TAILSCALE_NODE_HYGIENE.md — Device name scrub + auth fix
  • pmoves/docs/operations/FLEET_REMOTE_ACCESS_RUNBOOK.md — Canonical runbook with auth fix
  • pmoves/docs/operations/RUSTDESK_SELF_HOSTED.md — Watcher install + default NATS_URL
  • pmoves/env.shared.example — TAILSCALE_API_KEY + _FILE entries

Incremental review (2 new files + 5 modified):

  • .claude/context/nats-subjects.md — Fleet audit watcher NATS subjects (new)
  • .claude/context/services-catalog.md — Fleet Audit Watcher service entry (new)
  • pmoves/docs/AGENTS/CODEX_ECOSYSTEM_TRAVERSAL.md — Line wrap fix
  • pmoves/docs/PMOVES.AI PLANS/PMOVES_TAILSCALE_STATUS.md — Evidence annotations
  • pmoves/docs/TAILSCALE_NODE_HYGIENE.md — Auth method fix
  • pmoves/docs/operations/FLEET_REMOTE_ACCESS_RUNBOOK.md — Auth method fix + related docs
  • pmoves/docs/operations/RUSTDESK_SELF_HOSTED.md — Default NATS_URL documentation

Reviewed by glm-5.1 · incremental from 1acaf221084e81


Reviewed by glm-5.1 · 341,216 tokens

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 4

🧹 Nitpick comments (2)
pmoves/docs/AGENTS/CODEX_ECOSYSTEM_TRAVERSAL.md (1)

167-167: Minor readability tweak recommended for the new heuristic line.

This bullet is a bit long/dense; splitting it into two shorter clauses will make scan-time faster in the heuristics block.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@pmoves/docs/AGENTS/CODEX_ECOSYSTEM_TRAVERSAL.md` at line 167, Split the long
heuristic bullet into two shorter clauses: first state the trigger and immediate
remediation steps ("If the task touches VPN mesh, remote desktop, stale-node
cleanup, or VPS onboarding, start with FLEET_REMOTE_ACCESS_RUNBOOK.md, then
RUSTDESK_SELF_HOSTED.md"), and then a separate clause that describes the final
administrative step ("Finish by signing the lane in AGNOTE4482PHI.t1.md").
Update the heuristic line referencing FLEET_REMOTE_ACCESS_RUNBOOK.md,
RUSTDESK_SELF_HOSTED.md, and AGNOTE4482PHI.t1.md so it becomes two concise
sentences or two bullets for improved readability.
pmoves/docs/operations/RUSTDESK_SELF_HOSTED.md (1)

194-199: Clarify NATS_URL configuration guidance.

Line 196 states "Set Environment=NATS_URL=..." but the systemd unit file already includes Environment=NATS_URL=nats://nats:pmoves@nats:4222 as a default (per pmoves/scripts/fleet/fleet-audit-watcher.service). Consider rephrasing to: "The service defaults to Environment=NATS_URL=nats://nats:pmoves@nats:4222; override if your NATS broker is at a different reachable address."

📝 Proposed clarification
 Operational notes:
 - Create `/var/log/pmoves` before starting the service. The systemd unit uses `ReadWritePaths=/var/log/pmoves`, and the service will fail early if the directory does not exist.
-- Set `Environment=NATS_URL=...` in `fleet-audit-watcher.service` to a broker reachable from KVM2.
+- The service defaults to `Environment=NATS_URL=nats://nats:pmoves@nats:4222`. Override this in `fleet-audit-watcher.service` if your NATS broker is at a different reachable address from KVM2.
 - The repo default NATS config binds port `4222` to localhost only, so the watcher cannot publish remotely until one PMOVES node exposes NATS on a Tailscale-reachable interface.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@pmoves/docs/operations/RUSTDESK_SELF_HOSTED.md` around lines 194 - 199,
Update the docs text to clarify that the systemd unit already sets a default
NATS_URL and instruct how to override it: state that the service defaults to
Environment=NATS_URL=nats://nats:pmoves@nats:4222 (as defined in
fleet-audit-watcher.service / pmoves/scripts/fleet/fleet-audit-watcher.service)
and only change the Environment value if your NATS broker is at a different
reachable address.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In @.claude/CLAUDE.md:
- Around line 267-279: Add the new fleet-audit-watcher NATS topology and service
entry: in .claude/context/nats-subjects.md document the watcher’s published
subjects (e.g., fleet.audit.watcher.events, fleet.audit.watcher.errors), message
direction (publisher: fleet-audit-watcher, subscribers/consumers), and any
QoS/retry semantics and required ACL tag usage (TAILSCALE_API_KEY vs
TAILSCALE_AUTHKEY); in .claude/context/services-catalog.md add a service entry
for fleet-audit-watcher including exposed port(s) if any or NATS client-only
note, required runtime deps (nats CLI, /var/log/pmoves, NATS broker reachable
via Tailscale), healthcheck endpoint or startup probe guidance, and operational
notes about the default NATS bind (localhost:4222) and implications for KVM2
reachability.

In `@pmoves/docs/operations/FLEET_REMOTE_ACCESS_RUNBOOK.md`:
- Around line 108-130: Update the documentation for the GET
/api/v2/tailnet/{tailnet}/acl entry in FLEET_REMOTE_ACCESS_RUNBOOK.md: verify
against Tailscale's public API whether GET on /api/v2/tailnet/{tailnet}/acl is
supported, and if it is not publicly documented remove the GET example or mark
it explicitly as "undocumented/private/internal" with guidance to avoid relying
on it; keep the POST/PUT examples for policy updates intact and reference the
exact endpoint string GET /api/v2/tailnet/{tailnet}/acl in the note so readers
can see which entry was verified/changed.

In `@pmoves/docs/PMOVES.AI` PLANS/PMOVES_TAILSCALE_STATUS.md:
- Around line 8-17: Update the "implemented" checklist lines to include explicit
evidence links or pointers: for the ACL policy reference the actual artifact
`pmoves/configs/tailscale-acl-policy.json` (and add a permalink or commit SHA),
for the remote-access runbooks add direct anchors or file links to
`../operations/FLEET_REMOTE_ACCESS_RUNBOOK.md`,
`../operations/RUSTDESK_SELF_HOSTED.md`, and `../TAILSCALE_NODE_HYGIENE.md`
showing the verification steps, and for enrollment/KVM2 watcher add links to the
smoke/log artifact or CI job that proves deployment (e.g. a permalink to
`pmoves/scripts/fleet/generate-enrollment.py` run output or the
`fleet-audit-watcher.sh` / `fleet-audit-watcher.service` logs); ensure each
implemented-status line includes the evidence link or path and a one-line
descriptor of what the evidence demonstrates.

In `@pmoves/docs/TAILSCALE_NODE_HYGIENE.md`:
- Around line 43-51: Update the curl examples in TAILSCALE_NODE_HYGIENE.md to
use HTTP Basic auth instead of a Bearer header: replace the Authorization header
lines currently showing -H "Authorization: Bearer $TAILSCALE_API_KEY" with
curl's -u 'tskey-<api-key>:' form for both the devices list request and the
DELETE request (the two curl blocks shown) so the API key is supplied as the
username per Tailscale API requirements.

---

Nitpick comments:
In `@pmoves/docs/AGENTS/CODEX_ECOSYSTEM_TRAVERSAL.md`:
- Line 167: Split the long heuristic bullet into two shorter clauses: first
state the trigger and immediate remediation steps ("If the task touches VPN
mesh, remote desktop, stale-node cleanup, or VPS onboarding, start with
FLEET_REMOTE_ACCESS_RUNBOOK.md, then RUSTDESK_SELF_HOSTED.md"), and then a
separate clause that describes the final administrative step ("Finish by signing
the lane in AGNOTE4482PHI.t1.md"). Update the heuristic line referencing
FLEET_REMOTE_ACCESS_RUNBOOK.md, RUSTDESK_SELF_HOSTED.md, and AGNOTE4482PHI.t1.md
so it becomes two concise sentences or two bullets for improved readability.

In `@pmoves/docs/operations/RUSTDESK_SELF_HOSTED.md`:
- Around line 194-199: Update the docs text to clarify that the systemd unit
already sets a default NATS_URL and instruct how to override it: state that the
service defaults to Environment=NATS_URL=nats://nats:pmoves@nats:4222 (as
defined in fleet-audit-watcher.service /
pmoves/scripts/fleet/fleet-audit-watcher.service) and only change the
Environment value if your NATS broker is at a different reachable address.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: d4b4f193-0900-4e72-ae27-55095b146ace

📥 Commits

Reviewing files that changed from the base of the PR and between 0fbfbaf and 1acaf22.

📒 Files selected for processing (12)
  • .claude/CLAUDE.md
  • docs/SECRETS_ONBOARDING.md
  • pmoves/docs/AGENTS/AGNOTE4482PHI.t1.md
  • pmoves/docs/AGENTS/CODEX_CLAUDE_PARITY_MAP.md
  • pmoves/docs/AGENTS/CODEX_ECOSYSTEM_TRAVERSAL.md
  • pmoves/docs/AGENTS/CODEX_OPERATOR_HOME.md
  • pmoves/docs/PMOVES.AI PLANS/PMOVES_TAILSCALE_STATUS.md
  • pmoves/docs/PMOVES.AI PLANS/README_DOCS_INDEX.md
  • pmoves/docs/TAILSCALE_NODE_HYGIENE.md
  • pmoves/docs/operations/FLEET_REMOTE_ACCESS_RUNBOOK.md
  • pmoves/docs/operations/RUSTDESK_SELF_HOSTED.md
  • pmoves/env.shared.example

Comment thread .claude/CLAUDE.md
Comment on lines +108 to +130
- `GET /api/v2/tailnet/{tailnet}/devices` — list devices (`devices:core:read`)
- `DELETE /api/v2/device/{deviceId}` — delete a device (`devices:core`)
- `GET /api/v2/tailnet/{tailnet}/acl` — fetch policy file (`policy_file:read`)
- `POST /api/v2/tailnet/{tailnet}/acl` — update policy file (`policy_file`)

`-` can be used as a shorthand tailnet ID when the API key belongs to the active tailnet.

Examples:

```bash
curl -fsS \
-H "Authorization: Bearer $TAILSCALE_API_KEY" \
"https://api.tailscale.com/api/v2/tailnet/-/devices"

curl -fsS -X DELETE \
-H "Authorization: Bearer $TAILSCALE_API_KEY" \
"https://api.tailscale.com/api/v2/device/<deviceId>"

curl -fsS \
-H "Authorization: Bearer $TAILSCALE_API_KEY" \
-H "Accept: application/hujson" \
"https://api.tailscale.com/api/v2/tailnet/-/acl"
```

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

🧩 Analysis chain

🌐 Web query:

What are the correct Tailscale API v2 endpoints for listing devices, deleting a device, and getting/updating ACL policy?

💡 Result:

The Tailscale API v2 endpoints are: - List devices: GET https://api.tailscale.com/api/v2/tailnet/{tailnet}/devices - Delete device: DELETE https://api.tailscale.com/api/v2/device/{deviceID} - Get ACL policy: No direct read endpoint found in public documentation; ACL policy is managed via the tailnet policy file through the admin console, GitOps, or potentially undocumented API (Pulumi provider suggests update-only). Update likely via POST/PUT to /api/v2/tailnet/{tailnet}/acl with policy JSON. Base URL: https://api.tailscale.com/api/v2/ Authentication: API key via HTTP Basic auth (tskey-KEY:). Official interactive docs: https://tailscale.com/api

Citations:


The GET /api/v2/tailnet/{tailnet}/acl endpoint appears to be undocumented in official Tailscale API documentation.

The list devices and delete device endpoints are confirmed correct. However, the GET ACL endpoint is not found in Tailscale's public API documentation—only POST/PUT for updates is documented. This suggests the endpoint may be private or undocumented. Verify that this endpoint actually exists and works for reading ACL policy before documenting it in an operational runbook, or clarify its status (internal API, undocumented, etc.).

🧰 Tools
🪛 LanguageTool

[style] ~113-~113: To form a complete sentence, be sure to include a subject.
Context: ...update policy file (policy_file) - can be used as a shorthand tailnet ID when ...

(MISSING_IT_THERE)

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@pmoves/docs/operations/FLEET_REMOTE_ACCESS_RUNBOOK.md` around lines 108 -
130, Update the documentation for the GET /api/v2/tailnet/{tailnet}/acl entry in
FLEET_REMOTE_ACCESS_RUNBOOK.md: verify against Tailscale's public API whether
GET on /api/v2/tailnet/{tailnet}/acl is supported, and if it is not publicly
documented remove the GET example or mark it explicitly as
"undocumented/private/internal" with guidance to avoid relying on it; keep the
POST/PUT examples for policy updates intact and reference the exact endpoint
string GET /api/v2/tailnet/{tailnet}/acl in the note so readers can see which
entry was verified/changed.

Comment thread pmoves/docs/PMOVES.AI PLANS/PMOVES_TAILSCALE_STATUS.md
Comment thread pmoves/docs/TAILSCALE_NODE_HYGIENE.md
@POWERFULMOVES

Copy link
Copy Markdown
Owner Author

Addressed the current docs review pass in 1084e818 and the preceding 37dcbd50.

What changed:

  • switched the Tailscale API examples in the runbook + node-hygiene doc from Authorization: Bearer ... to HTTP Basic auth via -u ${TAILSCALE_API_KEY}:
  • synced the canonical operator context by adding the fleet watcher subjects to .claude/context/nats-subjects.md and a Fleet Audit Watcher entry to .claude/context/services-catalog.md
  • added evidence pointers in PMOVES_TAILSCALE_STATUS.md
  • clarified the default NATS_URL behavior in RUSTDESK_SELF_HOSTED.md
  • split the long fleet heuristic in CODEX_ECOSYSTEM_TRAVERSAL.md for readability

Validation note:

  • I kept GET /api/v2/tailnet/{tailnet}/acl in the runbook because Tailscale's current trust-credentials reference documents it under policy_file:read, alongside POST /api/v2/tailnet/{tailnet}/acl under policy_file.

@POWERFULMOVES
POWERFULMOVES merged commit beb2e44 into main Mar 28, 2026
3 checks passed
@POWERFULMOVES
POWERFULMOVES deleted the codex/agnote4482-fleet-remote-access-runbook branch March 28, 2026 19:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant