Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions docs/iroh-app-transport-architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -111,11 +111,11 @@ The fork must expose cancellation for an in-progress connect. Closing a QUIC con

`config/iroh/managed-relay-catalog.json` is the committed, server-owned source of truth for the managed fleet. `web/tools/generate-managed-iroh-relay-catalog.ts` validates it and writes the generated TypeScript consumed by the web API and presence worker. Build checks reject generated-file drift. Managed relay URLs do not come from deployment environment variables, and signing keys and relay credentials never enter the catalog or generated files.

Every catalog has a strictly increasing sequence and at most sixteen unique credential-free HTTPS origins. The backend rejects sequence rollback and same-sequence content changes. It signs a five-minute policy with an Ed25519 key whose public half is pinned by clients. A cached policy remains usable only until its signed expiry. Invalid, expired, rolled-back, or unverifiable policy fails closed to direct Iroh paths. Fleet rotations are add-before-remove: bump the sequence and add relays, regenerate and deploy both server consumers, wait at least one signed-policy lifetime, then bump the sequence again before regenerating, deploying, and removing the retired relays. A stable relay ID never changes meaning in place.
Every catalog has a strictly increasing sequence and at most sixteen unique credential-free HTTPS origins. The backend rejects sequence rollback and same-sequence content changes. It signs a one-hour policy with an Ed25519 key whose public half is pinned by clients. A cached policy remains usable only until its signed expiry. Invalid, expired, rolled-back, or unverifiable policy fails closed to direct Iroh paths. Fleet rotations are add-before-remove: bump the sequence and add relays, regenerate and deploy both server consumers, wait at least one signed-policy lifetime, then bump the sequence again before regenerating, deploying, and removing the retired relays. A stable relay ID never changes meaning in place.

The server may add, remove, or replace relays without a client update. A remote `EndpointAddr` contains only the remote endpoint's advertised home relay or relays, validated against the signed fleet. Fleet configuration and remote reachability remain separate wire fields.

A signed-in native client calls `POST /api/relay/token` with its canonical EndpointID. The web API returns a five-minute endpoint-bound relay JWT, the signed policy, and the account preference. Each cmux relay verifies its JWT offline. The app refreshes before expiry and replaces the verified relay policy on the live endpoint without changing EndpointID or application streams.
A signed-in native client calls `POST /api/relay/token` with its canonical EndpointID. The web API returns a one-hour endpoint-bound relay JWT, the signed policy, and the account preference. Each cmux relay verifies its JWT offline. The app refreshes before expiry and replaces the verified relay policy on the live endpoint without changing EndpointID or application streams.

Relay preferences are personal-account scoped:

Expand Down
2 changes: 1 addition & 1 deletion web/app/api/relay/token/route.ts
Original file line number Diff line number Diff line change
Expand Up @@ -214,7 +214,7 @@ export async function handleRelayTokenRequest(
const relayUrls = policy.payload.relays.map((relay) => relay.url);
// A fresh endpoint must fetch policy before registration, then fetch its
// bound credential immediately after registration. Renewals happen every
// four minutes because both artifacts expire after five. Give bootstrap
// ~55 minutes because both artifacts expire after sixty. Give bootstrap
// and credential issuance separate one-minute partitions so the external
// rule cannot make the valid two-leg bootstrap or renewal cadence
// impossible. Duplicate work inside one phase and minute is still bounded.
Expand Down
17 changes: 8 additions & 9 deletions web/services/iroh/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,15 +43,14 @@ limit while upgraded clients traverse every page. Sign-out callers must invoke
the authenticated revoke route with their captured binding id before discarding
the Stack credential.

There is no total active-binding limit per account or device. Postgres advisory
locks keep request-rate limits concurrency-safe: six challenges per device per
ten minutes, 32 outstanding challenges per account, 60 pair grants per account
per hour, three relay mints per endpoint per ten minutes, 12 relay mints per
endpoint per day, and 100 relay mints per account per day. A relay reservation
remains active for 60 seconds, then the next account-scoped reservation marks it
expired before applying those quotas. The optional Vercel Firewall rule is
defense in depth. A tagged-build override widens challenge issuance only after
an exact authenticated user-id and deployment-environment allowlist match.
There is no total active-binding limit per account or device. The broker's own
challenge, pair-grant, and relay-mint quotas were removed in
[#9269](https://github.com/manaflow-ai/cmux/pull/9269) after they locked
legitimate devices out of registration; auth and correctness guards (challenge
replay gates, binding-slot ownership, discovery pagination bounds) remain. A
relay reservation remains active for 60 seconds, then the next account-scoped
reservation marks it expired. The optional Vercel Firewall rules on the relay
routes are the only remaining rate limits.

Registration bootstraps a relay credential only when it creates a binding.
Signed refreshes of the same binding return `relay.status = "not_requested"`;
Expand Down
6 changes: 5 additions & 1 deletion web/services/relay/catalog.ts
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,11 @@ import {
MANAGED_IROH_RELAY_CATALOG,
} from "./generated/managedRelayCatalog";

export const RELAY_POLICY_TTL_SECONDS = 300;
// Matches RELAY_TOKEN_TTL_SECONDS: the policy ships in the same /api/relay/token
// response as the credential, and the client re-verifies the cached policy's exp
// on every load, so a shorter policy TTL would strand clients between refreshes.
// Raising this also raises the required add-before-remove rotation overlap below.
export const RELAY_POLICY_TTL_SECONDS = 3_600;
export const RELAY_ROTATION_MIN_OVERLAP_SECONDS = RELAY_POLICY_TTL_SECONDS;

export type RelayPolicySigningKey = {
Expand Down
6 changes: 4 additions & 2 deletions web/services/relay/token.ts
Original file line number Diff line number Diff line change
Expand Up @@ -12,8 +12,10 @@ import { configuredRelayCatalog } from "./catalog";

export const RELAY_TOKEN_ISS = "cmux";
export const RELAY_TOKEN_AUD = "cmux-relay";
export const RELAY_TOKEN_TTL_SECONDS = 300; // short-lived; the client refreshes
export const RELAY_TOKEN_REFRESH_LEAD_SECONDS = 60;
// One hour keeps the credential device-bound and revocable-by-rotation while
// cutting an always-on endpoint from ~288 refreshes/day (at 300 s) to 24.
export const RELAY_TOKEN_TTL_SECONDS = 3_600; // the client refreshes before expiry

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Preserve the legacy 300-second response contract

When a client predating relayCredentials support calls this endpoint, it ignores the new array and falls back to the still-emitted token/ttlSeconds fields. The current client fallback still requires ttlSeconds == 300, so changing this shared constant to 3600 makes those responses invalid and prevents older clients from obtaining relay connectivity. Fresh evidence in the current tree is the hard-coded legacy check in CmxIrohTrustBrokerClient.relayTokenResponse; retain the legacy TTL for that compatibility path or version/negotiate the contract before raising it.

Useful? React with 👍 / 👎.

export const RELAY_TOKEN_REFRESH_LEAD_SECONDS = 300;
Comment on lines +17 to +18

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Preserve the legacy 300-second response contract

When an installed client predating relayCredentials support calls this endpoint, its decoder ignores the new array and falls back to the legacy token/ttlSeconds fields that are still emitted by this route; CmxIrohTrustBrokerClient.relayTokenResponse currently requires ttlSeconds == 300 in that fallback path, so the new 3600 value makes the response invalid and prevents those clients from obtaining relay connectivity. Either retain the legacy TTL for that compatibility path or version/negotiate the contract before raising it.

Useful? React with 👍 / 👎.


export type ManagedRelayCredentialGrant = {
readonly relayUrl: string;
Expand Down
11 changes: 9 additions & 2 deletions web/tests/iroh-db-behavior.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ import {
accountDeletionUserHash,
} from "../services/account/deletionLock";
import type { PairGrantPeer } from "../services/iroh/crypto";
import { RELAY_ROTATION_MIN_OVERLAP_SECONDS } from "../services/relay/catalog";
import {
IROH_RETENTION_BATCH_SIZE,
IrohRepository,
Expand Down Expand Up @@ -118,12 +119,18 @@ describe("Iroh trust broker database behavior", () => {
await Effect.runPromise(acceptCatalog({ catalog: added, nowSeconds: 1_001 }));

const earlyRemoval = await Effect.runPromiseExit(
acceptCatalog({ catalog: removed, nowSeconds: 1_300 }),
acceptCatalog({
catalog: removed,
nowSeconds: 1_001 + RELAY_ROTATION_MIN_OVERLAP_SECONDS - 1,
}),
);
expect(earlyRemoval._tag).toBe("Failure");
expect(String(earlyRemoval)).toContain("unsafe_transition");

await Effect.runPromise(acceptCatalog({ catalog: removed, nowSeconds: 1_301 }));
await Effect.runPromise(acceptCatalog({
catalog: removed,
nowSeconds: 1_001 + RELAY_ROTATION_MIN_OVERLAP_SECONDS,
}));
const [state] = await requiredSql()<Array<{
sequence: string;
catalog: RelayCatalog;
Expand Down
2 changes: 1 addition & 1 deletion web/tests/relay-policy.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -230,7 +230,7 @@ describe("signed relay policy", () => {
sequence: 17,
iat: 1_700_000_000,
nbf: 1_699_999_995,
exp: 1_700_000_300,
exp: 1_700_003_600,
aud: RELAY_POLICY_AUDIENCE,
relay_protocol: RELAY_POLICY_PROTOCOL,
relays: catalog.relays,
Expand Down
14 changes: 7 additions & 7 deletions web/tests/relay-token-route.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ const PAYLOAD: RelayPolicyPayload = {
sequence: 4,
iat: 1_700_000_000,
nbf: 1_700_000_000,
exp: 1_700_000_300,
exp: 1_700_003_600,
aud: "cmux-iroh-relay-policy",
relay_protocol: "iroh-relay-v1",
relays: [{
Expand Down Expand Up @@ -103,9 +103,9 @@ describe("POST /api/relay/token", () => {
expect(body.relayCredentials).toEqual([{
relayUrl: "https://relay-one.cmux.dev/",
token: body.token,
expiresAt: 1_700_000_300,
refreshAfter: 1_700_000_240,
ttlSeconds: 300,
expiresAt: 1_700_003_600,
refreshAfter: 1_700_003_300,
ttlSeconds: 3_600,
Comment thread
coderabbitai[bot] marked this conversation as resolved.
}]);
expect(body.policy).toBe("signed.policy.value");
expect(body.preference).toEqual({
Expand All @@ -114,8 +114,8 @@ describe("POST /api/relay/token", () => {
customRelays: [],
});
expect(body.preferenceRevision).toBe(3);
expect(body.ttlSeconds).toBe(300);
expect(body.expiresAt).toBe(1_700_000_300);
expect(body.ttlSeconds).toBe(3_600);
expect(body.expiresAt).toBe(1_700_003_600);

const [header, payload, signature] = (body.token as string).split(".");
expect(edVerify(
Expand All @@ -129,7 +129,7 @@ describe("POST /api/relay/token", () => {
aud: "cmux-relay",
sub: "account-a",
iat: 1_700_000_000,
exp: 1_700_000_300,
exp: 1_700_003_600,
endpoint_id: ENDPOINT_ID,
});
});
Expand Down