feat: add upstream lifecycle controls

README.md

@@ -108,7 +108,7 @@ For account bundles, define `upstreams` instead of `upstream`. Tools, resource n
 
 Multi-upstream resource and prompt lists use opaque, bounded in-memory Miftah cursors so individual upstream cursors are not exposed. Re-list after a profile change or restart instead of retaining an old cursor. A standard `upstream` or a one-entry `upstreams` map preserves credential-free raw resource URIs, prompt names, and native upstream cursors for compatibility; URI and icon fields with userinfo, query values, or fragments are structurally redacted before they cross the boundary. A zero-entry bundle still omits resource and prompt capabilities.
 
-`tooling.toolDiscoveryMode` defaults to `"permissive"`. In that mode, a bundled list returns only healthy upstream contributions, removes routes for failed upstreams, and retries incomplete tool discovery on later tool requests. If every upstream fails a capability list, Miftah returns `UPSTREAM_DISCOVERY_FAILED` instead of advertising an empty upstream surface. Set it to `"strict"` to reject any unavailable upstream; strict tool discovery additionally checks every configured profile for an identical client-visible tool contract. `miftah_health` reports the profile, upstream name, overall and process state, transition time, restart count, redacted error, and per-capability discovery state for each started upstream. See `examples/multi-upstream.miftah.json`.
+`tooling.toolDiscoveryMode` defaults to `"permissive"`. In that mode, a bundled list returns only healthy upstream contributions, removes routes for failed upstreams, and retries incomplete tool discovery on later tool requests. If every upstream fails a capability list, Miftah returns `UPSTREAM_DISCOVERY_FAILED` instead of advertising an empty upstream surface. Set it to `"strict"` to reject any unavailable upstream; strict tool discovery additionally checks every configured profile for an identical client-visible tool contract. `miftah_health` reports the profile, upstream name, overall and process state, transition time, automatic restart count, redacted error, intentional stop reason, pending recovery time, and per-capability discovery state for each started upstream. See `examples/multi-upstream.miftah.json`.
 
 ## Routing and safety
 
@@ -167,7 +167,7 @@ Use `miftah doctor` to inspect config and upstream readiness without printing pr
 
 ## Current boundaries
 
-The current experimental code implements local STDIO and remote HTTP/SSE upstream clients, profile switching, hybrid routing rules, policies, namespaced tools/resources/prompts for account bundles, resilient healthy-upstream discovery, in-memory process/session caching, redacted JSONL audit logging, and a packageable CLI. Unsupported lifecycle tuning, persisted state, UI, routing plugins, profile matchers, and configurable tool namespaces are rejected with `UNSUPPORTED_CONFIG_OPTION` rather than silently ignored.
+The current experimental code implements local STDIO and remote HTTP/SSE upstream clients, profile switching, hybrid routing rules, policies, namespaced tools/resources/prompts for account bundles, resilient healthy-upstream discovery, configurable local process lifecycle controls, in-memory process/session caching, redacted JSONL audit logging, and a packageable CLI. Local process controls cover startup and shutdown deadlines, optional idle cleanup, opt-in crash recovery with a bounded retry budget, and no-eviction profile-session capacity limits. Persisted state, UI, routing plugins, profile matchers, `process.startMode`, `process.cache`, and configurable tool namespaces are rejected with `UNSUPPORTED_CONFIG_OPTION` rather than silently ignored.
 
 ## License
 

docs/architecture.md

@@ -12,7 +12,7 @@ MCP client
         -> upstream STDIO server
 ```
 
-The public server is built with the official `@modelcontextprotocol/sdk` `Server` and `StdioServerTransport`. Each profile gets an SDK `Client` and `StdioClientTransport` on first use. A single-flight start map prevents concurrent calls from launching duplicate processes. Tool discovery is held in immutable, per-profile capability snapshots; concurrent callers share one discovery operation. Complete snapshots are cached, while a permissive partial snapshot remains callable but is retried by the next tool list or call request. A call captures its active profile before discovery, so a later profile switch cannot change that call's active-profile fallback.
+The public server is built with the official `@modelcontextprotocol/sdk` `Server` and `StdioServerTransport`. Each profile gets an SDK `Client` and `StdioClientTransport` on first use. Per-profile start and restart single-flight state, attempt tokens, and lifecycle generations prevent concurrent calls or stale transport close events from creating duplicate children or replacing a newer session. Operations report begin/end activity so idle shutdown cannot close an in-flight request. Unexpected close is health `failed`; idle, explicit restart, wrapper shutdown, and timeout-enforced shutdown are intentional `stopped` states with a reason. Optional crash recovery uses bounded exponential backoff with jitter and a terminal retry budget; it retains its capacity reservation across the backoff window. A shared no-eviction limiter counts distinct profile bundles across named upstreams, so capacity pressure returns a typed refusal instead of terminating a live credential session. Tool discovery is held in immutable, per-profile capability snapshots; concurrent callers share one discovery operation. Complete snapshots are cached, while a permissive partial snapshot remains callable but is retried by the next tool list or call request. A call captures its active profile before discovery, so a later profile switch cannot change that call's active-profile fallback.
 
 The server advertises management tools plus tools discovered from the active profile. It advertises `tools.listChanged` and emits `notifications/tools/list_changed` after profile changes, restarts, and recovery that changes the public tool snapshot. Unknown names are rejected from the capability snapshot rather than guessed or forwarded. If a routing rule targets another profile, Miftah forwards only when that profile exposes the same name with an identical client-visible schema; otherwise it returns `TOOL_SCHEMA_MISMATCH`. In strict discovery mode, Miftah preflights every configured profile and rejects unavailable upstreams or any mismatched exposed tool contract before publishing a snapshot.
 

docs/config.md

@@ -25,7 +25,7 @@ A standard `upstream` and a one-entry `upstreams` map retain credential-free raw
 - `"permissive"` returns tools, resources, and prompts from healthy upstreams when a peer cannot start or list a capability. Failed resource and prompt routes are removed rather than remaining callable. Incomplete tool snapshots are retried by later tool list or call requests. If no configured upstream succeeds, the operation fails with `UPSTREAM_DISCOVERY_FAILED`.
 - `"strict"` rejects a capability list if any configured upstream is unavailable. For tools, it discovers every configured profile and requires the complete exposed tool names and client-visible schemas to match; an availability failure returns `UPSTREAM_DISCOVERY_FAILED` and a mismatch returns `TOOL_SCHEMA_MISMATCH`.
 
-Recovery that changes an aggregate surface emits the matching MCP list-change notification. `miftah_health` reports each started upstream with `upstreamName`, `profile`, overall `state`, `processState`, `lastTransition`, `restartCount`, redacted `error`, and `capabilities` for tools, resources, and prompts.
+Recovery that changes an aggregate surface emits the matching MCP list-change notification. `miftah_health` reports each started upstream with `upstreamName`, `profile`, overall `state`, `processState`, `lastTransition`, `restartCount`, redacted `error`, and `capabilities` for tools, resources, and prompts. Intentional stops include `lastStopReason`; pending automatic recovery includes `nextRestartAt`; exhausted recovery includes `restartLimitReached`.
 
 Before loading secret sources or starting an upstream, Miftah validates that `defaultProfile`, profile policy names, routing-rule profiles, `security.lockToProfile`, and per-profile upstream override names all exist. Validation errors are human-readable and expose `MiftahError.details.diagnostics` for programs; each diagnostic includes a stable `code`, dotted `path`, `severity`, `message`, and remediation.
 
@@ -50,6 +50,19 @@ Audit logging defaults to local JSONL when a path is configured. Arguments are e
 
 ## Runtime-supported controls
 
-Miftah rejects settings without a runtime implementation with `UNSUPPORTED_CONFIG_OPTION` and the exact config path. The only supported routing mode is `"hybrid"`; use `routing.rules` and `routing.fallback` to control its behavior. Routing plugins, profile metadata and matchers, persistent state, UI settings, configurable management-tool namespaces, and lifecycle tuning other than `process.startupTimeoutMs` are not available yet.
+Miftah rejects settings without a runtime implementation with `UNSUPPORTED_CONFIG_OPTION` and the exact config path. The only supported routing mode is `"hybrid"`; use `routing.rules` and `routing.fallback` to control its behavior. Routing plugins, profile metadata and matchers, persistent state, UI settings, and configurable management-tool namespaces are not available yet.
+
+### Process lifecycle
+
+`process` controls real local session behavior:
+
+- `startupTimeoutMs` is a positive integer and defaults to 30 seconds. A hung startup is terminated and reported as `UPSTREAM_START_FAILED`.
+- `shutdownTimeoutMs` is a positive integer and defaults to 5 seconds. Miftah first closes STDIO gracefully, then force-terminates a child that exceeds the deadline. Health records `lastStopReason: "shutdown-timeout"` rather than treating that intentional cleanup as a crash.
+- `idleTimeoutMs` is a positive integer. When omitted, sessions stay warm until an explicit restart or wrapper shutdown. When set, an inactive profile session closes after the timeout; in-flight upstream requests hold the session open.
+- `restartOnCrash` defaults to `false`. When `true`, only an unexpected upstream loss schedules automatic recovery; explicit restart, idle expiry, and wrapper shutdown do not.
+- `maxRestarts` is a non-negative integer and requires `restartOnCrash: true`. It defaults to 3 automatic attempts. Retry delay is bounded exponential backoff from 100 ms to 5 seconds with 20% jitter. A profile that keeps crashing exhausts its budget, reports `UPSTREAM_RESTART_LIMIT_EXCEEDED`, and rejects ordinary demand starts until `miftah_restart_profile` explicitly retries it. A recovered session must remain up for 30 seconds before its consecutive-crash budget resets.
+- `maxConcurrentProfiles` is a positive integer. It limits distinct live profile bundles across all named upstreams. Miftah never evicts a live credential session to make room: a new profile instead receives `UPSTREAM_CONCURRENCY_LIMIT`. All upstreams for one profile share one slot, and idle, final crash, failed start, or intentional close releases it.
+
+`startMode` and `cache` remain unsupported because Miftah currently always lazily creates cached sessions. They are rejected rather than silently ignored.
 
 Secret and audit redaction are force-on protections. `security.redactSecrets` and `audit.redact` may be declared as `true`, but setting either to `false` is rejected. Audit output is always JSONL; `audit.format` therefore accepts only `"jsonl"`.

src/cli/create-runtime.ts

@@ -55,12 +55,8 @@ export async function createRuntime(configPath: string) {
       }
     : undefined;
   const manager = resolvedConfig.upstreams
-    ? new MultiUpstreamProcessManager(resolvedConfig, {
-        startupTimeoutMs: config.process?.startupTimeoutMs
-      })
-    : new UpstreamProcessManager(upstream!, profiles, {
-        startupTimeoutMs: config.process?.startupTimeoutMs
-      });
+    ? new MultiUpstreamProcessManager(resolvedConfig, config.process)
+    : new UpstreamProcessManager(upstream!, profiles, config.process);
   const profileManager = new ProfileManager(resolvedConfig, resolvedConfig.security);
   return { config: resolvedConfig, manager, profileManager };
 }

src/config/schema.ts

@@ -129,17 +129,26 @@ const securitySchema = z
   })
   .strict();
 
-const publicProcessSchema = z.object({ startupTimeoutMs: z.number().int().positive().optional() }).strict();
+const publicProcessSchema = z
+  .object({
+    startupTimeoutMs: z.number().int().positive().optional(),
+    shutdownTimeoutMs: z.number().int().positive().optional(),
+    idleTimeoutMs: z.number().int().positive().optional(),
+    restartOnCrash: z.boolean().optional(),
+    maxRestarts: z.number().int().nonnegative().optional(),
+    maxConcurrentProfiles: z.number().int().positive().optional()
+  })
+  .strict();
 const processSchema = z
   .object({
     startMode: unsupportedOptionSchema,
     cache: unsupportedOptionSchema,
-    idleTimeoutMs: unsupportedOptionSchema,
-    restartOnCrash: unsupportedOptionSchema,
-    maxRestarts: unsupportedOptionSchema,
     startupTimeoutMs: z.number().int().positive().optional(),
-    shutdownTimeoutMs: unsupportedOptionSchema,
-    maxConcurrentProfiles: unsupportedOptionSchema
+    shutdownTimeoutMs: z.number().int().positive().optional(),
+    idleTimeoutMs: z.number().int().positive().optional(),
+    restartOnCrash: z.boolean().optional(),
+    maxRestarts: z.number().int().nonnegative().optional(),
+    maxConcurrentProfiles: z.number().int().positive().optional()
   })
   .strict();
 
@@ -400,22 +409,23 @@ export const miftahConfigSchema = z
       rejectUnsupportedOption(["routing", "plugins"], "routing plugins are not implemented");
     }
 
-    for (const option of [
-      "startMode",
-      "cache",
-      "idleTimeoutMs",
-      "restartOnCrash",
-      "maxRestarts",
-      "shutdownTimeoutMs",
-      "maxConcurrentProfiles"
-    ] as const) {
+    for (const option of ["startMode", "cache"] as const) {
       if (value.process?.[option] !== undefined) {
         rejectUnsupportedOption(
           ["process", option],
-          `${option} is not implemented; only process.startupTimeoutMs currently changes runtime behavior`
+          `${option} is not implemented; Miftah always starts cached upstream sessions lazily`
         );
       }
     }
+    if (value.process?.maxRestarts !== undefined && value.process.restartOnCrash !== true) {
+      addConfigIssue(
+        context,
+        "CONFIG_SCHEMA_INVALID",
+        ["process", "maxRestarts"],
+        "maxRestarts requires process.restartOnCrash to be true",
+        "Set process.restartOnCrash to true or remove process.maxRestarts."
+      );
+    }
 
     if (value.security?.requireProfileSwitchConfirmation !== undefined) {
       rejectUnsupportedOption(

src/config/types.ts

@@ -65,6 +65,11 @@ export interface SecurityConfig {
 
 export interface ProcessConfig {
   startupTimeoutMs?: number;
+  shutdownTimeoutMs?: number;
+  idleTimeoutMs?: number;
+  restartOnCrash?: boolean;
+  maxRestarts?: number;
+  maxConcurrentProfiles?: number;
 }
 
 export interface AuditConfig {

src/mcp/server/miftah-server.ts

@@ -98,7 +98,7 @@ export class MiftahServer {
   private readonly operationPipeline: OperationPipeline;
   private readonly resourcePromptRegistry?: ResourcePromptRegistry;
   private readonly invalidatedToolSnapshots = new Map<string, ToolSnapshot>();
-  private readonly restartingProfiles = new Set<string>();
+  private readonly restartingProfiles = new Map<string, Promise<void>>();
   private readonly pendingResourceListChanges = new Set<string>();
   private readonly pendingPromptListChanges = new Set<string>();
 
@@ -363,22 +363,7 @@ export class MiftahServer {
       }
       if (name === "miftah_restart_profile") {
         const profile = requiredString(args, "profile");
-        this.restartingProfiles.add(profile);
-        try {
-          if (this.upstreams instanceof MultiUpstreamProcessManager) {
-            await this.upstreams.restartProfile(profile);
-          } else {
-            await this.upstreams.restart(profile);
-          }
-        } finally {
-          this.restartingProfiles.delete(profile);
-          this.toolRegistry.invalidate(profile);
-          this.resourcePromptRegistry?.invalidate(profile);
-          if (profile === this.profiles.current().activeProfile) {
-            await this.notifyToolListChanged(undefined, undefined);
-            await this.notifyResourcePromptListChanged();
-          }
-        }
+        await this.restartUpstreamProfile(profile);
         return textResult("Profile restarted.");
       }
       if (name === "miftah_route_preview") {
@@ -735,6 +720,39 @@ export class MiftahServer {
     }
   }
 
+  private restartUpstreamProfile(profile: string): Promise<void> {
+    const existing = this.restartingProfiles.get(profile);
+    if (existing) return existing;
+    const restart = this.restartUpstreamProfileOnce(profile);
+    this.restartingProfiles.set(profile, restart);
+    void restart.then(
+      () => {
+        if (this.restartingProfiles.get(profile) === restart) this.restartingProfiles.delete(profile);
+      },
+      () => {
+        if (this.restartingProfiles.get(profile) === restart) this.restartingProfiles.delete(profile);
+      }
+    );
+    return restart;
+  }
+
+  private async restartUpstreamProfileOnce(profile: string): Promise<void> {
+    try {
+      if (this.upstreams instanceof MultiUpstreamProcessManager) {
+        await this.upstreams.restartProfile(profile);
+      } else {
+        await this.upstreams.restart(profile);
+      }
+    } finally {
+      this.toolRegistry.invalidate(profile);
+      this.resourcePromptRegistry?.invalidate(profile);
+      if (profile === this.profiles.current().activeProfile) {
+        await this.notifyToolListChanged(undefined, undefined);
+        await this.notifyResourcePromptListChanged();
+      }
+    }
+  }
+
   private handleUpstreamHealthChange(health: UpstreamHealth): void {
     if (health.processState !== "failed" || this.restartingProfiles.has(health.profile)) return;
     if (this.config.tooling?.toolDiscoveryMode === "strict") {

src/upstream/multi-upstream-process-manager.ts

@@ -6,17 +6,20 @@ import {
   type UpstreamHealth,
   type UpstreamManagerOptions
 } from "./upstream-process-manager.js";
+import { ProfileSessionLimiter } from "./profile-session-limiter.js";
 import { UpstreamSession } from "./upstream-session.js";
 import { MiftahError } from "../utils/errors.js";
 
 export class MultiUpstreamProcessManager {
   private readonly managers: Record<string, UpstreamProcessManager>;
   private readonly healthListeners = new Set<(health: UpstreamHealth) => void>();
+  private readonly limiter: ProfileSessionLimiter;
 
   constructor(config: MiftahConfig, options: UpstreamManagerOptions = {}) {
+    this.limiter = new ProfileSessionLimiter(options.maxConcurrentProfiles);
     this.managers = Object.fromEntries(
       Object.entries(config.upstreams ?? {}).map(([name, upstream]) => {
-        const manager = new UpstreamProcessManager(upstream, scopedProfiles(config.profiles, name), options, name);
+        const manager = new UpstreamProcessManager(upstream, scopedProfiles(config.profiles, name), options, name, this.limiter);
         manager.addHealthListener((health) => this.publishHealth(health));
         return [name, manager];
       })