Skip to content

test: make automatic restart recovery deterministic #152

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
mohanagy merged 5 commits into from
Jul 21, 2026
Merged

test: make automatic restart recovery deterministic #152

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
c1d5e87
test: make automatic restart recovery deterministic
mohanagy Jul 21, 2026
408b0a5
Merge remote-tracking branch 'origin/development' into fix/123-determ…
mohanagy Jul 21, 2026
cbf6939
test: harden automatic recovery control
mohanagy Jul 21, 2026
fe7394b
test: remove timing race from recovery check
mohanagy Jul 21, 2026
6958b97
test: wait for restart entry before releasing gate
mohanagy Jul 21, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions tests/fixtures/fake-upstream.mjs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -68,6 +68,7 @@ const failOnRestartPath = process.env.TEST_FAIL_ON_RESTART_PATH;
const failListResourcesPath = process.env.TEST_FAIL_LIST_RESOURCES_PATH;
const failListPromptsPath = process.env.TEST_FAIL_LIST_PROMPTS_PATH;
const crashOnCallToolPath = process.env.TEST_CRASH_ON_CALL_TOOL_PATH;
const crashOnCallToolObservedPath = process.env.TEST_CRASH_ON_CALL_TOOL_OBSERVED_PATH;
const crashAfterInitializedPath = process.env.TEST_CRASH_AFTER_INITIALIZED_PATH;
const startCountPath = process.env.TEST_START_COUNT_PATH;
const initializedPath = process.env.TEST_INITIALIZED_PATH;
Expand Down Expand Up @@ -176,6 +177,9 @@ if (stderrMessage) {
}
}
if (crashOnCallToolPath && existsSync(crashOnCallToolPath)) {
if (crashOnCallToolObservedPath) {
writeFileSync(crashOnCallToolObservedPath, "observed");
}
throw new Error("test upstream configured to stay unavailable after an abrupt exit");
}
if (failOnRestartPath) {
Expand Down
25 changes: 21 additions & 4 deletions tests/upstream-manager.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -493,6 +493,9 @@ describe("upstream process manager", () => {
it("automatically recovers a crashed profile after a bounded backoff", async () => {
const directory = await mkdtemp(join(tmpdir(), "miftah-auto-restart-"));
const crashPath = join(directory, "crash");
const recoveryCrashObservedPath = join(directory, "recovery-crash-observed");
const restartGatePath = join(directory, "restart-gate");
const restartReadyPath = join(directory, "restart-ready");
const startCountPath = join(directory, "starts");
await writeFile(startCountPath, "");
const manager = new UpstreamProcessManager(
Expand All @@ -502,26 +505,33 @@ describe("upstream process manager", () => {
args: [fixture],
env: {
TEST_CRASH_ON_CALL_TOOL_PATH: crashPath,
TEST_CRASH_ON_CALL_TOOL_OBSERVED_PATH: recoveryCrashObservedPath,
TEST_HANG_ON_START_PATH: restartGatePath,
TEST_HANG_ON_START_READY_PATH: restartReadyPath,
TEST_START_COUNT_PATH: startCountPath
}
},
{ work: {} },
{ startupTimeoutMs: 1_000, restartOnCrash: true, maxRestarts: 2 }
{ startupTimeoutMs: 5_000, restartOnCrash: true, maxRestarts: 2 }
);

try {
const session = await manager.get("work");
await writeFile(crashPath, "crash");
await Promise.all([writeFile(crashPath, "crash"), writeFile(restartGatePath, "restart")]);
await expect(session.callTool({ name: "whoami", arguments: {} })).rejects.toThrow();
expect(existsSync(recoveryCrashObservedPath)).toBe(false);
await waitFor(() => countStarts(startCountPath), (count) => count === 2);
await waitFor(() => existsSync(restartReadyPath), Boolean);
await unlink(crashPath);
await unlink(restartGatePath);
Comment thread
coderabbitai[bot] marked this conversation as resolved.

await waitFor(() => countStarts(startCountPath), (count) => count === 2);
const recovered = await waitFor(
() => manager.listHealth().find((health) => health.profile === "work"),
(health) => health?.processState === "running"
);
if (!recovered) throw new Error("Expected recovered health");
expect(recovered.restartCount).toBe(1);
expect(existsSync(recoveryCrashObservedPath)).toBe(false);
await expect((await manager.get("work")).callTool({ name: "whoami", arguments: {} })).resolves.toMatchObject({
content: [{ type: "text", text: "unknown" }]
});
Expand Down Expand Up @@ -601,6 +611,7 @@ describe("upstream process manager", () => {
it("releases a profile capacity reservation after a failed startup", async () => {
const directory = await mkdtemp(join(tmpdir(), "miftah-failed-start-capacity-"));
const failurePath = join(directory, "fail");
const crashObservedPath = join(directory, "crash-observed");
await writeFile(failurePath, "fail");
const manager = new UpstreamProcessManager(
{
Expand All @@ -609,14 +620,20 @@ describe("upstream process manager", () => {
args: [fixture]
},
{
work: { env: { TEST_CRASH_ON_CALL_TOOL_PATH: failurePath } },
work: {
env: {
TEST_CRASH_ON_CALL_TOOL_PATH: failurePath,
TEST_CRASH_ON_CALL_TOOL_OBSERVED_PATH: crashObservedPath
}
},
personal: {}
},
{ startupTimeoutMs: 1_000, maxConcurrentProfiles: 1 }
);

try {
await expect(manager.get("work")).rejects.toMatchObject({ code: "UPSTREAM_INIT_FAILED" });
expect(existsSync(crashObservedPath)).toBe(true);
await expect((await manager.get("personal")).listTools()).resolves.toMatchObject({ tools: expect.any(Array) });
} finally {
await manager.close();
Expand Down