Skip to content

fix(otel): await server.stop(true) to release port in stopOtelReceiver - #1177

Merged
namastex888 merged 1 commit into
devfrom
fix/otel-receiver-flake
Apr 18, 2026
Merged

namastex888 merged 1 commit into
devfrom
fix/otel-receiver-flake

Conversation

@namastex888

Copy link
Copy Markdown
Contributor

Summary

Fixes the intermittent push-event CI failure on otel-receiver > POST /v1/logs handles empty payload.

Root cause: stopOtelReceiver() called server.stop() without awaiting the returned Promise. State was nulled synchronously, but the TCP port lingered. The next test's beforeEach picked a random port from the 7000-slot window (57000-63999); when the window was wide enough the previous suite's test and the next start raced on the same port, startOtelReceiver() hit EADDRINUSE, caught it as non-fatal, and silently returned false. isOtelReceiverRunning() stayed true (server variable still set from a prior run in pathological ordering), so downstream fetch() calls reached the wrong endpoint and returned unexpected status codes.

Fix:

  • stopOtelReceiver() is now async and awaits server.stop(true) (force close) — see src/lib/otel-receiver.ts:374-386.
  • src/genie.ts:700,711 callers now await stopOtelReceiver().catch(() => {}) in their finally blocks.
  • src/lib/otel-receiver.test.ts afterEach awaits stop.
  • Every await startOtelReceiver() call site gets expect(started).toBe(true) so future silent-false regressions surface immediately.

Test plan

  • bun test src/lib/otel-receiver.test.ts — 9/9 pass (5 consecutive runs)
  • bun test — 2581/2581 pass
  • bun run typecheck — clean
  • bun run lint — no new warnings on changed files
  • Pre-push hook passes

Context

Blocks the auto-version bump that publishes PR #1175 (topology fix for auto-team-of-one) to the @automagik/genie@next dist-tag. Once this merges and CI goes green, the publish pipeline resumes and the team-of-one-auto-create wish can complete G-C verification.

The push-event CI intermittently failed on `otel-receiver > POST /v1/logs
handles empty payload` with EADDRINUSE because `stopOtelReceiver()`
called `server.stop()` without awaiting the returned Promise. The server
state was nulled synchronously, but the underlying TCP port lingered.
When the next test's `beforeEach` picked a random port from the 7000-slot
window (57000-63999) and `startOtelReceiver()` raced against the unreleased
port, start silently returned false (caught as non-fatal EADDRINUSE),
leaving `isOtelReceiverRunning()` true but the server unreachable — so
downstream fetch() calls returned 0 or hit the wrong route.

Fix:
- Make `stopOtelReceiver()` async, await `server.stop(true)` (force close).
- Update the two callers in `src/genie.ts` (finally blocks) to await.
- Update `afterEach` in otel-receiver.test.ts to await stop.
- Add `expect(started).toBe(true)` assertions after every
  `await startOtelReceiver()` call so any future silent-false failure
  surfaces immediately instead of cascading into misleading HTTP errors.

Validated: 5 consecutive local runs of otel-receiver.test.ts pass; full
`bun test` (2581 tests) passes; `bun run typecheck` clean.
@coderabbitai

coderabbitai Bot commented Apr 18, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: b89cb03a-6a82-4226-a861-ffb8e44d04e2

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/otel-receiver-flake

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request converts stopOtelReceiver into an asynchronous function that awaits the server's shutdown, ensuring the listening port is fully released to prevent EADDRINUSE errors during tests. Corresponding updates were made to src/genie.ts and src/lib/otel-receiver.test.ts to await the function and handle its promise. Feedback was provided regarding a potential race condition in stopOtelReceiver, suggesting that the server reference should be nulled before awaiting the stop operation to ensure the system state is updated immediately.

Comment thread src/lib/otel-receiver.ts
Comment on lines +381 to 386
export async function stopOtelReceiver(): Promise<void> {
if (server) {
server.stop();
await server.stop(true);
server = null;
}
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

There is a potential race condition in the current implementation of stopOtelReceiver. Because server = null is executed after the await, any calls to isOtelReceiverRunning() or startOtelReceiver() that occur while the stop operation is pending will see the server as still active. This could lead to situations where the system incorrectly assumes the receiver is running when it is actually being shut down.

It is safer to capture the server reference and null the module-level variable immediately before awaiting the stop operation. This also makes the function naturally idempotent if called multiple times concurrently.

Suggested change
export async function stopOtelReceiver(): Promise<void> {
if (server) {
server.stop();
await server.stop(true);
server = null;
}
}
export async function stopOtelReceiver(): Promise<void> {
const s = server;
if (s) {
server = null;
await s.stop(true);
}
}

@namastex888
namastex888 merged commit 69df2eb into dev Apr 18, 2026
6 checks passed
@automagik-genie
automagik-genie deleted the fix/otel-receiver-flake branch September 25, 2026 04:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants