Skip to content

GH-4100: bring the Pulsar reliability test's timeout back inside the CI job cap - #4101

Merged
jeremydmiller merged 1 commit into
mainfrom
gh-4100/pulsar-timeout-within-job-cap
Aug 24, 2026
Merged

GH-4100: bring the Pulsar reliability test's timeout back inside the CI job cap#4101
jeremydmiller merged 1 commit into
mainfrom
gh-4100/pulsar-timeout-within-job-cap

Conversation

@jeremydmiller

Copy link
Copy Markdown
Member

One character. Closes the first half of #4100.

PulsarNativeReliabilityTests.run_setup_with_simulated_exception_in_handler waited on its tracked
session for TimeSpan.FromSeconds(1000)16m40s, against this workflow's 20 minute
timeout-minutes cap.

That timeout can essentially never fire in CI

  • the job starts, so the cap lands 20 minutes later
  • CIPulsar spends ~6.5 minutes building and standing up the broker before the first test runs
  • a test starting then has ~13 minutes of budget, and 1000s does not fit

On run 32754218016 the arithmetic
was exact:

job started 17:15:32 → cap at 17:35:32
test started 17:22:11
its timeout would fire 17:38:51 — nearly 3 minutes after the job was already dead

Why that matters

The consequence is not a slow test. It is that when the underlying Pulsar flakiness hits — #4100:
seven producers and three consumers stalled in DotPulsar.Internal.Producer/Consumer.Setup, all 18
threads parked — the test cannot fail. The job runs to the cap and is cancelled instead, and a
cancelled job's logs are discarded outright (BlobNotFound, #4098), so the failure leaves no
evidence at all.

That has now happened twice, on unrelated branches, three seconds apart in duration (1218s / 1221s).

This PR does not fix the wedge. It converts a 20 minute silent cancellation into a 100 second
failure with a tracking dump — the difference between a bug that can be diagnosed and one that
cannot. The stall itself is the other half of #4100.

Why 100

  • every other test in the same file uses TimeSpan.FromSeconds(100) — lines 129, 187, 216 — for the
    same shape of assertion
  • the retry schedule under test is 4 + 2 + 3 seconds plus a dead-letter hop
  • it was the only TrackActivity(TimeSpan.FromSeconds(1000)) in the repository

Measured, not assumed

The risk in this change is that the test genuinely needs more than 100s. It does not — run against a
local broker on :6650:

run 1: passed (12s 874ms)
run 2: passed (12s 821ms)
run 3: passed (12s 837ms)

~12.8s, very tightly clustered, so 100s leaves roughly 8× headroom — and 1000s was ~78×.

Worth considering separately

Should any tracked-session timeout be allowed to exceed the job cap? One that does can only ever
produce a cancellation instead of a failure. This was the only 1000s case, but the general rule may
be worth enforcing. Raised in #4100, not addressed here.

Refs #4100. Refs #4098.

🤖 Generated with Claude Code

…CI job cap

run_setup_with_simulated_exception_in_handler waited on its tracked session for
TimeSpan.FromSeconds(1000) -- 16m40s, against this workflow's 20 minute
timeout-minutes cap. That timeout can essentially never fire in CI:

  * the job starts, so the cap lands 20 minutes later
  * CIPulsar spends ~6.5 minutes building and standing up the broker before the
    first test runs
  * so a test starting then would have to finish inside ~13 minutes, and 1000s
    does not fit

On run 32754218016 the arithmetic was exact: cap at 17:35:32, the test started
at 17:22:11, and its own timeout would have fired at 17:38:51 -- nearly three
minutes after the job was already dead.

The consequence is not a slow test. It is that when the underlying Pulsar
flakiness hits (GH-4100: producers and consumers stalling in
DotPulsar.Internal.Producer.Setup, every thread parked), the test cannot fail.
The job runs to the cap and is CANCELLED instead, and a cancelled job's logs are
discarded outright -- BlobNotFound, see GH-4098 -- so the failure leaves no
evidence at all. That has now happened twice, on unrelated branches.

100 seconds is what every other test in this file uses (lines 129, 187, 216) for
the same shape of assertion, and the retry schedule under test is 4 + 2 + 3
seconds plus a dead-letter hop. This was the only
TrackActivity(TimeSpan.FromSeconds(1000)) in the repository.

Measured against a local broker before changing it: the test passes in ~12.8s,
so 100s leaves roughly 8x headroom.

This does NOT fix the wedge. It converts a 20 minute silent cancellation into a
100 second failure with a tracking dump, which is the difference between a bug
that can be diagnosed and one that cannot.

Refs #4100. Refs #4098.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@jeremydmiller

Copy link
Copy Markdown
Member Author

All five runs finished after the PR was opened — adding the last two for completeness. Passes are unanimous and the spread is 66ms across the set:

run 1: passed (12s 874ms)
run 2: passed (12s 821ms)
run 3: passed (12s 837ms)
run 4: passed (12s 808ms)
run 5: passed (12s 808ms)

12.81–12.87s against a 100s budget — ~8x headroom, and none of the five came close to the flakiness described in #4100, so the local runs say nothing about how often the stall hits.

@jeremydmiller
jeremydmiller merged commit 5437c90 into main Aug 24, 2026
38 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant