Skip to content

Restore 55 Pulsar tests and skip only the four behaviours that are missing (GH-3763) - #3798

Merged
jeremydmiller merged 1 commit into
mainfrom
gh-3763/pulsar-flaky-tag-burndown
Aug 3, 2026
Merged

Restore 55 Pulsar tests and skip only the four behaviours that are missing (GH-3763)#3798
jeremydmiller merged 1 commit into
mainfrom
gh-3763/pulsar-flaky-tag-burndown

Conversation

@jeremydmiller

Copy link
Copy Markdown
Member

The Pulsar pass of #3763, after #3791 (AWS), #3794 (Kafka) and #3795 (RabbitMQ).

An exclusion with no recorded reason, and far too broad

Commit 02f531842 on 2026-06-08 — message "Disable Pulsar compliance tests", no other rationale — tagged all three Pulsar compliance classes Category=Flaky. That is 66 of the project's 224 tests; CIPulsar has been running 158.

Untagged: 213 pass, 11 fail — and the 11 are the same four behaviours repeated across the three fixtures.

Test Failure
will_requeue_and_increment_attempts Expected ending activity was not detected
can_schedule_retry Expected ending activity was not detected
will_move_to_dead_letter_queue_without_any_exception_match No ending activity detected
will_move_to_dead_letter_queue_with_exception_match No ending activity detected

They are not flaky — they fail deterministically, every run, alone or in a suite, and they look like genuinely unimplemented transport behaviour: native requeue with attempt tracking, scheduled retry, and dead-letter routing. Filed as #3797.

So: all three classes come off the ledger, only those four are skipped

Each skip carries the reason and the issue number, so what's missing is greppable instead of buried under three blanket exclusions. can_schedule_retry had to become virtual in TransportCompliance to be overridable; the other three already were.

with_cloud_events already opted out of will_move_to_dead_letter_queue_with_exception_match by overriding it with a bare return Task.CompletedTask — which reports as a PASS, so the suite was counting a test that ran nothing. That's now a real Skip carrying its original explanation.

Net: CIPulsar goes from 158 executing tests to 213, plus 11 named skips.

Two things worth knowing that came out of this

Verification

Result
CIPulsar, run 1 223 passed, 1 retry, exit 0
CIPulsar, run 2 224 passed, 0 retries, exit 0

Repo flaky tags 16 → 13. wolverine.slnx builds clean in Release.

🤖 Generated with Claude Code

https://claude.ai/code/session_0116vfBcKwcjWn8msM4ZjkuA

…ssing (GH-3763)

Commit 02f5318 on 2026-06-08 -- message "Disable Pulsar compliance tests", no
other rationale recorded -- tagged all three Pulsar compliance classes
Category=Flaky. That is 66 of the project's 224 tests; CIPulsar has been running
158.

Untagged, the exclusion turns out to be far too broad: 213 pass and 11 fail, and
the 11 are the SAME FOUR behaviours repeated across the three fixtures.

  will_requeue_and_increment_attempts                          Expected ending activity was not detected
  can_schedule_retry                                           Expected ending activity was not detected
  will_move_to_dead_letter_queue_without_any_exception_match    No ending activity detected
  will_move_to_dead_letter_queue_with_exception_match           No ending activity detected

They are not flaky. They fail deterministically, every run, alone or in a suite,
and they look like genuinely unimplemented transport behaviour: native requeue
with attempt tracking, scheduled retry, and dead-letter routing. Filed as #3797.

So all three classes come off the ledger and only those four are skipped, each
carrying the reason and the issue number. can_schedule_retry had to become
virtual in TransportCompliance to be overridable; the other three already were.

with_cloud_events already opted out of will_move_to_dead_letter_queue_with_
exception_match by overriding it with a bare `return Task.CompletedTask`, which
reports as a PASS -- the suite was counting a test that ran nothing. That is now
a real Skip carrying its original explanation.

Net: CIPulsar goes from 158 executing tests to 213, plus 11 named skips.

Two things worth knowing that came out of this:

- CIPulsar runs Pulsar in Testcontainers, not docker-compose (#3467). A local run
  hangs silently at "224 batched" if Docker is short on memory -- nine stale
  containers were holding 4.7 of 7.7 GiB here and no Pulsar container could
  start. Stopping them fixed it; nothing in the output said so.
- The supervisor's summary counts SKIPPED tests inside its "passed" number. Run 2
  reports "224 passed" for 213 executed + 11 skipped. The per-assembly runner
  reports them correctly (18 succeeded / 4 skipped for one fixture).

Verified: CIPulsar green twice -- 223 passed + 1 retry, then 224 passed with 0
retries. wolverine.slnx builds clean in Release. Repo flaky tags 16 -> 13.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0116vfBcKwcjWn8msM4ZjkuA
@jeremydmiller

Copy link
Copy Markdown
Member Author

Follow-ups filed out of this work, so the PR itself stays scoped to the tag burn-down:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant