Skip to content

ci(test-windows): retry the transient ChromaDB HNSW compaction flake - #1854

Merged
igorls merged 1 commit into
developfrom
ci/windows-chromadb-flake-retries
Jun 22, 2026
Merged

ci(test-windows): retry the transient ChromaDB HNSW compaction flake#1854
igorls merged 1 commit into
developfrom
ci/windows-chromadb-flake-retries

Conversation

@igorls

@igorls igorls commented Jun 22, 2026

Copy link
Copy Markdown
Member

Problem

test-windows goes red at random. The failure is always the same family —
ChromaDB's rust HNSW core failing compaction on Windows:

chromadb.errors.InternalError: Error in compaction: Failed to apply logs to the hnsw segment writer

It is not caused by our code: migrate.py / the affected tests are
unchanged across the runs, it hits different tests (test_migrate_wings,
test_closets) on unrelated commits, and it does not reproduce on
Linux/macOS. Across a single batch of merges this week roughly half the
test-windows runs passed and half failed on the same code. CLAUDE.md already
documents Windows ChromaDB instability (the Windows coverage threshold is
lowered for the same reason). It has been turning otherwise-green release runs
red and forcing manual re-runs.

Fix

Add pytest-rerunfailures and, only on the test-windows job, retry that
specific transient error:

--reruns 2 --reruns-delay 5 --only-rerun "Failed to apply logs to the hnsw segment writer"
  • --only-rerun scopes the retry to this one native-dependency error, so a
    real, deterministic test failure still fails on the first attempt — nothing
    is masked.
  • The Linux and macOS jobs keep zero reruns, so genuine regressions surface
    there loudly and immediately.
  • pytest-rerunfailures is added to both dev dependency lists
    ([project.optional-dependencies].dev for pip/CI and [dependency-groups].dev
    for uv); zero runtime cost (test-only).

Notes

If a different Windows flake recurs (e.g. the occasional test_closets
ranking assertion), extend the --only-rerun pattern or investigate that one
on its own merits rather than broadening to a blanket retry.

ChromaDB's rust HNSW core intermittently fails compaction on Windows with
"Failed to apply logs to the hnsw segment writer" during add/update — a
long-standing, non-reproducible-on-Linux/macOS flake that hits different tests
(test_migrate_wings, test_closets) across unrelated commits and has been
turning otherwise-green release/CI runs red at random.

Add pytest-rerunfailures and wire `--reruns 2 --only-rerun "Failed to apply
logs to the hnsw segment writer"` into the test-windows job only. The
--only-rerun scope means a real, deterministic failure still fails on the first
run; only this specific transient native-dependency error is retried. The
Linux and macOS jobs deliberately keep zero reruns so genuine regressions
surface there loudly.
@igorls
igorls requested a review from milla-jovovich as a code owner June 22, 2026 19:01
Copilot AI review requested due to automatic review settings June 22, 2026 19:01

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request adds the pytest-rerunfailures dependency to the development dependencies in pyproject.toml to handle transient ChromaDB-on-Windows HNSW compaction failures in CI. There are no review comments, and I have no additional feedback to provide.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

@igorls
igorls merged commit 270f1ae into develop Jun 22, 2026
9 checks passed
@igorls
igorls deleted the ci/windows-chromadb-flake-retries branch June 22, 2026 19:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants