test: increase WatchClusters test timeout to reduce flakiness (#5566)

[bharathjyothi]

Description

TestWatchClusters_LocalClusterModifications has been observed failing intermittently in CI with Failed due to timeout, even though re-running the job makes it pass (here is a recent example from the past couple days). Fixes #5566.

Checklist:

• Either (a) I've created an enhancement proposal and discussed it with the community, (b) this is a bug fix, or (c) this does not need to be in the release notes.
• The title of the PR states what changed and the related issues number (used for the release note).
• The title of the PR conforms to the Title of the PR
• I've included "Closes [ISSUE #]" or "Fixes [ISSUE #]" in the description to automatically close the associated issue.
• I've updated both the CLI and UI to expose my feature, or I plan to submit a second PR with them.
• Does this PR require documentation updates?
• I've updated documentation as required by this PR.
• I have signed off all my commits as required by DCO
• I have written unit and/or e2e tests for my change. PRs without these are unlikely to be merged.
• My build is green (troubleshooting builds).
• My new feature complies with the feature status guidelines.
• I have added a brief description of why this PR is necessary and/or what this PR solves.
• Optional. My organization is added to USERS.md.
• Optional. For bug fixes, I've indicated what older releases this fix should be cherry-picked into (this may or may not happen depending on risk/complexity).

[bunnyshell]

✅ Preview Environment deployed on Bunnyshell

Component	Endpoints
argocd	https://argocd-nj3x3z.bunnyenv.com/
argocd-ttyd	https://argocd-web-cli-nj3x3z.bunnyenv.com/

See: Environment Details | Pipeline Logs

Available commands (reply to this comment):

• 🔴 /bns:stop to stop the environment
• 🚀 /bns:deploy to redeploy the environment
• ❌ /bns:delete to remove the environment

[qodo-code-review]

PR Summary by Qodo

Increase WatchClusters test timeout and parameterize watch helper

🧪 Tests 🐞 Bug fix 🕐 10-20 Minutes

AI Description

• Increase WatchClusters watch-test timeout to reduce CI flakiness under load.
• Parameterize the runWatchTest helper timeout instead of using a fixed 5s.
• Use a shorter timeout for “no event expected” scenarios to keep tests fast.

Diagram

graph TD
  T["WatchClusters tests"] --> H["runWatchTest(timeout)"] --> W["ArgoDB.WatchClusters"] --> S["SettingsManager"] --> R["Informer resync"]
  H --> TO("time.After(timeout)")

Loading

High-Level Assessment

The following are alternative approaches to this PR:

1. Per-step deadlines instead of a single global timeout

• ➕ Better diagnostics (which step is slow)
• ➕ Less need to inflate overall timeout
• ➖ More test code/complexity
• ➖ Still sensitive to worst-case CI slowness per step

2. Decouple informer resync from the watch callback in tests

• ➕ Reduces synchronous work inside event handlers
• ➕ Can make tests both faster and less flaky
• ➖ May diverge from real execution behavior
• ➖ Requires additional test seams/mocking

3. Use a deterministic/fake clock for timeout handling

• ➕ Eliminates wall-clock dependency in tests
• ➕ Improves reproducibility
• ➖ Requires plumbing a clock dependency into helper or watch logic
• ➖ Not always feasible if underlying code uses real timers

Recommendation: The current approach (explicit, larger timeout + helper parameterization) is the best low-risk fix for CI flakiness because it preserves behavior while accommodating slower runners. If flakiness persists, consider per-step deadlines for improved failure attribution without further inflating total test time.

Files changed (2) +9 / -19

Tests (2) +9 / -19

cluster_norace_test.goRaise WatchClusters test timeouts and centralize constants +8/-16

Raise WatchClusters test timeouts and centralize constants

• Adds shared timeout constants and passes them into runWatchTest instead of relying on the helper’s prior hardcoded value. Uses a shorter timeout for the "no events expected" test case while increasing the main watch-sequence timeout to reduce intermittent CI timeouts.

util/db/cluster_norace_test.go

cluster_test.goParameterize runWatchTest with a caller-provided timeout +1/-3

Parameterize runWatchTest with a caller-provided timeout

• Updates runWatchTest to accept a time.Duration timeout parameter and removes the internal fixed 5-second timeout. This enables individual tests to choose appropriate time budgets based on expected workload.

util/db/cluster_test.go

[qodo-code-review]

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 📎 Requirement gaps (0)

Great, no issues found!

Qodo reviewed your code and found no material issues that require review

Tip of the day

💡 Did you know, you can group findings by type and pick your Finding display, from Minimal to Full

More tips ↗ | Customize Qodo ↗ | Qodo docs ↗

[codecov]

Bundle Report

Bundle size has no change ✅

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 65.60%. Comparing base (34fe81e) to head (84e5b9c).

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #29155      +/-   ##
==========================================
+ Coverage   65.59%   65.60%   +0.01%     
==========================================
  Files         427      427              
  Lines       60564    60564              
==========================================
+ Hits        39724    39736      +12     
+ Misses      17204    17193      -11     
+ Partials     3636     3635       -1     

Flag	Coverage Δ
e2e	26.81% <ø> (+<0.01%)	⬆️
unit-tests	61.21% <ø> (+0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.