Skip to content

Retry for stale reads in KV CAS chaos test#4769

Closed
mprimi wants to merge 3 commits intonats-io:mainfrom
mprimi:marco/chaos_test_kv_direct_get
Closed

Retry for stale reads in KV CAS chaos test#4769
mprimi wants to merge 3 commits intonats-io:mainfrom
mprimi:marco/chaos_test_kv_direct_get

Conversation

@mprimi
Copy link
Copy Markdown
Contributor

@mprimi mprimi commented Nov 8, 2023 •
edited
Loading

Relax KV CAS test which was failing immediately upon reading a key with a revision which is known to be stale.
Instead, retry the read until a newer version is returned.

Signed-off-by: Marco Primi marco@synadia.com

@mprimi mprimi requested a review from a team as a code owner November 8, 2023 19:07
derekcollison
derekcollison reviewed Nov 8, 2023
View reviewed changes
Copy link
Copy Markdown
Member

@derekcollison derekcollison left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Prefer to leave direct gets but you can use checkFor() helper to wait for a certain period of time for the get to be current.

@mprimi mprimi force-pushed the marco/chaos_test_kv_direct_get branch from 6ad58b4 to a743862 Compare November 8, 2023 23:13
@mprimi mprimi changed the title Turn off DirectGet for KV bucket in test Retry for stale reads in KV CAS chaos test Nov 8, 2023
@mprimi
Copy link
Copy Markdown
Contributor Author

mprimi commented Nov 8, 2023

@derekcollison updated to leave DirectGet on, be more tolerant of stale reads, and retry with checkFor.

(2 commits for easier review, squashable on merge)

@derekcollison derekcollison self-requested a review November 9, 2023 00:19
derekcollison
derekcollison requested changes Nov 9, 2023
View reviewed changes
@mprimi mprimi force-pushed the marco/chaos_test_kv_direct_get branch from a743862 to 564661c Compare November 9, 2023 16:53
@mprimi
Copy link
Copy Markdown
Contributor Author

mprimi commented Nov 9, 2023

Updated indentation in 2nd commit
Added 3rd commit to switch to require_ where appropriate

@mprimi mprimi force-pushed the marco/chaos_test_kv_direct_get branch from 564661c to 88a10b1 Compare November 15, 2023 18:28
@mprimi
Copy link
Copy Markdown
Contributor Author

mprimi commented Nov 15, 2023 •
edited
Loading

Just rebased, the Jenkins error comes from Run non JetStream/MQTT tests from the server package::TestProcessSignalMultipleProcessesGlob, the rest is green.

derekcollison
derekcollison reviewed Nov 16, 2023
View reviewed changes
derekcollison
derekcollison reviewed Nov 16, 2023
View reviewed changes
Copy link
Copy Markdown
Member

@derekcollison derekcollison left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One comment on the allow direct update to false.

@mprimi mprimi force-pushed the marco/chaos_test_kv_direct_get branch from 88a10b1 to 136687e Compare November 17, 2023 23:53
@derekcollison
Copy link
Copy Markdown
Member

derekcollison commented Dec 14, 2023

Can we check the failures, seems all tests fails so probably failed staticcheck or something.

Also, @mprimi is this the test that shows reset you wanted us to look at?

@mprimi
Copy link
Copy Markdown
Contributor Author

mprimi commented Dec 14, 2023

@derekcollison this got stale, I'll rebase.

The failing RAFT i mentioned is a different (draft) PR: #4626

@mprimi
Test helper to toggle DirectGet of KV buckets
4b456f2 
Signed-off-by: Marco Primi marco@synadia.com
@mprimi
Update KV CAS 'chaos' test to allow stale reads
dc76df4 
When detecting a stale value, retry rather than fail immediately.

Signed-off-by: Marco Primi marco@synadia.com
@mprimi
Update test to check error using require_* helpers
09ba556 
Signed-off-by: Marco Primi marco@synadia.com
@mprimi mprimi force-pushed the marco/chaos_test_kv_direct_get branch from 136687e to 09ba556 Compare December 14, 2023 23:54
@mprimi mprimi requested a review from derekcollison December 15, 2023 00:08
@mprimi
Copy link
Copy Markdown
Contributor Author

mprimi commented Dec 15, 2023

Freshly rebased on top of latest main.

@github-actions github-actions bot added the stale This issue has had no activity in a while label Mar 19, 2024
@MauriceVanVeen
Copy link
Copy Markdown
Member

MauriceVanVeen commented Jul 18, 2025

Chaos tests were removed in #5455, like mentioned:

Old and simple chaos tests have been replaced by newer and more comprehensive chaos testing infrastructure.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@neilalexander neilalexander Awaiting requested review from neilalexander

@derekcollison derekcollison Awaiting requested review from derekcollison

Assignees

No one assigned

Labels

stale This issue has had no activity in a while

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@mprimi @derekcollison @MauriceVanVeen @neilalexander