Skip to content

"Fix" flaky sharded cluster tests #682

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
nammn merged 10 commits into from
Jan 22, 2026
Merged

"Fix" flaky sharded cluster tests #682

Show file tree
Hide file tree
Changes from 9 commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
8e185f6
increase retries
nammn Jan 7, 2026
64b52ef
fix flaky tests
nammn Jan 7, 2026
91ff11a
tolerate elections
nammn Jan 9, 2026
3eb7bd1
tolerate elections
nammn Jan 12, 2026
6ae02a6
tolerate config down
nammn Jan 12, 2026
db1562d
tolerate config down
nammn Jan 12, 2026
6d88a50
tolerate config down
nammn Jan 12, 2026
bbae06f
try waiting for all agents to be healthy first
nammn Jan 13, 2026
2c6a9c1
fix autocommit
nammn Jan 14, 2026
ddfa4d5
Add CLOUDP-375105 code comments explaining threshold increases
nammn Jan 22, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docker/mongodb-kubernetes-tests/tests/replicaset/replica_set.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -360,6 +360,7 @@ def test_replica_set_sts_should_exist(self):
sts = self.appsv1.read_namespaced_stateful_set(RESOURCE_NAME, self.namespace)
assert sts

@pytest.mark.flaky(reruns=10, reruns_delay=3)

@nammn nammn Jan 12, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

that test is racy

def test_sts_update(self):
sts = self.appsv1.read_namespaced_stateful_set(RESOURCE_NAME, self.namespace)

Expand Down
2 changes: 1 addition & 1 deletion docker/mongodb-kubernetes-tests/tests/shardedcluster/sharded_cluster_migration.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ def mongo_tester(mdb: MongoDB):
def mdb_health_checker(mongo_tester: MongoTester) -> MongoDBBackgroundTester:
return MongoDBBackgroundTester(
mongo_tester,
allowed_sequential_failures=1,
Comment thread
nammn marked this conversation as resolved.

@lucian-tosa lucian-tosa Jan 8, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The assert_connectivity method of MongoDBBackgroundTester attempts a write operation. Should we start considering asserting reads instead?
We already expect writes to be down during re-elections, we might as well assert something that we expect to work.

@nammn nammn Jan 8, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hmmm fair point, but at the same time we don't expect writes to be down the whole migration, just for some amount of time.

@lucian-tosa lucian-tosa Jan 8, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, but the issue is that the downtime intervals are not deterministic. There can be one or N re-elections, we won't know until optimize the restart order. So from a testing standpoint, while there are any pods which are restarting, the writes will not be deterministically available

allowed_sequential_failures=5,
health_function_params={
"attempts": 1,
"write_concern": pymongo.WriteConcern(w="majority"),
Expand Down
1 change: 0 additions & 1 deletion docker/mongodb-kubernetes-tests/tests/shardedcluster/sharded_cluster_upgrade_downgrade.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,6 @@ def mongod_tester(sc: MongoDB) -> MongoTester:
def mdb_health_checker(mongod_tester: MongoTester) -> MongoDBBackgroundTester:
return MongoDBBackgroundTester(
mongod_tester,
# After running multiple tests, it seems that on sharded_cluster version changes we have more sequential errors.
allowed_sequential_failures=5,
health_function_params={
"attempts": 1,
Expand Down