[bug] race condition at controller action

[gthao313]

Problem we found:

When we ran ecs migration test, it would fail on the following errors.

[2023-05-03T17:16:27Z TRACE controller::resource_controller] Action: Creation(Done)
[2023-05-03T17:16:30Z TRACE controller::test_controller::reconcile] action Error(TestError("Resource 'x86-64-aws-ecs-1-migration-test-cluster-instances' not found"))
[2023-05-03T17:16:30Z ERROR controller::test_controller::reconcile] Error state for test 'x86-64-aws-ecs-1-migration-test-cluster-test-2-migrate': Resource 'x86-64-aws-ecs-1-migration-test-cluster-instances' not found
[2023-05-03T17:16:30Z TRACE controller::test_controller::reconcile] action Error(TestError("Resource 'x86-64-aws-ecs-1-migration-test-cluster-instances' not found"))
[2023-05-03T17:16:30Z ERROR controller::test_controller::reconcile] Error state for test 'x86-64-aws-ecs-1-migration-test-cluster-test-2-migrate': Resource 'x86-64-aws-ecs-1-migration-test-cluster-instances' not found
[2023-05-03T17:16:30Z TRACE controller::test_controller::reconcile] action Error(TestError("Unknown error"))

Solution

The resources that the test depends on sometimes come out later. At this time, the controller will think that the resource has not been found and then raise error.

bottlerocket-test-system/controller/src/test_controller/action.rs

Line 127 in 8e3eca8

if result.is_status_code(StatusCode::NOT_FOUND) {

We should add a retry logic to keep finding the resources until exhaust all attempts to avoid "false" NOT FOUND