Fix Intermittently Failing Unit Test

[aphralG]

The unit test on the pipeline fails intermittently with the error below. It seems to be the commander_test.go that is failing.

Example of pipeline failing: https://github.com/nginx/agent/actions/runs/4436873229/jobs/7785851614

Error:

time="2023-03-16T12:12:39Z" level=info msg="Commander received meta:<message_id:\"1234\" > , <nil>"
Recv Command: <nil>
Recv Command Error: rpc error: code = Canceled desc = context canceled
time="2023-03-16T12:12:39Z" level=info msg="Commander received <nil>, rpc error: code = Canceled desc = grpc: the client connection is closing"
time="2023-03-16T12:12:39Z" level=error msg="Commander Channel Recv: error communicating with bufnet, code=Canceled, message=grpc: the client connection is closing"
time="2023-03-16T12:12:39Z" level=info msg="Commander Channel Recv: retrying to connect to bufnet"
time="2023-03-16T12:12:39Z" level=warning msg="Error closing old grpc connection: rpc error: code = Canceled desc = grpc: the client connection is closing"
time="2023-03-16T12:12:39Z" level=error msg="Unable to create command channel: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing closed\""
time="2023-03-16T12:12:39Z" level=info msg="Commander retrying to connect to bufnet"
Recv Command: meta:<message_id:"1234" > 
Recv Command: <nil>
Recv Command Error: rpc error: code = Canceled desc = context canceled```

[redaHamioui]

how can i reproduce the bug ?

[oliveromahony]

Thanks for the interest, I think running the tests in the pipeline saw inconsistent behavior and the intermittent failures. Running the test locally might reproduce the issue. I added this fix #271 to tackle the problem. I'll update this if anything else needs to be done, otherwise I'll close it in a week or so

[Dean-Coakley]

Please keep open. Still seeing intermittent failures: https://github.com/nginx/agent/actions/runs/4629318982/jobs/8189426092 and https://github.com/nginx/agent/actions/runs/4627873493/jobs/8186316791

[oliveromahony]

@redaHamioui we still have this issue if you want to take a look? Some changes were made in #268 to address it but I can get it to fail locally running the following command go test -timeout 30s github.com/nginx/agent/sdk/v2/client -v -count=10

It looks like a race condition with the channels is causing the problem

[oliveromahony]

this appears to be resolved