[bugfix]: increase test timeout

[anencore94]

What this PR does / why we need it:

• It might fix unit test fail issue described in Go Test failed sometimes #1649

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
#1649

Checklist:

• Docs included if any changes are user facing

[andreyvelich]

@anencore94 Thank you for taking this!
Are you sure that increasing timeout will help to avoid flaky unit tests ?

For example in this failed test: https://github.com/kubeflow/katib/runs/3523610062?check_suite_focus=true.
Trial status was updated to Metrics Unavailable, but tests were failed on the previous step, where we compare observation results.

I think to fix it, we might need to split running Experiments in the different unit tests.

For example, currently in the TestReconcileBatchJob we run 3 Experiments:

1. Trial run with "Failed" BatchJob
2. Trail with "Complete" BatchJob and Available metrics.
3. Trail with "Complete" BatchJob and Unavailable metrics.

Maybe we should split it in 3 different unit tests with separate test managers.

WDYT @anencore94 @gaocegege @johnugeorge ?

[gaocegege]

/lgtm
/approve

[google-oss-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: anencore94, gaocegege

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [gaocegege]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[gaocegege]

Sorry for the quick merge.

I think we can increase it and see If it works. In the best case, we should split the test cases. Should we create an issue for it?

[andreyvelich]

No worries @gaocegege :)
Yes, let's continue the discussion under #1649.