IBMCloud: Use unique mutex for Client retrieval

[cjschaef]

A shared mutex for all ibmcloud.metadata functions, including the
public Client method, deadlocks calls. Since the Client is a public
method, we should still use a mutex, but it should be a unique
mutex to prevent deadlocking.

[rvanderp3]

/assign @rvanderp3

[rvanderp3]

/hold
waiting for tests

[jstuever]

/assign

[cjschaef]

e2e-ibmcloud install failure due to
time="2022-08-18T20:33:42Z" level=error msg="failed to initialize the cluster: Cluster operators image-registry, storage are not available"
Which might be a timing issue in regard to IBM Cloud LB's delaying the CCM from proper worker node labels for the Storage operator
"msg":"Failed to initialize driver...","name":"ibm-vpc-block-csi-driver","CSIDriverName":"IBM VPC block driver","error":"Controller_Helper: Failed to initialize node metadata: error: One or few required node label(s) is/are missing
and potentially the same reason for issues with image-registry
http.response.status=503

Will retest once I push up changes recommended with mutex

[cjschaef]

Install worked as expected

INFO[2022-08-18T23:01:30Z] Running step e2e-ibmcloud-ipi-install-install. 
INFO[2022-08-18T23:47:00Z] Step e2e-ibmcloud-ipi-install-install succeeded after 45m30s. 

Failure due to conformance tests, which is currently expected

INFO[2022-08-19T00:42:52Z] Step e2e-ibmcloud-openshift-e2e-test failed after 55m32s. 
INFO[2022-08-19T00:42:52Z] Step phase test failed after 55m32s.

[rvanderp3]

/hold cancel
/lgtm

[jeffnowicki]

@jstuever this should be ready for your approval... thanks!

[jstuever]

/approve

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jstuever

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• pkg/asset/installconfig/ibmcloud/OWNERS [jstuever]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[barbacbd]

/test yaml-lint
/test verify-vendor
/test verify-codegen
/test unit
/test tf-lint
/test shellcheck
/test okd-verify-codegen
/test okd-unit
/test okd-images
/test images
/test golint

[MayXuQQ]

/test aro-unit
/test e2e-aws
/test gofmt
/test govet
/test e2e-ibmcloud

[openshift-ci]

@cjschaef: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-ibmcloud	61ff6c8	link	false	/test e2e-ibmcloud

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.