delay between app becoming available and cf ssh becoming available

[sesmith177]

The buildpacks team has been experimenting with running some of our pipelines against GCP. Parts of these pipelines run buildpack integration tests and CATS.

While running these tests, we've noticed that cf ssh sometimes fails when called immediately after a successful cf push. cf ssh eventually (~seconds) becomes available. However, the initial cf ssh failure is enough to cause the specs to fail. This occurs roughly 40% of the time.

Our test environments are BOSH-lite VMs running CF + diego. We've observed this on VMs with 4cpu + 15 GB memory (n1-standard-4) and on VMs with 16cpu + 15 GB memory (n1-highcpu-16).

Example failing job

Example failing CATs run

We are filing this issue as we did not see this problem on AWS. Our AWS configuration is 16cpu + 30 GB memory (c3.4xlarge), so we are unsure if it is configuration or platform related.

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/132352631

The labels on this github issue will be updated when the story is started.

[emalm]

Thanks, @sesmith177, are you still seeing this problem in your BOSH-Lite environments?

Best,
Eric

[sesmith177]

@ematpl we ran CATs / buildpack integration tests that use cf ssh against GCP BOSH-lites for a few days and regularly saw these failures. Right now, this probably our biggest obstacle to moving our pipelines from AWS over to GCP.

Our GCP Concourse is not currently deployed, though we could stand it up pretty easily if you would like / there is data we could collect.

[emalm]

Thanks, @sesmith177, we'll see about getting a Diego pair to examine this issue soon.

Best,
Eric

[bdshroyer]

Closing this issue. According to @RochesterinNYC, upgrading to the latest stemcell eliminated the delay.