Enabling ssh connections to app containers in the starting state

[gberche-orange]

Summary

As a developer, in order to troubleshoot web applications that are hanging during startup, I need to ssh into their containers (e.g. to trigger a thread dump by sending a signal to the process) while the app health check is'nt yet reporting the app as started.

Expected Result

Ssh connection opens if sshd is available.

Actual Result

Ssh connection is systematically refused, when the app is starting.

Workaround is to use "process" healthcheck instead of "port" or "http" health check

Context

Steps to Reproduce

• Deploy a webapp configured to hang before honoring its port/http health check
• cf ssh app

Possible Causes or Fixes (optional)

Chatting with @emalm, it might be that LRP in the CLAIMING state don't yet provide networking details necessary to access ssh connection

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/168433515

The labels on this github issue will be updated when the story is started.

[sunjayBhatia]

Is using https://docs.cloudfoundry.org/devguide/deploy-apps/manifest-attributes.html#timeout and the healthcheck workaround sufficient to help debug these sorts of issues?

Have you had any luck or tried running this sort of app locally with something like cfdev?

[sunjayBhatia]

Also, at the moment, the ssh daemon running in your app container that the ssh proxy connects to is not something the rep "knows" about. It is a process that CC adds as part of the DesiredLRP definition for your app and it is just bundled as a codependent process with your app process. The ssh proxy knows to connect to it b/c the CC puts an "ssh route" on your desired LRP with the container port to connect to (which is used to find the appropriate host port). While an LRP may be in the CLAIMED state and have networking info b/c the container exists, we don't necessarily know which of the generic list of codependent processes is started and listening on the appropriate ports until they become healthy so it is safe to only make their networking info available after that happens.

[sunjayBhatia]

closing this, feel free to reopen with more questions

[gberche-orange]

@sunjayBhatia sorry for late feedback on this, I had missed the github update (wrong email notifications filtering). I don't seem to have permission to reopen this issue, would you mind reopening it ?

Is using https://docs.cloudfoundry.org/devguide/deploy-apps/manifest-attributes.html#timeout and the healthcheck workaround sufficient to help debug these sorts of issues?

Yes, as mentioned in the issue, the workaround was to disable http health check to be able to ssh into the app.

Have you had any luck or tried running this sort of app locally with something like cfdev?

Thanks for the reminder, I'll keep it in mind for next troubleshootings.

A range of problems can't however be reproduced locally such as environment specific issues:

• dns look latency/failures
• ASG-related
• CF network policy related

While an LRP may be in the CLAIMED state and have networking info b/c the container exists, we don't necessarily know which of the generic list of codependent processes is started and listening on the appropriate ports until they become healthy so it is safe to only make their networking info available after that happens.

Would be following rewording be correct ? Fixing this problem would require a lot of work given current diego design ?

Could be same limitation (delayed networking info publication) also apply to any failure of other co-dependent processes beyond ssh daemons? In other words, could a fix also benefit to other use-cases ?

Would this user-facing feature be easier to implement in Cf4K8s ?

/CC @emalm

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/170343166

The labels on this github issue will be updated when the story is started.