Bug 1823406: Upstream: 89160: Remove potentially unhealthy symlink only for dead containers

[tedyu]

As the discussion over #52172 showed, there is race condition between the container log rotation and the kubelet GC which may result in the loss of symlink.

Here is how container log rotation works (see containerLogManager#rotateLatestLog):

• rename current log to rotated log file whose filename contains current timestamp (fmt.Sprintf("%s.%s", log, timestamp))
• reopen the container log
• if Add alpha API documentation #2 fails, rename rotated log file back to container log

There is small but indeterministic amount of time during which log file doesn't exist (between steps #1 and #2, between #1 and #3). Hence the symlink may be deemed unhealthy during that period.

This PR resorts to runtimeService.ContainerStatus() to check whether the container corresponding to the potentially unhealthy symlink is alive or not. The symlink would only be removed for dead containers.

[tedyu]

/cc @rphillips @sjenning

[rphillips]

/lgtm

/assign @sjenning

[rphillips]

Upstream ref

[sjenning]

/approve

[openshift-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: rphillips, sjenning, tedyu

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [sjenning]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[rphillips]

/retitle Bug 1823406: Remove potentially unhealthy symlink only for dead containers

[rphillips]

/retitle Bug 1823406: Upstream: 89160: Remove potentially unhealthy symlink only for dead containers

[openshift-ci-robot]

@tedyu: This pull request references Bugzilla bug 1823406, which is valid. The bug has been moved to the POST state. The bug has been updated to refer to the pull request using the external bug tracker.

3 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target release (4.5.0) matches configured target release for branch (4.5.0)
• bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

Details

In response to this:

Bug 1823406: Remove potentially unhealthy symlink only for dead containers

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[tedyu]

/retest

[openshift-bot]

/retest

Please review the full test history for this PR and help us cut down flakes.

[openshift-bot]

/retest

Please review the full test history for this PR and help us cut down flakes.

[openshift-bot]

/retest

Please review the full test history for this PR and help us cut down flakes.

[openshift-bot]

/retest

Please review the full test history for this PR and help us cut down flakes.

[openshift-ci-robot]

@tedyu: All pull requests linked via external trackers have merged: openshift/origin#24926. Bugzilla bug 1823406 has been moved to the MODIFIED state.

Details

In response to this:

Bug 1823406: Upstream: 89160: Remove potentially unhealthy symlink only for dead containers

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.