Update to latest klog 0.4.0

[dims]

kubernetes/klog@47ffc4e - Add test case for detecting data race
kubernetes/klog@959d342 - Prevent data race in SetOutput* methods
kubernetes/klog@34123a4 - Test with golang 1.12.x
kubernetes/klog@bf4884f - Fix the log duplication issue for --log-file
kubernetes/klog@dc5546c - Backfill integration tests for selecting log destinations
kubernetes/klog@baef93d - fix broken links
kubernetes/klog@07b218b - Add go modules files
kubernetes/klog@b33ae69 - Add flag to include file's dir in logs
kubernetes/klog@7c58910 - correct documentation
kubernetes/klog@a4033db - Code Hygene - glog to klog
kubernetes/klog@941d47b - Revert "Fix the log duplication issue for --log-file."
kubernetes/klog@314f6c4 - Update godoc for the default value of logtostderr
kubernetes/klog@7eb35e2 - Fix the log duplication issue for --log-file.

What type of PR is this?
/kind cleanup

What this PR does / why we need it:

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

NONE

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

[dims]

/milestone v1.16
/priority important-soon
/area code-organization

[dims]

/sig release

[dims]

@liggitt the first step i was told was to run /test pull-kubernetes-e2e-gce-large-performance which i triggered here

[dims]

/test pull-kubernetes-e2e-gce-large-performance

[liggitt]

kubernetes/klog@bf4884f - Fix the log duplication issue for --log-file

was any functional test added to verify kubernetes/klog@bf4884f fixed the duplication issue?

[dims]

@liggitt tested this manually

[liggitt]

Since we are using a single log file, all of the items in l.file array
will point to the same file, so just use one of them to write data.

is that accurate if SetOutputBySeverity was called?

[dims]

@liggitt i should probably add godoc comments that when using SetOutput* they should avoid setting any command line params.

[liggitt]

if CLI flags and SetOutputBySeverity are mutually exclusive, and kube doesn't use SetOutputBySeverity, that's probably ok. Definitely needs documenting

[liggitt]

@liggitt tested this manually

a follow-up (in klog) that exercises this in a test would be good. this is fragile enough that I could see it silently regressing

[liggitt]

/lgtm
/approve

/hold
for scale test input

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: dims, liggitt

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [liggitt]
• staging/src/k8s.io/api/OWNERS [liggitt]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[dims]

@liggitt added kubernetes/klog#85 and kubernetes/klog#84 as follow ups

[dims]

@liggitt @wojtek-t @mborsz as i understand it, this PR and #78466 need to be tested together in a 5k environment for us to see the regression. right?

[dims]

/test pull-kubernetes-e2e-gce-large-performance

[mborsz]

In the previous attempt, the klog update itself (without actually switching to --log-file flag) was a root cause of significant performance regression.

I think it would be good to test this in 5k environment before submission.

[dims]

@mborsz ack agree! @wojtek-t mentioned that he can start a 5k run over the next few days

[wojtek-t]

@dims @liggitt - I've kicked this tests and after less than an hour (~1 minute after first logrotation) kube-apiserver started crashlooping.

I will try to run it one more time, but our suspicion with @mborsz is that it may actually be exactly the same problem.

/hold

[dims]

@wojtek-t did you try just this PR like @mborsz suggested above? (don't introduce #78466)

[wojtek-t]

@wojtek-t did you try just this PR like @mborsz suggested above? (don't introduce #78466)

I wanted to run the previous experiment one more time to ensure it wasn't flake.
I just did that and the result was exactly the same - apiserver started crashlooping immediatley after first log-rotation. So that definitely wasn't flake.

I didn't yet try only this PR - let me try doing that now.
But even if it will work, it doesn't seem to visible help with going towards distroless, right?

[dims]

@wojtek-t it will give me more information on where the problem could be. "immediately after first log-rotation" seems like a pretty big hint at this point. I just want to confirm that without the distroless just this PR works fine.

[liggitt]

But even if it will work, it doesn't seem to visible help with going towards distroless, right?

That would seem to indicate the issue is not a performance issue, but an actual bug in the log rotation code (possibly when combined with the distroless environment... unexpectedly readonly filesystems, etc?)

[dims]

@liggitt exactly!

[wojtek-t]

That would seem to indicate the issue is not a performance issue, but an actual bug in the log rotation code (possibly when combined with the distroless environment... unexpectedly readonly filesystems, etc?)

I think that it can still be performance issue, just with distroless we're configuring stuff differently and using a different code-path.

That said, I've run scalability tests against just this PR (5k-node cluster) and the results look fine (i.e. they are comparable with head).
So from scalability POV I'm fine with this PR (as long as we don't merge the distroless one).

@dims @liggitt - feel free to hold cancel.

[dims]

@wojtek-t agree. I'll start looking at the code path for rotation in klog and file follow up PR(s). Can you please point me to logs from the broken run (with distroless) please?

/hold cancel