healthchecks : Improve self logs processing and healthchecks logs.

[franciscovalentecastro]

Description

This PR adds several features and fixes that improve the runtime checks and self logs processing. Here is a detailed description :

• Disable export of fluent-bit severity: debug logs (add self logs debug severity grep filter #1437, b/272779619).
  • Added integration test TestNoFluentBitDebugSelfLogs.
• Simplify and clean self_logs.go implementation after fluent-bit 2.2 upgrade bug fixes. (b/328463822).
• Remove sourceLocation field from health-checks.log which doesn't bring enough value and makes it harder to read.
• Refactor all Runtime Check errors into healthchecks/error.go.

Before health-checks.log entry

{"severity":"INFO","time":"2024-04-16T16:24:01Z","logging.googleapis.com/sourceLocation":{"file":"/work/internal/healthchecks/healthchecks.go","function":"github.com/GoogleCloudPlatform/ops-agent/internal/healthchecks.HealthCheckResult.LogResult","line":"57"},"message":"[API Check] Result: PASS"}

After health-checks.log entry

{"severity":"INFO","time":"2024-04-15T21:33:08Z","message":"[API Check] Result: PASS"}

Note : This link points to the committed .go files without the goldens to make it easier for review.

Related issue

b/272779619, b/303073892, b/328463822

How has this been tested?

Checklist:

• Unit tests
  • Unit tests do not apply.
  • Unit tests have been added/modified and passed for this PR.
• Integration tests
  • Integration tests do not apply.
  • Integration tests have been added/modified and passed for this PR.
• Documentation
  • This PR introduces no user visible changes.
  • This PR introduces user visible changes and the corresponding documentation change has been made.
• Minor version bump
  • This PR introduces no new features.
  • This PR introduces new features, and there is a separate PR to bump the minor version since the last release already.
  • This PR bumps the version.

[jefferbrecht]

Is this actually a fatal error? Won't fluent-bit continue to run if it fails to parse a log?

[franciscovalentecastro]

It is not a fatal error. Thanks for mentioning. I changed to IsFatal: false.

[jefferbrecht]

Remove this comment line if it's no longer needed

[franciscovalentecastro]

I removed the // It also translates "code" to the rich text message from the "selfLogTranslationList" part of the comment which is not relevant anymore.