fix: make audit more fault tolerant, log error instead of skipping update

[ritazh]

Signed-off-by: Rita Zhang rita.z.zhang@gmail.com

What this PR does / why we need it:
make audit more fault tolerant, log error instead of skipping status update

Which issue(s) this PR fixes (optional, using fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when the PR gets merged):
Fixes #2120

Special notes for your reviewer:

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 0% with 4 lines in your changes missing coverage. Please review.

Project coverage is 54.38%. Comparing base (f5eacc7) to head (3aa1b9d).
Report is 882 commits behind head on master.

Files with missing lines	Patch %	Lines
pkg/audit/manager.go	0.00%	4 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #2162      +/-   ##
==========================================
- Coverage   54.41%   54.38%   -0.04%     
==========================================
  Files         111      111              
  Lines        9530     9529       -1     
==========================================
- Hits         5186     5182       -4     
- Misses       3946     3949       +3     
  Partials      398      398              

Flag	Coverage Δ
unittests	54.38% <0.00%> (-0.04%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[maxsmythe]

LGTM

thanks for doing this!

[maxsmythe]

I wonder if we should also be working to make it easier to verify that audit is healthy. Maybe Prometheus metrics listing # of audit errors for the last audit run? Could be separate work.

[ritazh]

I wonder if we should also be working to make it easier to verify that audit is healthy. Maybe Prometheus metrics listing # of audit errors for the last audit run? Could be separate work.

I will open another issue/PR for this.