[GOBBLIN-1838] Introduce total count based completion watermark

[wsarecv]

Dear Gobblin maintainers,

Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below!

JIRA

• My PR addresses the following Gobblin JIRA issues and references them in the PR title.
  • https://issues.apache.org/jira/browse/GOBBLIN-1838

Description

• Here are some details about my PR, including screenshots (if applicable):
  • Currently the completion watermark is determined according to the completeness percentage: max of srcCount/refCount, for each refTier. This change introduces a new "total count based completion watermark", which is determined by a new completeness percentage: srcCount / sum of all refCount, for each refTier. The new total count based watermark will be used to check the completeness of data which is aggregated from multiple tiers.
  • This PR also refactors existing codes and introduces CompletenessWatermarkUpdater.

Tests

• My PR adds new test cases in the following unit tests:
  • adds new test cases in the existing KafkaAuditCountVerifierTest and IcebergMetadataWriterTest
  • adds new unit test CompletenessWatermarkUpdaterTest

Commits

• My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":
  1. Subject is separated from body by a blank line
  2. Subject is limited to 50 characters
  3. Subject does not end with a period
  4. Subject uses the imperative mood ("add", not "adding")
  5. Body wraps at 72 characters
  6. Body explains "what" and "why", not "how"

[codecov-commenter]

Codecov Report

Merging #3701 (6afb4cd) into master (51a852d) will decrease coverage by 1.96%.
The diff coverage is 79.72%.

@@             Coverage Diff              @@
##             master    #3701      +/-   ##
============================================
- Coverage     46.97%   45.02%   -1.96%     
+ Complexity    10794     2123    -8671     
============================================
  Files          2138      414    -1724     
  Lines         84132    17823   -66309     
  Branches       9356     2172    -7184     
============================================
- Hits          39518     8024   -31494     
+ Misses        41015     8930   -32085     
+ Partials       3599      869    -2730     

Impacted Files	Coverage Δ
...ceberg/writer/IcebergMetadataWriterConfigKeys.java	0.00% <ø> (ø)
...completeness/verifier/KafkaAuditCountVerifier.java	67.64% <72.72%> (+16.48%)	⬆️
.../gobblin/iceberg/writer/IcebergMetadataWriter.java	72.53% <74.35%> (+0.21%)	⬆️
...g/writer/AbstractCompletenessWatermarkUpdater.java	78.43% <78.43%> (ø)
...n/iceberg/writer/CompletenessWatermarkUpdater.java	100.00% <100.00%> (ø)
...writer/TotalCountCompletenessWatermarkUpdater.java	100.00% <100.00%> (ø)

... and 1730 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

[ZihanLi58]

+1