[GOBBLIN-1805] Check watermark for the most recent hour for quiet topics

[vikrambohra]

Dear Gobblin maintainers,

Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below!

JIRA

• My PR addresses the following Gobblin JIRA issues and references them in the PR title. For example, "[GOBBLIN-XXX] My Gobblin PR"
  • https://issues.apache.org/jira/browse/GOBBLIN-1805

Description

• Here are some details about my PR, including screenshots (if applicable):
  Currently, Audit check is done for [currentWatermark, currentWatermark +1hour] window for quiet topics. However when the audit check for this window fails (due to an upstream bug in audit), the watermark never progress as the currentWatermark is never updated. To avoid this we keep progressing the watermark by checking the window [currentHour -1, currentHour] window

Tests

• My PR adds the following unit tests OR does not need testing for this extremely good reason:
  Updated unit test to reflect this change

Commits

• My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":
  1. Subject is separated from body by a blank line
  2. Subject is limited to 50 characters
  3. Subject does not end with a period
  4. Subject uses the imperative mood ("add", not "adding")
  5. Body wraps at 72 characters
  6. Body explains "what" and "why", not "how"

[codecov-commenter]

Codecov Report

Merging #3698 (a7d398d) into master (158d6ff) will decrease coverage by 2.48%.
The diff coverage is 100.00%.

@@             Coverage Diff              @@
##             master    #3698      +/-   ##
============================================
- Coverage     49.45%   46.98%   -2.48%     
- Complexity     9309    10796    +1487     
============================================
  Files          1756     2138     +382     
  Lines         68468    84132   +15664     
  Branches       7809     9356    +1547     
============================================
+ Hits          33862    39526    +5664     
- Misses        31466    41006    +9540     
- Partials       3140     3600     +460     

Impacted Files	Coverage Δ
.../gobblin/iceberg/writer/IcebergMetadataWriter.java	72.31% <100.00%> (-0.05%)	⬇️

... and 388 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

[ZihanLi58]

+1