[GOBBLIN- 1856] Add flow trigger handler leasing metrics

[meethngala]

Dear Gobblin maintainers,

Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below!

JIRA

• My PR addresses the following Gobblin JIRA issues and references them in the PR title. For example, "[GOBBLIN-XXX] My Gobblin PR"
  • https://issues.apache.org/jira/browse/GOBBLIN-1856

Description

• Here are some details about my PR, including screenshots (if applicable):
  As part of this PR, I want to add metrics for FlowTriggerHandler to help understand if we were able to obtain, handle and transition the different lease acquiring statuses when we switch to multi-active scheduler mode where each scheduler would attempt to lease a flow event and process it further based on the status of the attempt.

Tests

• My PR adds the following unit tests OR does not need testing for this extremely good reason:
• Builds Successfully

Commits

• My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":
  1. Subject is separated from body by a blank line
  2. Subject is limited to 50 characters
  3. Subject does not end with a period
  4. Subject uses the imperative mood ("add", not "adding")
  5. Body wraps at 72 characters
  6. Body explains "what" and "why", not "how"

[umustafi]

Overall flow of creating and using these metrics should be

1. create counter at class level
2. register the metric with metric context
3. call counter.count/mark() every time you want it to increase

Here you are creating a new one every time you mean to actually increment it. For ex:

gobblin/gobblin-service/src/main/java/org/apache/gobblin/service/monitoring/DagActionStoreChangeMonitor.java

Line 113 in 4a9dd53

messageProcessedMeter.mark();

[meethngala]

I had a different idea in mind since I only wanted to check if the FlowTriggerHandler gets different types of statuses or not. But I guess initializing at class level and inc() the counter will help gauge if the traffic distribution is happening uniformly between the hosts or not!

[umustafi]

Discussed offline and agreed the counter will provide more information about load balancing and relative amount of occurrence of each case

[umustafi]

do you want to add logs here for now for the other cases as well? Maybe they can be debug level logs

[meethngala]

I wanted to but then realized that we do spit out some logs as part of this method scheduleReminderForEvent and didn't want to make it noisy for emitting similar logs. However, I have added a debug level log when we no longer attempt for a lease... since I found that to be missing :)

[codecov-commenter]

Codecov Report

❌ Patch coverage is 0% with 10 lines in your changes missing coverage. Please review.
✅ Project coverage is 46.94%. Comparing base (c4a466b) to head (e27cdd1).
⚠️ Report is 406 commits behind head on master.

Files with missing lines	Patch %	Lines
...vice/modules/orchestration/FlowTriggerHandler.java	0.00%	10 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master    #3717      +/-   ##
============================================
- Coverage     46.96%   46.94%   -0.03%     
+ Complexity    10818    10811       -7     
============================================
  Files          2143     2143              
  Lines         84600    84555      -45     
  Branches       9404     9390      -14     
============================================
- Hits          39733    39694      -39     
- Misses        41241    41245       +4     
+ Partials       3626     3616      -10     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.