[GOBBLIN-1847] Exceptions in the JobLauncher should try to delete the existing workflow if it is launched

[homatthew]

Dear Gobblin maintainers,

Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below!

JIRA

• My PR addresses the following Gobblin JIRA issues and references them in the PR title. For example, "[GOBBLIN-1847] My Gobblin PR"
  • https://issues.apache.org/jira/browse/GOBBLIN-1847

Description

• Here are some details about my PR, including screenshots (if applicable):
  If the JobLauncher encounters any errors, Gobblin code cleans up the working directory that contains Gobblin state. But we do not clean up the helix workflows. This can cause us to spin up helix workflows that do not have valid Gobblin State on HDFS / S3.

When this occurs, it causes rapid container recycling, which puts tremendous load on Zookeeper due to Helix API calls, causing a peak in ZK callbacks

The cancellation logic is obnoxious, so here is the high level of what happens.

1. Calling AbstractJobLauncher#cancelJob(JobListener) sets AbstractJobLauncher#cancellationRequested to true
2. This causes the AbstractJobLauncher#cancellationExecutor created by the AbstractJobLauncher#startCancellationExecutor() method to call the abstract method AbstractJobLauncher#executeCancellation()
3. That method implemented in GobblinHelixJobLauncher#executeCancellation(), which eventually calls HelixUtils.cancelWorkflow(...)

Considerations

• I considered modifying the runWorkunits itself since that is where the cleanup directory is called. But I wanted to leverage the existing the cancellation request logic (which is admittedly super overkill and has a ton of concurrency). We also collect a ton of metrics during the cancellation process, so it's important that I call the cancel method that is part of the AbstractJobLauncher. Since we need access to the job listener, it makes most sense to have it be part of the launchJob method

• The reason I clean up even when there is a successful finish, is because there isn't really a reason not to cancel a workflow to run after the job is finished for Fast Ingest. The cancel call is safe to call multiple times and there are clearly still edge cases where we don't call cancel properly

• I am not a fan of adding a new flag, but I want to maintain backward compatibility since this job launcher is used beyond Fast Ingest

Testing

• My PR adds the following unit tests OR does not need testing for this extremely good reason:
  

Commits

• My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":
  1. Subject is separated from body by a blank line
  2. Subject is limited to 50 characters
  3. Subject does not end with a period
  4. Subject uses the imperative mood ("add", not "adding")
  5. Body wraps at 72 characters
  6. Body explains "what" and "why", not "how"

[homatthew]

Fun fact: verify without a parameter verifies the method is called exactly once! 😃

[codecov-commenter]

Codecov Report

❌ Patch coverage is 50.00000% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 46.86%. Comparing base (fc508ca) to head (cabb968).
⚠️ Report is 420 commits behind head on master.

Files with missing lines	Patch %	Lines
...pache/gobblin/cluster/GobblinHelixJobLauncher.java	50.00%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master    #3711      +/-   ##
============================================
- Coverage     46.87%   46.86%   -0.01%     
- Complexity    10771    10808      +37     
============================================
  Files          2138     2141       +3     
  Lines         84139    84433     +294     
  Branches       9357     9386      +29     
============================================
+ Hits          39436    39569     +133     
- Misses        41102    41264     +162     
+ Partials       3601     3600       -1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[ZihanLi58]

+1