SWODLR-SDS Issues

[joshgarde]

Issues which have been observed on the SWODLR HySDS instance:

• Jobs failing upon retry due to noclobber setting
• Odd API behavior where jobs report as completed during intermediate step and change to failed later
• Job timeouts being too low
• ElasticSearch failures lead to unpredictable/difficult to diagnose API issues

---

Current plan for corrective action:

• SDS: handle noclobber issue
• SWODLR: Investigated and found to be false issue
• SWODLR: Timeouts to be lowered to 10 hours instead of increased (TODO)
• SDS: implement automated monitoring of ElasticSearch on instance

[joshgarde]

• HySDS ticket launched: https://hysds-core.atlassian.net/browse/HC-516
• SWOT aware of noclobber behavior and ticket launched
• SWODLR fixes deployed
  • Issues with exception handling fixed in latest commons PR
    • Handler Updates swodlr-common-py#4

[viviant100]

3/25/24: The update of "SDS: handle noclobber issue" was developed in v5.1 and will be deployed on the week of 4/8/24. v5.1 is currently in SDR and should be wrapped up on the week of 4/1/14.