Add master scrontab runner to serialize CI experiment rocoto calls

[Copilot]

Description

On SLURM-managed scron systems (e.g. Gaea), generate_workflows.sh was writing one independent scrontab entry per experiment. All fired every 5 minutes simultaneously, causing multiple concurrent rocotorun processes that exhausted head-node memory (OOM).

Changes

dev/workflow/generate_workflows.sh

• Instead of writing a per-experiment scrontab entry in the loop, collect each experiment's .scron.sh path into _scron_sh_files[]
• After all experiments are created, generate ${RUNTESTS}/EXPDIR/rocoto_master_run.sh — a sequential runner that calls each .scron.sh one at a time:

  if [[ -x "/path/to/exp1.scron.sh" ]]; then "/path/to/exp1.scron.sh"; fi
  if [[ -x "/path/to/exp2.scron.sh" ]]; then "/path/to/exp2.scron.sh"; fi
  ...

• Write a single scrontab entry (with --dependency=singleton) pointing to the master script — --partition and --account sourced from the first experiment's .crontab
• Wall-time for the master scrontab entry is fixed at 00:10:00
• Error out clearly if required #SCRON --partition= / #SCRON --account= directives are missing from the crontab
• Email failure notifications unchanged — each individual .scron.sh still runs its own rocotostat-based monitoring and sends alerts independently

Resolves NOAA-EMC#2619

Type of change

• Bug fix (fixes something broken)
• New feature (adds functionality)
• Maintenance (code refactor, clean-up, new CI test, etc.)

Change characteristics

• Is this change expected to change outputs (e.g. value changes to existing outputs, new files stored in COM, files removed from COM, filename changes, additions/subtractions to archives)? NO
  • GFS
  • GEFS
  • SFS
  • GCAFS
• Is this a breaking change (a change in existing functionality)? NO — regular crontab (non-scron) path is unchanged; scron behavior changes from N parallel entries to 1 master entry
• Does this change require a documentation update? NO
• Does this change require an update to any of the following submodules? NO
  • EMC verif-global
  • GDAS
  • GFS-utils
  • GSI
  • GSI-monitor
  • GSI-utils
  • UFS-utils
  • UFS-weather-model
  • wxflow

How has this been tested?

• End-to-end shell simulation verified: single */5 timing line in scrontab, correct shebang, executable permissions, all experiment scripts referenced in master, static 10-minute wall time in generated scrontab entry

Checklist

• Any dependent changes have been merged and published
• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have documented my code, including function, input, and output descriptions
• My changes generate no new warnings
• New and existing tests pass with my changes
• This change is covered by an existing CI test or a new one has been added
• Any new scripts have been added to the .github/CODEOWNERS file with owners
• I have made corresponding changes to the system documentation if necessary