Upgrade to spack-stack 1.6.0

[DavidHuber-NOAA]

This upgrades the modules to the newest spack-stack version (1.6.0). In particular, prod_util is upgraded from 1.2.2 to 2.1.1. Since Cheyenne has been decommissioned, this also removes support for that system.

[DavidHuber-NOAA]

@EdwardSafford-NOAA Would you mind taking a look at the monitoring files generated by an experiment on Hera and see if everything looks OK? I've placed the files into /scratch1/NCEPDEV/global/David.Huber/monitor/ss_160/{minmon,ozmon,radmon}.

[EdwardSafford-NOAA]

I took a quick look at the output. I haven't tried plotting it yet but the RadMon data looks good.

The MinMon looks good except for the gnorm_data.txt file. That should have an entry for every processed cycle but it only has the last cycle's data. That probably means the file isn't getting found, which suggests the TANK definition could be out of date. That may not be a problem with these changes. If you point me to your log file I can probably figure out what's going on.

The OznMon data is missing. Only the stdout*.tar and warning message files have made it. I don't have permission to open up the stdout*.tar files. If you can open those up I can take a look and see what's up. Also the log file location would be useful too.

[DavidHuber-NOAA]

Thanks for taking a look.

For the minmon gnorm_data.txt files, I copied these from each cycle, so they didn't get renamed and indeed were overwritten. The originals are in the directories /scratch1/NCEPDEV/global/David.Huber/monitor/ss_160/gdas.202111*/*/products/atmos/minmon/. If you still need to take a look at the logs, they can be found here: /scratch1/NCEPDEV/global/David.Huber/para/comrot/ss_160/logs/202111*/gdasvminmon.log.

Similarly, for the oznmon stdout tar files, the originals are located in /scratch1/NCEPDEV/global/David.Huber/monitor/ss_160/gdas.202111*/*/products/atmos/oznmon/{horiz,time}. I've gone through and untarred each one. The log files for these can be found here: /scratch1/NCEPDEV/global/David.Huber/para/comrot/ss_160/logs/202111*/gdasverfozn.log.

[EdwardSafford-NOAA]

Good news! It looks like in both cases the issue is a problem with copying data to the /scratch1/NCEPDEV/global/David.Huber/monitor/ss_160/{minmon,ozmon} directory. When I look at the original output in /scratch1/NCEPDEV/global/David.Huber/para/comrot/ss_160/gdas.202111*/1*/products/atmos I see both the OznMon ieee_d and *ctl files and a complete MinMon gnorm_data.txt file. It all looks as expected.

I'd like to copy some of your data and do some test runs with the new obs-monitor plot mechanism. If you can leave that data in place for ~30 min that would be great. Thanks!

[DavidHuber-NOAA]

@EdwardSafford-NOAA Great! I have placed the files in /scratch1/NCEPDEV/global/David.Huber/for_ed.

[EdwardSafford-NOAA]

Here's a summary plot I made from your data using the new obs-monitor plotting mechanism. I only used 4 cycles so the 30 day averages look a lot like the 1 day averages, but the data is good:

[DavidHuber-NOAA]

Fantastic, thanks @EdwardSafford-NOAA!

[EdwardSafford-NOAA]

And here's the summary ozone plot, again using the obs-monitor:

[EdwardSafford-NOAA]

The MinMon plots look good too.

[EdwardSafford-NOAA]

Changes look good. Thanks!