Safeguard to prevent NaNs from CRTM affecting minimisation

[ADCollard]

Description

As noted in #916, the CRTM can occasionally produce NaNs in the output brightness temperature and Jacobians. The existing QC tests do not always get triggered by this and so an explicit test for NaNs is required.

Type of change

Please delete options that are not relevant.

• Bug fix (non-breaking change which fixes an issue)

How Has This Been Tested?

Standalone runs to ensure that cases with NaNs in the CRTM brightness temperature are screened out.

Checklist

• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• New and existing tests pass with my changes
• Any dependent changes have been merged and published

[ADCollard]

At @emilyhcliu 's request, added some more info when a NaN is trapped:

 WARNING: NaN found in CRTM simulated radiance output
 NaN for channel           10  at latitude =   -29.3777900000000      
  longitude =    202.419620000000     
 NaN for channel           11  at latitude =   -29.3777900000000      
  longitude =    202.419620000000     

[RussTreadon-NOAA]

Looks good. One minor suggestion for this PR and PR #925.

[RussTreadon-NOAA]

WCOSS2 ctests

Install ADCollard:develop-v16 at f8b8c3c and NOAA-EMC:develop-v16 at 734b8d9 on Cactus. Run ctests with the following results

Test project /lfs/h2/emc/da/noscrub/russ.treadon/git/gsi/pr924/build
    Start 2: rtma
    Start 4: hafs_4denvar_glbens
    Start 1: global_4denvar
    Start 5: hafs_3denvar_hybens
    Start 6: global_enkf
    Start 3: rrfs_3denvar_rdasens
1/6 Test #3: rrfs_3denvar_rdasens .............   Passed  1217.44 sec
2/6 Test #6: global_enkf ......................   Passed  1295.16 sec
3/6 Test #5: hafs_3denvar_hybens ..............   Passed  1584.69 sec
4/6 Test #2: rtma .............................   Passed  1698.47 sec
5/6 Test #4: hafs_4denvar_glbens ..............   Passed  1715.91 sec
6/6 Test #1: global_4denvar ...................   Passed  2288.05 sec

100% tests passed, 0 tests failed out of 6

Total Test time (real) = 2288.30 sec

All tests Passed.

[CatherineThomas-NOAA]

@ADCollard @RussTreadon-NOAA - Hm, this is interesting. I agree that the differences are very very small. I wouldn't expect any differences at all if there weren't any NaNs captured, which when looking at the stdout for this pr, I don't see any WARNING for this new QC in the output. We're likely dealing with reproducibility in general with some more recent updates. That shouldn't impact this PR though.

The code looks good to me. Approve.

[RussTreadon-NOAA]

Approve.