Skip to content

SPRINT: Upgrade many converters of satellite data bufr2ioda.x to align with JEDI Data Conventions #874

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
rhoneyager merged 37 commits into from
Mar 29, 2022
Merged

SPRINT: Upgrade many converters of satellite data bufr2ioda.x to align with JEDI Data Conventions #874

Show file tree
Hide file tree
Changes from 30 commits
Commits
Show all changes
37 commits
Select commit Hold shift + click to select a range
3e14516
allow units attribute to a variable to be optional
gthompsnWRF Mar 10, 2022
aa3a8c3
allow units attribute to a variable to be optional
gthompsnWRF Mar 10, 2022
a700602
skip units if set to none or empty quotes
gthompsnWRF Mar 10, 2022
0ff4aaf
updates to aircar, airep, amdar to match naming conventions
gthompsnWRF Mar 10, 2022
f8a02e9
Merge branch 'feature/sprint-ioda-converters' into feature/sprint_iod…
gthompsnWRF Mar 10, 2022
7cc9e85
abbeviate degree for windDirection
gthompsnWRF Mar 10, 2022
441a81b
fix up windDirection and couple other items
gthompsnWRF Mar 10, 2022
e6b53eb
many yaml files changed for JEDI naming conventions
gthompsnWRF Mar 10, 2022
dd37dfe
Merge branch 'feature/sprint-ioda-converters' into feature/sprint_iod…
gthompsnWRF Mar 10, 2022
74a67f5
adjust many files to align with JEDI naming conventions
gthompsnWRF Mar 10, 2022
82b8eec
make all new test ref files for JEDI naming conventions during sprint
gthompsnWRF Mar 10, 2022
970cd23
Merge branch 'feature/sprint-ioda-converters' into feature/sprint_iod…
srherbener Mar 11, 2022
aa66ec5
Merge branch 'feature/sprint-ioda-converters' into feature/sprint_iod…
gthompsnWRF Mar 15, 2022
535845e
start of changes for matching JEDI Data Conventions
gthompsnWRF Mar 15, 2022
53f4b84
Merge branch 'feature/sprint_ioda_greg_bufr2ioda' into feature/sprint…
gthompsnWRF Mar 15, 2022
84dec78
able to combine v2 into prior MHS file
gthompsnWRF Mar 15, 2022
62ee0bd
able to merge v2 yaml into HRS earlier file when updating to JEDI Dat…
gthompsnWRF Mar 15, 2022
af59867
update testref files compliant with JEDI Data Conventions
gthompsnWRF Mar 15, 2022
e9894e5
new addition of AVHRR testinput/output files and consistent with JEDI…
gthompsnWRF Mar 15, 2022
0daa2c7
new addition of AVHRR testinput/output files and consistent with JEDI…
gthompsnWRF Mar 15, 2022
db0009d
add ctest for AVHRR
gthompsnWRF Mar 15, 2022
68eadbe
new addition of AIRS testinput/output yaml and ctest consistent with …
gthompsnWRF Mar 15, 2022
9d6ddac
new addition of AMSUA testinput/output yaml and ctest consistent with…
gthompsnWRF Mar 15, 2022
6a57bba
new addition of ATMS testinput/output yaml and ctest consistent with …
gthompsnWRF Mar 16, 2022
dfc5a2c
new addition of CRIS testinput/output yaml and ctest consistent with …
gthompsnWRF Mar 16, 2022
ca8b0ab
new addition of SEVIRI (CSR) testinput/output yaml and ctest consiste…
gthompsnWRF Mar 16, 2022
71df5a2
new addition of SSMIS testinput/output yaml and ctest consistent with…
gthompsnWRF Mar 16, 2022
c04b0bc
new addition of IASI testinput/output yaml and ctest consistent with …
gthompsnWRF Mar 16, 2022
d47bc24
remove the v2 reference of filename because we will be using JEDI Dat…
gthompsnWRF Mar 16, 2022
333bdce
Merge branch 'feature/sprint-ioda-converters' into feature/sprint_iod…
rhoneyager Mar 16, 2022
b24882c
eliminate channelBlock, scaleFactor, and start/end channel info for now
gthompsnWRF Mar 18, 2022
97c16f4
Merge branch 'feature/sprint_ioda_greg_bufr2ioda3' of https://github.…
gthompsnWRF Mar 18, 2022
06db436
fix up the test ref files
gthompsnWRF Mar 18, 2022
54cf93f
try fixing merge conflict on MHS data file
gthompsnWRF Mar 24, 2022
c4b0d85
major reduction in file sizes
gthompsnWRF Mar 24, 2022
3b5bc81
fix test filenames and such
gthompsnWRF Mar 29, 2022
73d6fc3
add in a missing test ref file
gthompsnWRF Mar 29, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
112 changes: 112 additions & 0 deletions doc/bufr_table_samples/satellite_IASI_mnemonics.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,112 @@
Here is the DX table that was generated:

.------------------------------------------------------------------------------.
| ------------ USER DEFINITIONS FOR TABLE-A TABLE-B TABLE D -------------- |
|------------------------------------------------------------------------------|
| MNEMONIC | NUMBER | DESCRIPTION |
|----------|--------|----------------------------------------------------------|
| | | |
| NC021241 | A61207 | MTYP 021-241 IASI 1C RADIANCES (VARIABLE CHNS) (METOP) |
| | | |
| NC021241 | SAID GCLONG SIID SCLF YEAR MNTH DAYS HOUR MINU 202131 |
| NC021241 | 201138 SECO 201000 202000 CLATH CLONH SAZA BEARAZ SOZA |
| NC021241 | SOLAZI FOVN ORBN 201133 SLNM 201000 201132 MJFC 201000 |
| NC021241 | 202126 SELV 202000 QGFQ QGQI QGQIL QGQIR QGQIS QGSSQ |
| NC021241 | "IASIL1CB"10 (IASICHN) SIID AVHCST "IASIL1CS"7 |
| | | |
| IASIL1CB | 340002 | IASI LEVEL 1C BAND DESCRIPTION SEQUENCE |
| IASIL1CS | 340004 | IASI LEVEL 1C AVHRR SINGLE SCENE SEQUENCE |
| IASICHN | 361137 | IASI LEVEL 1C SCALED RADIANCE SEQUENCE |
| | |
| IASIL1CB | STCH ENCH CHSF |
| | |
| IASIL1CS | YAPCG ZAPCG FCPH "AVHRCHN"6 |
| | |
| AVHRCHN | 361139 | IASI LEVEL 1C MEAN AND STANDARD DEVIATION RADIANCE SEQ |
| AVHRCHN | CHNM CHSF SMRA CHSF SSDR |
| | |
| IASICHN | 201136 CHNM 201000 SCRA |
| | | |
| SAID | 001007 | SATELLITE IDENTIFIER |
| GCLONG | 001031 | ORIGINATING/GENERATING CENTER |
| SIID | 002019 | SATELLITE INSTRUMENTS |
| SCLF | 002020 | SATELLITE CLASSIFICATION |
| YEAR | 004001 | YEAR |
| MNTH | 004002 | MONTH |
| DAYS | 004003 | DAY |
| HOUR | 004004 | HOUR |
| MINU | 004005 | MINUTE |
| SECO | 004006 | SECOND |
| CLATH | 005001 | LATITUDE (HIGH ACCURACY) |
| CLONH | 006001 | LONGITUDE (HIGH ACCURACY) |
| SAZA | 007024 | SATELLITE ZENITH ANGLE |
| SOZA | 007025 | SOLAR ZENITH ANGLE |
| BEARAZ | 005021 | BEARING OR AZIMUTH |
| SOLAZI | 005022 | SOLAR AZIMUTH |
| FOVN | 005043 | FIELD OF VIEW NUMBER |
| ORBN | 005040 | ORBIT NUMBER |
| SLNM | 005041 | SCAN LINE NUMBER |
| MJFC | 025070 | MAJOR FRAME COUNT |
| SELV | 007001 | HEIGHT OF STATION |
| QGFQ | 033060 | INDIVIDUAL IASI-SYSTEM QUALITY FLAG |
| QGQI | 033061 | INSTR. NOISE PERF. INDICATOR (SPECTRAL & RADIOMETRIC) |
| QGQIL | 033062 | GEOMETRIC QUALITY INDEX INDICATOR |
| QGQIR | 033063 | INSTR. NOISE PERF. INDICATOR (RADIOMETRIC CALIBRATION) |
| QGQIS | 033064 | INSTRUMENT NOISE PERF. INDICATOR (SPECTRAL CALIBRATION) |
| QGSSQ | 033065 | OUTPUT OF TEC FUNCTION |
| AVHCST | 025051 | AVHRR CHANNEL COMBINATION |
| STCH | 025140 | START CHANNEL |
| ENCH | 025141 | END CHANNEL |
| CHSF | 025142 | CHANNEL SCALE FACTOR |
| YAPCG | 005060 | Y ANGULAR POSITION OF CENTER OF GRAVITY |
| ZAPCG | 005061 | Z ANGULAR POSITION OF CENTER OF GRAVITY |
| FCPH | 025085 | FRACTION OF CLEAR PIXELS IN HIRS FOV |
| CHNM | 005042 | CHANNEL NUMBER |
| SMRA | 014047 | SCALED MEAN AVHRR RADIANCE |
| SSDR | 014048 | SCALED STANDARD DEVIATION OF AVHRR RADIANCE |
| SCRA | 014046 | SCALED IASI RADIANCE |
| | |
|------------------------------------------------------------------------------|
| MNEMONIC | SCAL | REFERENCE | BIT | UNITS |-------------|
|----------|------|-------------|-----|--------------------------|-------------|
| | | | | |-------------|
| SAID | 0 | 0 | 10 | CODE TABLE |-------------|
| GCLONG | 0 | 0 | 16 | CODE TABLE |-------------|
| SIID | 0 | 0 | 11 | CODE TABLE |-------------|
| SCLF | 0 | 0 | 9 | CODE TABLE |-------------|
| YEAR | 0 | 0 | 12 | YEAR |-------------|
| MNTH | 0 | 0 | 4 | MONTH |-------------|
| DAYS | 0 | 0 | 6 | DAY |-------------|
| HOUR | 0 | 0 | 5 | HOUR |-------------|
| MINU | 0 | 0 | 6 | MINUTE |-------------|
| SECO | 0 | 0 | 6 | SECOND |-------------|
| CLATH | 5 | -9000000 | 25 | DEGREE |-------------|
| CLONH | 5 | -18000000 | 26 | DEGREE |-------------|
| SAZA | 2 | -9000 | 15 | DEGREE |-------------|
| SOZA | 2 | -9000 | 15 | DEGREE |-------------|
| BEARAZ | 2 | 0 | 16 | DEGREE TRUE |-------------|
| SOLAZI | 2 | 0 | 16 | DEGREE TRUE |-------------|
| FOVN | 0 | 0 | 8 | NUMERIC |-------------|
| ORBN | 0 | 0 | 24 | NUMERIC |-------------|
| SLNM | 0 | 0 | 8 | NUMERIC |-------------|
| MJFC | 0 | 0 | 4 | NUMERIC |-------------|
| SELV | 0 | -400 | 15 | M |-------------|
| QGFQ | 0 | 0 | 2 | CODE TABLE |-------------|
| QGQI | 0 | 0 | 7 | % |-------------|
| QGQIL | 0 | 0 | 7 | % |-------------|
| QGQIR | 0 | 0 | 7 | % |-------------|
| QGQIS | 0 | 0 | 7 | % |-------------|
| QGSSQ | 0 | 0 | 24 | NUMERIC |-------------|
| AVHCST | 0 | 0 | 7 | FLAG TABLE |-------------|
| STCH | 0 | 0 | 14 | NUMERIC |-------------|
| ENCH | 0 | 0 | 14 | NUMERIC |-------------|
| CHSF | 0 | 0 | 6 | NUMERIC |-------------|
| YAPCG | 6 | -8000000 | 24 | DEGREE |-------------|
| ZAPCG | 6 | -8000000 | 24 | DEGREE |-------------|
| FCPH | 0 | 0 | 7 | NUMERIC |-------------|
| CHNM | 0 | 0 | 6 | NUMERIC |-------------|
| SMRA | 0 | 0 | 31 | W M**-2 SR**-1 M |-------------|
| SSDR | 0 | 0 | 31 | W M**-2 SR**-1 M |-------------|
| SCRA | 0 | 0 | 16 | W M**-2 SR**-1 M |-------------|
| | | | | |-------------|
`------------------------------------------------------------------------------'
98 changes: 97 additions & 1 deletion test/CMakeLists.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -136,13 +136,29 @@ if( iodaconv_bufr_ENABLED )
testinput/bufr_tables
testinput/gdas.t18z.1bmhs.tm00.bufr_d
testinput/gdas.t00z.1bhrs4.tm00.bufr_d
testinput/gdas.t00z.airsev.tm00.bufr_d
testinput/gdas.t00z.1bamua.tm00.bufr_d
testinput/gdas.t00z.avcsam.tm00.bufr_d
testinput/gdas.t00z.atms.tm00.bufr_d
Comment thread
gthompsnJCSDA marked this conversation as resolved.
testinput/gdas.t06z.adpsfc.tm00.bufr_d
testinput/gdas.t00z.crisf4.tm00.bufr_d
testinput/gdas.t00z.mtiasi.tm00.bufr_d
testinput/gdas.t00z.sevcsr.tm00.bufr_d
testinput/gdas.t00z.ssmisu.tm00.bufr_d
Comment thread
gthompsnJCSDA marked this conversation as resolved.
testinput/bufr_read_2_dim_blocks.bufr
testinput/bufr_read_wmo_radiosonde.bufr
testinput/bufr_satwnd_new_format.bufr
testinput/bufr_satwnd_old_format.bufr
testinput/bufr_mhs.yaml
testinput/bufr_airs.yaml
testinput/bufr_amsua.yaml
testinput/bufr_atms.yaml
testinput/bufr_avhrr.yaml
testinput/bufr_cris.yaml
testinput/bufr_hrs.yaml
testinput/bufr_iasi.yaml
testinput/bufr_mhs.yaml
testinput/bufr_seviri.yaml
testinput/bufr_ssmis.yaml
testinput/bufr_filtering.yaml
testinput/bufr_splitting.yaml
testinput/bufr_filter_split.yaml
Expand Down Expand Up @@ -184,8 +200,16 @@ if( iodaconv_bufr_ENABLED )
testoutput/gdas.t18z.1bmhs.tm00.filtering.nc
testoutput/gdas.t18z.1bmhs.tm00.15.seven.split.nc
testoutput/gdas.t18z.1bmhs.tm00.15.7.filter_split.nc
testoutput/gdas.t00z.airsev.tm00.nc
testoutput/gdas.t00z.1bamsua.tm00.bufr_d
testoutput/gdas.t00z.avcsam.tm00.nc
testoutput/gdas.t00z.atms.tm00.nc
testoutput/gdas.t06z.adpsfc.tm00.nc
testoutput/gdas.t06z.adpsfc_snow.tm00.nc
testoutput/gdas.t00z.crisf4.tm00.nc
testoutput/gdas.t00z.mtiasi.tm00.nc
testoutput/gdas.t00z.sevcsr.tm00.nc
testoutput/gdas.t00z.ssmisu.tm00.nc
testoutput/bufr_read_2_dim_blocks.nc
testoutput/bufr_read_wmo_radiosonde.nc
testoutput/NC005031.nc
Expand Down Expand Up @@ -840,6 +864,51 @@ if(iodaconv_bufr_ENABLED)
gdas.t18z.1bmhs.tm00.nc ${IODA_CONV_COMP_TOL_ZERO}
DEPENDS bufr2ioda.x )

ecbuild_add_test( TARGET test_iodaconv_bufr_airs2ioda
TYPE SCRIPT
COMMAND bash
ARGS ${CMAKE_BINARY_DIR}/bin/iodaconv_comp.sh
netcdf
"${CMAKE_BINARY_DIR}/bin/bufr2ioda.x testinput/bufr_airs.yaml"
gdas.t00z.airsev.tm00.nc ${IODA_CONV_COMP_TOL_ZERO}
DEPENDS bufr2ioda.x )

ecbuild_add_test( TARGET test_iodaconv_bufr_amsua2ioda
Comment thread
rhoneyager marked this conversation as resolved.
TYPE SCRIPT
COMMAND bash
ARGS ${CMAKE_BINARY_DIR}/bin/iodaconv_comp.sh
netcdf
"${CMAKE_BINARY_DIR}/bin/bufr2ioda.x testinput/bufr_amsua.yaml"
gdas.t00z.1bamsua.tm00.nc ${IODA_CONV_COMP_TOL_ZERO}
DEPENDS bufr2ioda.x )

ecbuild_add_test( TARGET test_iodaconv_bufr_atmsioda
TYPE SCRIPT
COMMAND bash
ARGS ${CMAKE_BINARY_DIR}/bin/iodaconv_comp.sh
netcdf
"${CMAKE_BINARY_DIR}/bin/bufr2ioda.x testinput/bufr_atms.yaml"
gdas.t00z.atms.tm00.nc ${IODA_CONV_COMP_TOL_ZERO}
DEPENDS bufr2ioda.x )

ecbuild_add_test( TARGET test_iodaconv_bufr_avhrr2ioda
Comment thread
rhoneyager marked this conversation as resolved.
TYPE SCRIPT
COMMAND bash
ARGS ${CMAKE_BINARY_DIR}/bin/iodaconv_comp.sh
netcdf
"${CMAKE_BINARY_DIR}/bin/bufr2ioda.x testinput/bufr_avhrr.yaml"
gdas.t00z.avcsam.tm00.nc ${IODA_CONV_COMP_TOL_ZERO}
DEPENDS bufr2ioda.x )

ecbuild_add_test( TARGET test_iodaconv_bufr_cris2ioda
TYPE SCRIPT
COMMAND bash
ARGS ${CMAKE_BINARY_DIR}/bin/iodaconv_comp.sh
netcdf
"${CMAKE_BINARY_DIR}/bin/bufr2ioda.x testinput/bufr_cris.yaml"
gdas.t00z.crisf4.tm00.nc ${IODA_CONV_COMP_TOL_ZERO}
DEPENDS bufr2ioda.x )

ecbuild_add_test( TARGET test_iodaconv_bufr_hrs2ioda
TYPE SCRIPT
COMMAND bash
Expand All @@ -849,6 +918,33 @@ if(iodaconv_bufr_ENABLED)
gdas.t00z.1bhrs4.tm00.nc ${IODA_CONV_COMP_TOL_ZERO}
DEPENDS bufr2ioda.x )

ecbuild_add_test( TARGET test_iodaconv_bufr_iasi2ioda
TYPE SCRIPT
COMMAND bash
ARGS ${CMAKE_BINARY_DIR}/bin/iodaconv_comp.sh
netcdf
"${CMAKE_BINARY_DIR}/bin/bufr2ioda.x testinput/bufr_iasi.yaml"
gdas.t00z.mtiasi.tm00.nc ${IODA_CONV_COMP_TOL_ZERO}
DEPENDS bufr2ioda.x )

ecbuild_add_test( TARGET test_iodaconv_bufr_seviri2ioda
TYPE SCRIPT
COMMAND bash
ARGS ${CMAKE_BINARY_DIR}/bin/iodaconv_comp.sh
netcdf
"${CMAKE_BINARY_DIR}/bin/bufr2ioda.x testinput/bufr_seviri.yaml"
gdas.t00z.sevcsr.tm00.nc ${IODA_CONV_COMP_TOL_ZERO}
DEPENDS bufr2ioda.x )

ecbuild_add_test( TARGET test_iodaconv_bufr_ssmis2ioda
Comment thread
rhoneyager marked this conversation as resolved.
TYPE SCRIPT
COMMAND bash
ARGS ${CMAKE_BINARY_DIR}/bin/iodaconv_comp.sh
netcdf
"${CMAKE_BINARY_DIR}/bin/bufr2ioda.x testinput/bufr_sssmis.yaml"
gdas.t00z.ssmisu.tm00.nc ${IODA_CONV_COMP_TOL_ZERO}
DEPENDS bufr2ioda.x )

ecbuild_add_test( TARGET test_iodaconv_bufr_filtering
TYPE SCRIPT
COMMAND bash
Expand Down
Loading