Skip to content

SPRINT: IODA-converters staging branch #817

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
PatNichols merged 62 commits into from
Jan 18, 2023
Merged

SPRINT: IODA-converters staging branch #817

Show file tree
Hide file tree
Changes from 43 commits
Commits
Show all changes
62 commits
Select commit Hold shift + click to select a range
1a7cd53
Staging branch
rhoneyager Mar 7, 2022
59a9f1e
Retarget ioda clone to the sprint branch
rhoneyager Mar 8, 2022
b19a123
Merge branch 'develop' into feature/sprint-ioda-converters
rhoneyager Mar 8, 2022
83c521b
Merge branch 'develop' into feature/sprint-ioda-converters
rhoneyager Mar 8, 2022
38a37f0
nlocs to locations (#825)
CoryMartin-NOAA Mar 8, 2022
80b45dd
Merge branch 'develop' into feature/sprint-ioda-converters
rhoneyager Mar 10, 2022
30a65aa
SPRINT: Change METAR converter to align variable names with JEDI nami…
gthompsnJCSDA Mar 10, 2022
5bf58ef
Convert ISO date string to Python datetime object (#840)
CoryMartin-NOAA Mar 11, 2022
8d967fe
Merge branch 'develop' into feature/sprint-ioda-converters
rhoneyager Mar 11, 2022
b272ba7
Feature/ioda sprint atms hdf5 (#844)
BenjaminRuston Mar 16, 2022
1774305
Sprint: Upgrade converters that use bufr2ioda.x to follow the JEDI Da…
gthompsnJCSDA Mar 16, 2022
2f8ae50
SPRINT: Update a series of converters (bufr2ioda.x, see list below) …
gthompsnJCSDA Mar 16, 2022
cc8ccca
Merge branch 'develop' into feature/sprint-ioda-converters
srherbener Mar 16, 2022
b36c190
Modify gsidiag converter to follow JEDI data convention and update th…
emilyhcliu Mar 16, 2022
72b9fe2
Feature/sprint ioda converters yan (#853)
YanHao-NOAA Mar 16, 2022
e4e69b8
SPRINT: Upgrade conventional data BUFR-to-IODA converters (shown belo…
gthompsnJCSDA Mar 18, 2022
6a1ef48
Merge branch 'develop' into feature/sprint-ioda-converters
srherbener Mar 21, 2022
a11b680
Create Feature/sprint ghcn for 6 land converters during the ioda-conv…
YoulongXia-NOAA Mar 22, 2022
13b6a41
Feature/argo clim2ioda changes ne (#857)
nicholasesposito Mar 22, 2022
dfc14bb
Modify marine/smos_sss, marine/smap_sss, and marine/hgodas_sst conve…
PraveenKumar-NOAA Mar 22, 2022
215e42f
Merge branch 'develop' into feature/sprint-ioda-converters
rhoneyager Mar 22, 2022
c2c7fec
Feature/sprint modis aod converters hb (#851)
bhuang95 Mar 23, 2022
cceb60e
Remove two land converters with grib input (#883)
YoulongXia-NOAA Mar 23, 2022
d1c60a7
SPRINT: Upgrade many converters of satellite data bufr2ioda.x to ali…
gthompsnJCSDA Mar 29, 2022
b9041ee
Feature/sprint ioda greg bufr2ioda3 (#891)
gthompsnJCSDA Mar 30, 2022
91b728e
SPRINT: upgrade radiosonde python-eccodes bufr (WMO) converter to JE…
gthompsnJCSDA Apr 7, 2022
c07577f
Merge branch 'develop' into feature/sprint-ioda-converters
srherbener Apr 13, 2022
e5dd0a3
Feature/sprint aeronet converters hb (#849)
bhuang95 Apr 13, 2022
e566512
Merge branch 'develop' into feature/sprint-ioda-converters
srherbener Apr 13, 2022
5cd9713
Feature/sprint viirs aod converters (#850)
bhuang95 Apr 13, 2022
75848f7
Sprint: Update the GODAE (marine) converters listed below to adopt t…
gthompsnJCSDA Apr 14, 2022
45a0af1
Feature/sprint ioda converters fcvdb (#848)
fcvdb Apr 14, 2022
5c41d2b
Update three marine converters for ioda-converters code sprint (#896)
YoulongXia-NOAA Apr 14, 2022
d509115
Merge branch 'develop' into feature/sprint-ioda-converters
srherbener Apr 15, 2022
cc6f944
SPRINT: update 3 marine converters to follow JEDI Data Conventions (…
gthompsnJCSDA Apr 15, 2022
daa1dca
Update the SatWinds AMV (from SSEC) ascii2ioda convert to follow JEDI…
gthompsnJCSDA Apr 18, 2022
52bbd7e
Remove bias files from validation
rhoneyager Apr 18, 2022
016ab9a
Merge branch 'develop' into feature/sprint-ioda-converters
srherbener Apr 19, 2022
0789858
Update gsidiag converter conventional data and satellite winds to com…
emilyhcliu Apr 20, 2022
4481007
Bugfix/ascii satwind datetime fix for SPRINT (#926)
BenjaminRuston Apr 28, 2022
285883a
Feature/sprint ioda converters ne viirs modis l2 oc2ioda (#873)
nicholasesposito May 4, 2022
f44990d
Feature/sprint ioda converters ne viirs modis l3 oc2ioda (#929)
nicholasesposito May 5, 2022
44de6a1
Sprint: update src/marine/gds2_sst2ioda.py code for JEDI Data Convent…
gthompsnJCSDA May 6, 2022
c59882c
Changes to resolve conflicts with develop branch.
srherbener May 12, 2022
ae108a9
Merge branch 'develop' into feature/sprint-ioda-converters
srherbener May 12, 2022
4504cff
Fixed errors introduced with the prior commit containing the resoluti…
srherbener May 12, 2022
e8324ae
Updated smap9k test reference file.
srherbener May 13, 2022
7bf40a0
Removed units attribute from vegetationOpacity variable according to …
srherbener May 18, 2022
f15fc68
Merge branch 'develop' into feature/sprint-ioda-converters
srherbener May 28, 2022
e60ba6c
Update sfc_tv_obs_2018041500.nc4 to be consistent with #953 (#954)
FanHan-NOAA Jun 1, 2022
6a82ed7
Make dateTime fill value consistent across platforms (#968)
srherbener Jun 24, 2022
a941c9c
When BUFR-formatted sonde data has "subsets" the prior code wasn't ab…
gthompsnJCSDA Jul 19, 2022
ab14da4
Merge branch 'develop' into feature/sprint-ioda-converters
srherbener Aug 1, 2022
669d6ba
Needed to go with feature/sprint-ioda-converters branch code for sett…
srherbener Aug 1, 2022
62f4105
Update the sprint branch with as much of ```develop``` branch as poss…
gthompsnJCSDA Nov 14, 2022
be4cd94
Sprint: Updating two GOES and three marine converters for naming con…
gthompsnJCSDA Jan 3, 2023
4e93cfb
Merge branch 'develop' into feature/sprint-ioda-converters
srherbener Jan 6, 2023
3a7af98
More cleanup from merging in develop.
srherbener Jan 6, 2023
47d3397
Update the sprint branch again with develop changes mostly for BUFR (…
gthompsnJCSDA Jan 11, 2023
198310f
Fix a collection of bufr2ioda.x YAML files and test-ref files for nam…
gthompsnJCSDA Jan 11, 2023
ec47029
make L2 trace gas converters work and be consistent with what the upg…
Jan 12, 2023
48eacbe
changes for L2 trace gas varnames to follow conventions (#1145)
Jan 12, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 0 additions & 16 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -106,22 +106,6 @@ For method option (-m) of bias and uncertainty calculation (default/nesdis), dea

The land converters include all converter scripts for snowpack, soil, vegeation, and the other surface related land variables.

For snow cover fraction(scf), IMS grib2 files are supported with `ims_scf2ioda.py`.
```
Usage: ims_scf2ioda.py -i input_ims_file.grib2 -o output_ioda_file.nc -m maskout
```
For -i you can specify an input file and the converter will write it to one output file. For maskout option (-m) default/maskout, default means to keep all missing values and maskout means to not write out missing values.


For snow depth (snod), afwa grib1 files are supported with `afwa_snod2ioda.py`.
```
Usage: afwa_snod2ioda.py -i input_afwa_file.grb -o output_ioda_file.nc -m maskout
```
For -i you can specify an input file and the converter will write it to one output file. For maskout option (-m) default/maskout, default means to keep all missing values and maskout means to not write out missing values.

It should be noted that both ims_scf2ioda.py and afwa_snod2ioda.py are depending on the python pygrib module. To enable the testing of these two scripts when the user has a pygrib module available, during the ecbuild process, please add -DUSE_PYGRIB=True to the ecbuild command line.


For snow depth (snod), GHCN csv files are supported with `ghcn_snod2ioda.py`.
```
Usage: ghcn_snod2ioda.py -i input_ghcn_file.csv -o output_ioda_file.nc -f ghcn_station.txt -d YYYYMMDD -m maskout
Expand Down
238 changes: 238 additions & 0 deletions doc/bufr_obs_samples/sonde_bufr_sample.txt
View file
Open in desktop

Large diffs are not rendered by default.

112 changes: 112 additions & 0 deletions doc/bufr_table_samples/satellite_IASI_mnemonics.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,112 @@
Here is the DX table that was generated:

.------------------------------------------------------------------------------.
| ------------ USER DEFINITIONS FOR TABLE-A TABLE-B TABLE D -------------- |
|------------------------------------------------------------------------------|
| MNEMONIC | NUMBER | DESCRIPTION |
|----------|--------|----------------------------------------------------------|
| | | |
| NC021241 | A61207 | MTYP 021-241 IASI 1C RADIANCES (VARIABLE CHNS) (METOP) |
| | | |
| NC021241 | SAID GCLONG SIID SCLF YEAR MNTH DAYS HOUR MINU 202131 |
| NC021241 | 201138 SECO 201000 202000 CLATH CLONH SAZA BEARAZ SOZA |
| NC021241 | SOLAZI FOVN ORBN 201133 SLNM 201000 201132 MJFC 201000 |
| NC021241 | 202126 SELV 202000 QGFQ QGQI QGQIL QGQIR QGQIS QGSSQ |
| NC021241 | "IASIL1CB"10 (IASICHN) SIID AVHCST "IASIL1CS"7 |
| | | |
| IASIL1CB | 340002 | IASI LEVEL 1C BAND DESCRIPTION SEQUENCE |
| IASIL1CS | 340004 | IASI LEVEL 1C AVHRR SINGLE SCENE SEQUENCE |
| IASICHN | 361137 | IASI LEVEL 1C SCALED RADIANCE SEQUENCE |
| | |
| IASIL1CB | STCH ENCH CHSF |
| | |
| IASIL1CS | YAPCG ZAPCG FCPH "AVHRCHN"6 |
| | |
| AVHRCHN | 361139 | IASI LEVEL 1C MEAN AND STANDARD DEVIATION RADIANCE SEQ |
| AVHRCHN | CHNM CHSF SMRA CHSF SSDR |
| | |
| IASICHN | 201136 CHNM 201000 SCRA |
| | | |
| SAID | 001007 | SATELLITE IDENTIFIER |
| GCLONG | 001031 | ORIGINATING/GENERATING CENTER |
| SIID | 002019 | SATELLITE INSTRUMENTS |
| SCLF | 002020 | SATELLITE CLASSIFICATION |
| YEAR | 004001 | YEAR |
| MNTH | 004002 | MONTH |
| DAYS | 004003 | DAY |
| HOUR | 004004 | HOUR |
| MINU | 004005 | MINUTE |
| SECO | 004006 | SECOND |
| CLATH | 005001 | LATITUDE (HIGH ACCURACY) |
| CLONH | 006001 | LONGITUDE (HIGH ACCURACY) |
| SAZA | 007024 | SATELLITE ZENITH ANGLE |
| SOZA | 007025 | SOLAR ZENITH ANGLE |
| BEARAZ | 005021 | BEARING OR AZIMUTH |
| SOLAZI | 005022 | SOLAR AZIMUTH |
| FOVN | 005043 | FIELD OF VIEW NUMBER |
| ORBN | 005040 | ORBIT NUMBER |
| SLNM | 005041 | SCAN LINE NUMBER |
| MJFC | 025070 | MAJOR FRAME COUNT |
| SELV | 007001 | HEIGHT OF STATION |
| QGFQ | 033060 | INDIVIDUAL IASI-SYSTEM QUALITY FLAG |
| QGQI | 033061 | INSTR. NOISE PERF. INDICATOR (SPECTRAL & RADIOMETRIC) |
| QGQIL | 033062 | GEOMETRIC QUALITY INDEX INDICATOR |
| QGQIR | 033063 | INSTR. NOISE PERF. INDICATOR (RADIOMETRIC CALIBRATION) |
| QGQIS | 033064 | INSTRUMENT NOISE PERF. INDICATOR (SPECTRAL CALIBRATION) |
| QGSSQ | 033065 | OUTPUT OF TEC FUNCTION |
| AVHCST | 025051 | AVHRR CHANNEL COMBINATION |
| STCH | 025140 | START CHANNEL |
| ENCH | 025141 | END CHANNEL |
| CHSF | 025142 | CHANNEL SCALE FACTOR |
| YAPCG | 005060 | Y ANGULAR POSITION OF CENTER OF GRAVITY |
| ZAPCG | 005061 | Z ANGULAR POSITION OF CENTER OF GRAVITY |
| FCPH | 025085 | FRACTION OF CLEAR PIXELS IN HIRS FOV |
| CHNM | 005042 | CHANNEL NUMBER |
| SMRA | 014047 | SCALED MEAN AVHRR RADIANCE |
| SSDR | 014048 | SCALED STANDARD DEVIATION OF AVHRR RADIANCE |
| SCRA | 014046 | SCALED IASI RADIANCE |
| | |
|------------------------------------------------------------------------------|
| MNEMONIC | SCAL | REFERENCE | BIT | UNITS |-------------|
|----------|------|-------------|-----|--------------------------|-------------|
| | | | | |-------------|
| SAID | 0 | 0 | 10 | CODE TABLE |-------------|
| GCLONG | 0 | 0 | 16 | CODE TABLE |-------------|
| SIID | 0 | 0 | 11 | CODE TABLE |-------------|
| SCLF | 0 | 0 | 9 | CODE TABLE |-------------|
| YEAR | 0 | 0 | 12 | YEAR |-------------|
| MNTH | 0 | 0 | 4 | MONTH |-------------|
| DAYS | 0 | 0 | 6 | DAY |-------------|
| HOUR | 0 | 0 | 5 | HOUR |-------------|
| MINU | 0 | 0 | 6 | MINUTE |-------------|
| SECO | 0 | 0 | 6 | SECOND |-------------|
| CLATH | 5 | -9000000 | 25 | DEGREE |-------------|
| CLONH | 5 | -18000000 | 26 | DEGREE |-------------|
| SAZA | 2 | -9000 | 15 | DEGREE |-------------|
| SOZA | 2 | -9000 | 15 | DEGREE |-------------|
| BEARAZ | 2 | 0 | 16 | DEGREE TRUE |-------------|
| SOLAZI | 2 | 0 | 16 | DEGREE TRUE |-------------|
| FOVN | 0 | 0 | 8 | NUMERIC |-------------|
| ORBN | 0 | 0 | 24 | NUMERIC |-------------|
| SLNM | 0 | 0 | 8 | NUMERIC |-------------|
| MJFC | 0 | 0 | 4 | NUMERIC |-------------|
| SELV | 0 | -400 | 15 | M |-------------|
| QGFQ | 0 | 0 | 2 | CODE TABLE |-------------|
| QGQI | 0 | 0 | 7 | % |-------------|
| QGQIL | 0 | 0 | 7 | % |-------------|
| QGQIR | 0 | 0 | 7 | % |-------------|
| QGQIS | 0 | 0 | 7 | % |-------------|
| QGSSQ | 0 | 0 | 24 | NUMERIC |-------------|
| AVHCST | 0 | 0 | 7 | FLAG TABLE |-------------|
| STCH | 0 | 0 | 14 | NUMERIC |-------------|
| ENCH | 0 | 0 | 14 | NUMERIC |-------------|
| CHSF | 0 | 0 | 6 | NUMERIC |-------------|
| YAPCG | 6 | -8000000 | 24 | DEGREE |-------------|
| ZAPCG | 6 | -8000000 | 24 | DEGREE |-------------|
| FCPH | 0 | 0 | 7 | NUMERIC |-------------|
| CHNM | 0 | 0 | 6 | NUMERIC |-------------|
| SMRA | 0 | 0 | 31 | W M**-2 SR**-1 M |-------------|
| SSDR | 0 | 0 | 31 | W M**-2 SR**-1 M |-------------|
| SCRA | 0 | 0 | 16 | W M**-2 SR**-1 M |-------------|
| | | | | |-------------|
`------------------------------------------------------------------------------'
Loading