NCEI usage of VARIABLE:gts_ingest = boolean to determine GTS data

[kerfoot]

What should we add?

The IOOS Metadata Profile specification allows for the variable based gts_ingest attribute to mark a variable that should or should not be released to GTS.

I believe that currently the _NC_GLOBAL:gts_ingest global attribute is the only one used by NCEI to determine this and it marks an entire data set that should or should not be released to GTS.

For example, we have a dataset in which we noticed that the conductivity went bad, resulting in bad salinity profiles. As far as we can tell, the temperature data is good and can be used in models without the salinity. So we would like to have the temperature data sent, but not the salinity.

Are there plans to implement variable based GTS release and have NCEI use the VARIABLE:gts_ingest attribute to determine what variables are released to GTS and what variables are not?

Reference

No response

[leilabbb]

The GDAC can add gts_ingest to indicate variables intended for NDBC/GTS harvest. This requires the GDAC to implement another level of QC that verifies the QARTOD flags to assign the Boolean value True or False to the variable's gts_ingest attribute.

[kerfoot]

My understanding of the gts_ingest attribute (both global and variable based) is a boolean flag set by the operator to let the DAC/NDBC know that they want these variables transmitted to GTS. As such, I don't think it should be modified by the DAC based on the results of QC tests. Rather, the QC should be applied and then the primary_flag results should be used by NDBC to omit observations that fail the test.

This approach is in line with original design of the DAC in which no information in the files submitted by the data providers should be modified. Has this approach changed?

[mwengren]

I can offer some insight as far as the IOOS Metadata Profile goes. What @kerfoot says in relation to that is correct, there's a separation between the gts_ingest flag and results of QARTOD tests (i.e. a QARTOD process shouldn't make any changes to the gts_ingest flag, it should only output results to the relevant ancillary QC variables for an observed variable).

This was based on the assumption that NDBC would still want to ingest the data into their own database, but along with the QARTOD information, which would then be used to decide whether or not to submit it the GTS.

Not sure how that aligns with the GDAC process, but, ideally, we'd have some consistency in the approach between the two. FWIW, the IOOS MP references an ancillary variable of standard name aggregate_quality_flag rather than primary_flag, so it appears there's some difference there.

Details about the requirements for NDBC/GTS ingest are in the IOOS Metadata Profile at:

• https://ioos.github.io/ioos-metadata/ioos-metadata-profile-v1-2#for-ndbc-to-pull-data-for-a-dataset-to-the-gts
• https://ioos.github.io/ioos-metadata/ioos-metadata-profile-v1-2#requirements-for-the-qartod-aggregaterollup-flag