Noaaweather

[rewsssiestedbo]

Description:

Add sensor to get weather measurements from US NOAA/National Weather Service

Related issue (if applicable): fixes #

Pull request with documentation for home-assistant.io (if applicable): home-assistant/home-assistant.io#<home-assistant.io PR number goes here>
Not yet submitted

Example entry for configuration.yaml (if applicable):

sensor:
  - platform: noaaweather
    name: Kailua NWS conditions
    latitude: 21.4131155
    longitude: -157.7573696
    monitored_conditions:
        - temperature
        - textDescription
        - dewpoint
        - windChill
        - windSpeed
        - windDirection
        - windGust
        - barometricPressure
        - seaLevelPressure
        - precipitationLastHour
        - relativeHumidity
        - visibility
        - cloudLayers

Checklist:

• The code change is tested and works locally.
• Local tests pass with tox. Your PR cannot be merged unless tests pass
• There is no commented out code in this PR.
• I have followed the development checklist

If user exposed functionality or configuration variables are added/changed:

• Documentation added/updated in home-assistant.io

If the code communicates with devices, web services, or third-party tools:

• The manifest file has all fields filled out correctly. Update and include derived files by running python3 -m script.hassfest.
• New or updated dependencies have been added to requirements_all.txt by running python3 -m script.gen_requirements_all.
• Untested files have been added to .coveragerc.

If the code does not interact with devices:

• Tests have been added to verify that the new code works.

[homeassistant]

Hi @rewsssiestedbo,

It seems you haven't yet signed a CLA. Please do so here.

Once you do that we will be able to review and accept this pull request.

Thanks!

[MatthewFlamm]

This needs to be considered along with the weather platform PR in #23647. I think there should be consistent naming and methodology. I propose NWS since it the sub-agency of NOAA that provides the data. I have no strong position here however.

There are also several different methods that were chosen in our approaches. For example, here, the HA condition is derived through some rules from the long text description while the weather platform PR uses the icon url provided in the nws data. These will sometimes give inconsistent results. I like the icon url approach as it is straightforward, but I'd like to hear if there is a benefit to your approach that I haven't considered.

Another example is that you use the raw metar code as the fallback for missing or low quality data. The weather platform PR does not use a fall back at all currently and in fact will use data marked as poor quality. I planned on falling back to the previous observation in this case, to be introduced in a future PR. We should align on a common fallback methodology.

I propose that we work together to iron out these differences, starting with the above two points, but I will read your PR in more detail.

[rewsssiestedbo]

This needs to be considered along with the weather platform PR in #23647. I think there should be consistent naming and methodology. I propose NWS since it the sub-agency of NOAA that provides the data. I have no strong position here however.

There are also several different methods that were chosen in our approaches. For example, here, the HA condition is derived through some rules from the long text description while the weather platform PR uses the icon url provided in the nws data. These will sometimes give inconsistent results. I like the icon url approach as it is straightforward, but I'd like to hear if there is a benefit to your approach that I haven't considered.

Another example is that you use the raw metar code as the fallback for missing or low quality data. The weather platform PR does not use a fall back at all currently and in fact will use data marked as poor quality. I planned on falling back to the previous observation in this case, to be introduced in a future PR. We should align on a common fallback methodology.

I propose that we work together to iron out these differences, starting with the above two points, but I will read your PR in more detail.

Yes, we should work together on this. I have no particular concern about using nws instead of noaa, If I recall, the only reason I went with noaa it seemed less likely to have a conflict with some other countries weather service abbreviation, and there was already the NOAA Tides sensor.
As far as using the icon URLs versus parsing the text description, it was not clear to me whether the icon URLs are at all stable long term. Overall the documentation on the observation values returned is fairly sparse, so it is not clear whether they have any commitment to keeping the icon URLs stable. The text description values have been stable for a long time, but not guaranteed either. If my code is coming up with different results from yours, I may have just interpreted something differently or made a different arbitrary choice: there is certainly not a one to one mapping to the HA condition values.

Finally, about dealing with missing information in responses. I initially just had the code to avoid updating a condition if the response was missing a value for that condition. However, for a few stations I was monitoring (PHNG and PHNL show this), there would be long periods with no valid temperature value (for example) in the temperature field, but with an updated value in the METAR record. It is not clear to me why this happens, but it is not unusual for there to be multiple observation records without valid values for a number of conditions, but with valid values in the METAR record.

I'll try to take a look are your PR and see how best to align.

[MatthewFlamm]

I've noticed too that sometimes some stations are better than others. The closest station to me is always sunny or cloudy, but never rainy. My nearby stations will only be missing/contain poor quality data for the current observation, however.

I think your metar fall back is the right approach for your case, but adds quite the complexity to the HA code base. It seems to me like it would be better to have these decisions unified in one module(or a new pypi library) rather than spread across two platforms. I need to think about the path forward, but one suggestion is to co-create a new pypi library that does all the data cleansing and presents HA with a very clean API.

For the icon, the big upside I see is that they publish a list of icons. So if they change the icon names and they continue to publish this list, it will be easy to maintain. Your method could be a nightmare if it ever changes. You'd have to deduce the new structure of the text.

[rewsssiestedbo]

I've noticed too that sometimes some stations are better than others. The closest station to me is always sunny or cloudy, but never rainy. My nearby stations will only be missing/contain poor quality data for the current observation, however.
I suspect that the closest station to you does not have a rain gauge. There are clearly a number of different types of stations: those at airports that are fully staffed with weather related personal and full sets of instruments; fully automated stations at smaller airports, which may or may not have a full set of instruments; stations run by organizations other than the NWS, which only have the instruments they care about. An example of the last case is GGWPW, which appears to have temperature, dewpoint, wind speed and direction and some precipitation detector. It appears to have no cloud detection, so it will never return cloudy, and I'm not sure if it ever gives anything other then Clear plus a wind indication.

I think your metar fall back is the right approach for your case, but adds quite the complexity to the HA code base. It seems to me like it would be better to have these decisions unified in one module(or a new pypi library) rather than spread across two platforms. I need to think about the path forward, but one suggestion is to co-create a new pypi library that does all the data cleansing and presents HA with a very clean API.
I tend to agree with you here. When I started I assumed that I could simply take the values and deal with unit conversion and be done. The complexity that is there now is a result of the incremental increases that came from realizing how messy the data really is. Creating an independent modules for getting the observation data and dealing with the mess can simplify the HA specific code enormously.
Given the organization of the NWS APIs, it would probably be best to have an observation station object model and an independent forecast object model, since the observations are dependent upon particular stations, while the forecast are for a specific grid point area, and they update at different rates.

For the icon, the big upside I see is that they publish a list of icons. So if they change the icon names and they continue to publish this list, it will be easy to maintain. Your method could be a nightmare if it ever changes. You'd have to deduce the new structure of the text.

Note that the translation to the HA weather code values was the last thing I added, and is only used as an option, since the NWS text descriptions have much more content than the HA codes, and I would rather have the direct NWS text.

Any thoughts on how to get started on creating the separate (from HA) module(s)?

For the forecast data, had you looked at using the NDFD API instead of the one you are using? It would be somewhat more complex to handle the XML responses, but it provides much more data, including precipitation amounts, dewpoint, cloud cover, etc.

[MatthewFlamm]

After reading through your PR again, it seems like the inconsistencies are mainly the ones I already mentioned, so I don't think we need a separate library for this. The biggest decision is to coalesce on the name in my mind. I will look into implementing a metar fallback in my PR. Do you have a reference for the qc code info you used?

For the forecast data, had you looked at using the NDFD API instead of the one you are using? It would be somewhat more complex to handle the XML responses, but it provides much more data, including precipitation amounts, dewpoint, cloud cover, etc.

I did look at it, but it again adds huge complexity. The weather.gov API is very simple and clean.

[rewsssiestedbo]

After reading through your PR again, it seems like the inconsistencies are mainly the ones I already mentioned, so I don't think we need a separate library for this. The biggest decision is to coalesce on the name in my mind. I will look into implementing a metar fallback in my PR. Do you have a reference for the qc code info you used?

I was never able to find any documentation on the qc codes. The comments in sensors.py are a result of reverse engineering/review of the results from a number of different stations over a few months.

For the HA conditions, it appears that the differences in the results from using the icons versus the text are for:

1. "lightning" versus "lightning-rainy", since I chose to have "Thunderstorm in vicinity" be treated as "lightning", and other values as "lightning-rainy".
2. "pouring" versus "rainy", since the NWS icons for rain and heavy rain are the same, but the text is different.
3. "hail" - I just added this, since it is not in the documentation page for a weather component, but is in the demo code.

As far as the name, it makes sense to use nws.

For the forecast data, had you looked at using the NDFD API instead of the one you are using? It would be somewhat more complex to handle the XML responses, but it provides much more data, including precipitation amounts, dewpoint, cloud cover, etc.

I did look at it, but it again adds huge complexity. The weather.gov API is very simple and clean.

[MatthewFlamm]

Some comments

[MatthewFlamm]

This whole function should use the built-in unit conversions, see pressure comment above.

[rewsssiestedbo]

Changed to use the built in unit conversions for temperature and pressure. As the distance one does not have inches for millimeters, and there are no speed conversions, I left those as is. In a future pull request I may try to add the builtin conversion for speed and the additional distance units.

[MatthewFlamm]

I added the metar fallback logic into the upstream pynws library with a new class. My implementation was simply to check for missing values in the observation. For missing values, replace with metar values, if available. I only put in the metar fallback for the weather platform variables (temperature, pressure, etc.) so far, but it is written in a way that should be straightforward for other variables if they are contained in the metar report and can be parsed with the metar library.

I'm willing to consider more complex logic in the pynws library, too if warranted.

[rewsssiestedbo]

Closing to wait for integration of nws weather component, and then integration of the sensor code with that component.