Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Handle requests.exceptions.SSLError in manager._slack_notification() #59

Open
douglatornell opened this issue Nov 6, 2024 · 0 comments
Assignees
Labels
bug Something isn't working
Milestone

Comments

@douglatornell
Copy link
Member

An unhandled requests.exceptions.SSLError exception in manager._slack_notification() caused the SalishSeaNowcast
manager to crash and restart in such a way that processing didn't continue and completed collect_weather and crop_gribs workers were left running.

We should catch the exception and emit an ERROR level log message so that processing continues but automation administrators are notified by email.

Traceback (most recent call last):
  File "/SalishSeaCast/NEMO_Nowcast/nemo_nowcast/manager.py", line 267, in _process_messages
    self._try_messages()
  File "/SalishSeaCast/NEMO_Nowcast/nemo_nowcast/manager.py", line 287, in _try_messages
    reply, next_workers = self._message_handler(message)
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/SalishSeaCast/NEMO_Nowcast/nemo_nowcast/manager.py", line 309, in _message_handler
    reply, next_workers = self._handle_continue_msg(msg)
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/SalishSeaCast/NEMO_Nowcast/nemo_nowcast/manager.py", line 357, in _handle_continue_msg
    self._slack_notification(msg)
  File "/SalishSeaCast/NEMO_Nowcast/nemo_nowcast/manager.py", line 467, in _slack_notification
    requests.post(slack_url, json=slack_msg)
  File "/SalishSeaCast/nowcast-env/lib/python3.12/site-packages/requests/api.py", line 115, in post
    return request("post", url, data=data, json=json, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/SalishSeaCast/nowcast-env/lib/python3.12/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/SalishSeaCast/nowcast-env/lib/python3.12/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/SalishSeaCast/nowcast-env/lib/python3.12/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/SalishSeaCast/nowcast-env/lib/python3.12/site-packages/requests/adapters.py", line 698, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='hooks.slack.com', port=443): Max retries exceeded with url: /services/TFR25L4LU/BFXM1R6AK/qJyGMDo9KPEXIwJob0SwuxD3 (Caused by SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1000)')))
@douglatornell douglatornell added the bug Something isn't working label Nov 6, 2024
@douglatornell douglatornell added this to the v24.1 milestone Nov 6, 2024
@douglatornell douglatornell modified the milestones: v24.1, v25.1 Nov 27, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant