[Workaround Patch Available] Blocking call in aiohttp CookieJar causing stability issues

[jleinenbach]

Describe the bug

The alexa_media integration is causing blocking calls within the event loop due to synchronous file operations in aiohttp.cookiejar.CookieJar. Specifically, the methods save and load in CookieJar perform synchronous file I/O operations, leading to stability issues in Home Assistant and generating warnings about blocking calls to open.

To Reproduce

1. Install and configure the alexa_media integration in Home Assistant.
2. Start Home Assistant.
3. Monitor the logs for warnings or errors.
4. The following error messages appear, indicating blocking calls:

Detected blocking call to open with args (PosixPath('/config/.storage/[email protected]'),) in /usr/local/lib/python3.12/site-packages/aiohttp/cookiejar.py, line 113: with file_path.open(mode="wb") as f: inside the event loop; This is causing stability issues. Please create a bug report at https://github.com/home-assistant/core/issues?q=is%3Aopen+is%3Aissue For developers, please see https://developers.home-assistant.io/docs/asyncio_blocking_operations/#open

And:

Detected blocking call to open with args (PosixPath('/config/.storage/[email protected]'),) inside the event loop by custom integration 'alexa_media' at custom_components/alexa_media/__init__.py, line 251: cookies = await login.load_cookie() (offender: /usr/local/lib/python3.12/site-packages/aiohttp/cookiejar.py, line 118: with file_path.open(mode="rb") as f:), please create a bug report at https://github.com/alandtse/alexa_media_player/issues For developers, please see https://developers.home-assistant.io/docs/asyncio_blocking_operations/#open

Expected behavior

Asynchronous operations should not perform blocking file I/O within the event loop. The integration should avoid blocking calls to maintain stability in Home Assistant.

Screenshots

N/A

System details

• Home Assistant version: 2024.9.1
• alexa_media version (from const.py or HA startup log): e.g., 4.12.12
• alexapy version (from pip show alexapy in Home Assistant container or HA startup log): e.g., 1.29.2
• Is Amazon 2FA/2SV enabled (y/n): Yes
• Amazon Domain: amazon.de

Debug Logs (alexa_media & alexapy)

Detected blocking call to open with args (PosixPath('/config/.storage/[email protected]'),) in /usr/local/lib/python3.12/site-packages/aiohttp/cookiejar.py, line 113: with file_path.open(mode="wb") as f: inside the event loop; This is causing stability issues. Please create a bug report at https://github.com/home-assistant/core/issues?q=is%3Aopen+is%3Aissue For developers, please see https://developers.home-assistant.io/docs/asyncio_blocking_operations/#open

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/usr/src/homeassistant/homeassistant/__main__.py", line 223, in <module>
    sys.exit(main())
  File "/usr/src/homeassistant/homeassistant/__main__.py", line 209, in main
    exit_code = runner.run(runtime_conf)
  File "/usr/src/homeassistant/homeassistant/runner.py", line 189, in run
    return loop.run_until_complete(setup_and_run_hass(runtime_config))
  File "/usr/local/lib/python3.12/asyncio/base_events.py", line 674, in run_until_complete
    self.run_forever()
  File "/usr/local/lib/python3.12/asyncio/base_events.py", line 641, in run_forever
    self._run_once()
  File "/usr/local/lib/python3.12/asyncio/base_events.py", line 1990, in _run_once
    handle._run()
  File "/usr/local/lib/python3.12/asyncio/events.py", line 88, in _run
    self._context.run(self._callback, *self._args)
  File "/usr/local/lib/python3.12/site-packages/alexapy/helpers.py", line 137, in wrapper
    return await func(*args, **kwargs)
  File "/usr/local/lib/python3.12/site-packages/alexapy/alexaapi.py", line 1228, in get_devices
    response = await AlexaAPI._static_request(
  File "/usr/local/lib/python3.12/site-packages/backoff/_async.py", line 151, in retry
    ret = await target(*args, **kwargs)
  File "/usr/local/lib/python3.12/site-packages/alexapy/alexaapi.py", line 281, in _static_request
    await login.save_cookiefile()
  File "/usr/local/lib/python3.12/site-packages/alexapy/alexalogin.py", line 719, in save_cookiefile
    self._outputpath(f"{self._hass_domain}.{self.email}.pickle"),
  File "/usr/local/lib/python3.12/site-packages/aiohttp/cookiejar.py", line 113, in save
    with file_path.open(mode="wb") as f:

Additional similar error during cookie loading:

Detected blocking call to open with args (PosixPath('/config/.storage/[email protected]'),) inside the event loop by custom integration 'alexa_media' at custom_components/alexa_media/__init__.py, line 251: cookies = await login.load_cookie() (offender: /usr/local/lib/python3.12/site-packages/aiohttp/cookiejar.py, line 118: with file_path.open(mode="rb") as f:), please create a bug report at https://github.com/alandtse/alexa_media_player/issues For developers, please see https://developers.home-assistant.io/docs/asyncio_blocking_operations/#open

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/usr/src/homeassistant/homeassistant/__main__.py", line 223, in <module>
    sys.exit(main())
  File "/usr/src/homeassistant/homeassistant/__main__.py", line 209, in main
    exit_code = runner.run(runtime_conf)
  File "/usr/src/homeassistant/homeassistant/runner.py", line 189, in run
    return loop.run_until_complete(setup_and_run_hass(runtime_config))
  File "/usr/local/lib/python3.12/asyncio/base_events.py", line 674, in run_until_complete
    self.run_forever()
  File "/usr/local/lib/python3.12/asyncio/base_events.py", line 641, in run_forever
    self._run_once()
  File "/usr/local/lib/python3.12/asyncio/base_events.py", line 1990, in _run_once
    handle._run()
  File "/usr/local/lib/python3.12/asyncio/events.py", line 88, in _run
    self._context.run(self._callback, *self._args)
  File "/usr/src/homeassistant/homeassistant/config_entries.py", line 752, in async_setup_locked
    await self.async_setup(hass, integration=integration)
  File "/usr/src/homeassistant/homeassistant/config_entries.py", line 604, in async_setup
    result = await component.async_setup_entry(hass, self)
  File "/config/custom_components/alexa_media/__init__.py", line 474, in async_setup_entry
    cookies = await perform_login(login)
  File "/config/custom_components/alexa_media/__init__.py", line 251, in perform_login
    cookies = await login.load_cookie()

Additional context

Please find the adjusted cookiejar.py file attached or consider integrating similar changes to address the issue.

You can apply this patch by:

1. Install the Patch integration with HACS.

2. Download the cookiejar-patched.py.txt file, rename it to cookiejar.py, and place it in a config/patches/aiohttp/3.10.5/patch directory.

3. Copy the original /usr/local/lib/python3.12/site-packages/aiohttp/cookiejar.py to config/patches/aiohttp/3.10.5/base.

Then add this to your configuration.yaml:

patch:
  delay: 10
  restart: true
  files:
    - name: cookiejar.py
      base: patches/aiohttp/3.10.5/base
      destination: "/usr/local/lib/python3.12/site-packages/aiohttp"
      patch: patches/aiohttp/3.10.5/patch

This workaround modifies the cookiejar.py file to offload blocking file operations to a separate thread and ensures synchronization using threading.Event. This allows the methods to remain synchronous from the caller's perspective, avoiding changes to alexapy or alexa_media.

---

Proposed Solution

Implement the adjusted cookiejar.py to resolve the blocking call issue. The key changes include:

• Offloading file operations in the save and load methods to separate threads using run_in_executor.
• Using threading.Event to wait for the completion of loading and saving operations.
• Ensuring synchronization, so methods accessing self._cookies wait until the cookies are fully loaded.

This adjustment prevents the event loop from being blocked by synchronous file operations and maintains compatibility with existing code that relies on aiohttp.cookiejar.CookieJar.

---

Note: While this workaround resolves the immediate problem, a long-term solution would involve updating aiohttp to handle cookie file operations asynchronously or adjusting alexapy and alexa_media to avoid blocking calls within the event loop.

Additionally, if this workaround proves effective, it would be beneficial for someone to open an issue on the aiohttp GitHub repository to address this problem.

[jleinenbach]

@danielbrunt57 FYI

[danielbrunt57]

@jleinenbach See aio-libs/aiohttp#9430 and @bdraco's comments.
I managed to resolve the issue in alexalogin.py and will issue a PR there.

[danielbrunt57]

@jleinenbach BTW, your solution in cookiejar.py would still block the event loop even though HA fails to detect it:

This will block the event loop for the duration of the operation in the thread @bdraco

So if I understand this correctly then, the event loop is still blocked while the code is now being run in the executor so the net effect is no real change in behaviour except that HA is now unable to detect and warn of a blocking operation in the event loop? @danielbrunt57

If there's not await or async for/with, then it's not yielding back to the event loop, so it'd wait here till the function completes. @Dreamsorcerer

[danielbrunt57]

I managed to resolve the issue in alexalogin.py and will issue a PR there.

See alexapy merge request !394