SIGSEGV, Segmentation fault.

[magicse]

Describe the bug

Arm docker Alpine 3.16
Python 3.10.7

bash-5.1# gdb python3 -c "import aiohttp"
(gdb) r
Starting program: /usr/local/bin/python3 
Python 3.10.7 (main, Nov 24 2022, 13:02:43) [GCC 11.2.1 20220219] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import aiohttp
Program received signal SIGILL, Illegal instruction.
0x754c49fe in ?? () from /lib/libcrypto.so.1.1
(gdb) handle SIGILL nostop 
Signal        Stop      Print   Pass to program Description
SIGILL        No        Yes     Yes             Illegal instruction
(gdb) r
The program being debugged has been started already.
Start it from the beginning? (y or n) y
Starting program: /usr/local/bin/python3 
Python 3.10.7 (main, Nov 24 2022, 13:02:43) [GCC 11.2.1 20220219] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import aiohttp
Program received signal SIGILL, Illegal instruction.
Program received signal SIGILL, Illegal instruction.
Program received signal SIGILL, Illegal instruction.
Program received signal SIGILL, Illegal instruction.

Program received signal SIGSEGV, Segmentation fault.
0x75fb86c2 in gnu_lookup_filtered () from /lib/ld-musl-armhf.so.1
(gdb) 

(gdb) bt
#0  0x75fb86c2 in gnu_lookup_filtered () from /lib/ld-musl-armhf.so.1
#1  0x75fb87a8 in find_sym () from /lib/ld-musl-armhf.so.1
#2  0x75fa6f7c in load_library () from /lib/ld-musl-armhf.so.1
#3  0x75ffb4e8 in rtld_fail () from /lib/ld-musl-armhf.so.1
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
(gdb) 

To Reproduce

python3 -c "import aiohttp"

Expected behavior

simple import aiohttp in python

Logs/tracebacks

Arm docker Alpine 3.16
Python 3.10.7

The program being debugged has been started already.
Start it from the beginning? (y or n) y
Starting program: /usr/local/bin/python3 
Python 3.10.7 (main, Nov 24 2022, 13:02:43) [GCC 11.2.1 20220219] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import aiohttp

Program received signal SIGILL, Illegal instruction.

Program received signal SIGILL, Illegal instruction.

Program received signal SIGILL, Illegal instruction.

Program received signal SIGILL, Illegal instruction.

Program received signal SIGSEGV, Segmentation fault.
0x75fb86c2 in gnu_lookup_filtered () from /lib/ld-musl-armhf.so.1
(gdb)

Python Version

bash-5.1# python --version
Python 3.10.7
bash-5.1#

aiohttp Version

$bash-5.1#  python -m pip show aiohttp
Name: aiohttp
Version: 3.8.1
Summary: Async http client/server framework (asyncio)
Home-page: https://github.com/aio-libs/aiohttp
Author: 
Author-email: 
License: Apache 2
Location: /usr/local/lib/python3.10/site-packages
Requires: aiosignal, async-timeout, attrs, charset-normalizer, frozenlist, multidict, yarl
Required-by: accuweather, adax, Adax-local, adguardhome, advantage-air, afsapi, aio-geojson-client, aio-geojson-generic-client, aio-geojson-geonetnz-quakes, aio-geojson-geonetnz-volcano, aio-geojson-nsw-rfs-incidents, aio-geojson-usgs-earthquakes, aio-georss-client, aioairq, aioairzone, AIOAladdinConnect, aioambient, aioaseko, aioazuredevops, aiobotocore, aioeafm, aioeagle, aioecowitt, aioemonitor, aioflo, aiogithubapi, aioguardian, aioharmony, aiohttp-cors, aiohue, aiolivisi, aiolookin, aiolyric, aiomodernforms, aiomusiccast, aionanoleaf, aionotion, aiooncue, aioopenexchangerates, aiopurpleair, aiopvapi, aiopvpc, aiopyarr, aioqsw, aiorecollect, aioridwell, aioshelly, aioskybell, AIOSomecomfort, aiosteamist, aiosyncthing, aiotractive, aiounifi, aiowatttime, aioymaps, airly, airthings-cloud, alpha-vantage, Ambiclimate, asmog, async-upnp-client, asyncpysupla, asyncsleepiq, blebox-uniapi, bluetooth-adapters, bond-async, brunt, connect-box, coronavirus, crownstone-cloud, crownstone-sse, demetriek, directv, elgato, eliqonline, emulated-roku, energyzero, enturclient, env-canada, epson-projector, esphome-dashboard-api, eternalegypt, evohome-async, faadelays, fivem-api, foobot-async, forecast-solar, freebox-api, gcal-sync, geniushub-client, geocachingapi, gios, goalzero, google-nest-sdm, goslide-api, greeneye-monitor, gridnet, hass-nabucasa, here-routing, here-transit, hole, homeassistant, homematicip, homepluscontrol, ibeacon-ble, intellifire4py, jsonrpc-async, jsonrpc-websocket, kaiterra-async-client, konnected, lacrosse-view, laundrify-aio, life360, logi-circle, london-tube-status, mac-vendor-lookup, mbddns, meater-python, mill-local, millheater, moehlenhoff-alpha2, motioneye-client, mutesync, nettigo-air-monitor, nexia, nextcord, nextdns, odp-amsterdam, omnilogic, open-garage, open-meteo, openai, opensensemap-api, ovoenergy, p1monitor, peco, plexauth, plexwebsocket, plugwise, poolsense, ProgettiHWSW, pubnub, pvo, py-melissa-climate, py-nightscout, py-synologydsm-api, py17track, pyaftership, pyairnow, pyairvisual, pyatag, pyatmo, pyatv, pyaussiebb, pybravia, pycfdns, pydaikin, pydeconz, pydelijn, pydroid-ipcam, pyebox, pyeconet, pyefergy, pyEight, pyeverlights, pyevilgenius, pyfido, pyforked-daapd, PyFronius, pygti, pyhaversion, pyhiveapi, pyinsteon, pyintesishome, pyipma, pyipp, pyiqvia, pyisy, pykaleidescape, pykmtronic, pykodi, pykoplenti, pylaunches, pylibrespot-java, pylitterbot, pymazda, pymelcloud, PyMetEireann, PyMetno, pymyq, PyNINA, pynws, pyoctoprintapi, pyopenuv, pyoverkiz, pyprosegur, pyprusalink, pyps4-2ndscreen, pyrisco, pyrituals, pyrympro, pysabnzbd, pysensibo, pyskyqhub, pysma, pysmartthings, pyspcwebgw, pysqueezebox, pyswitchbee, PySyncThru, pytautulli, python-awair, python-bsblan, python-family-hub-local, python-homewizard-energy, python-izone, python-juicenet, python-matter-server, python-opendata-transport, python-smarttub, python-songpal, pyTibber, pytile, pytomorrowio, pytraccar, pytrafikverket, pyunifiprotect, pyuptimerobot, pyversasense, pyvizio, pyvolumio, pyxeoma, radios, regenmaschine, renault-api, reolink-aio, rokuecp, rtsp-to-webrtc, sense-energy, sharkiq, simplepush, simplisafe-python, sisyphus-control, slackclient, smart-meter-texas, solax, starline, subarulink, surepy, systembridgeconnector, tailscale, tesla-wall-connector, toonapi, tp-connected, ttls, twentemilieu, twitchAPI, uasiren, unifi-discovery, vehicle, volkszaehler, volvooncall, vulcan-api, waqiasync, whirlpool-sixth-sense, wled, xbox-webapi, yalexs, yolink-api, zamg, zigpy, zwave-js-server-python
bash-5.1#

multidict Version

bash-5.1# $ python -m pip show multidict
bash: $: command not found
bash-5.1# python -m pip show multidict
Name: multidict
Version: 6.0.4
Summary: multidict implementation
Home-page: https://github.com/aio-libs/multidict
Author: Andrew Svetlov
Author-email: [email protected]
License: Apache 2
Location: /usr/local/lib/python3.10/site-packages
Requires: 
Required-by: aiohttp, yarl
bash-5.1#

yarl Version

bash-5.1# python -m pip show yarl
Name: yarl
Version: 1.8.1
Summary: Yet another URL library
Home-page: https://github.com/aio-libs/yarl/
Author: Andrew Svetlov
Author-email: [email protected]
License: Apache 2
Location: /usr/local/lib/python3.10/site-packages
Requires: idna, multidict
Required-by: adguardhome, aiohttp, aiomodernforms, aiosyncthing, aiotractive, demetriek, directv, elgato, elmax-api, energyzero, geocachingapi, gql, gridnet, here-routing, here-transit, homeassistant, homepluscontrol, netdata, odp-amsterdam, p1monitor, pvo, pyipp, python-bsblan, radios, rokuecp, tailscale, toonapi, twentemilieu, vehicle, vulcan-api, wled, zigpy
bash-5.1#

OS

Alpine 3.16 Docker

Related component

Client

Additional context

homeassistant docker image

Code of Conduct

• I agree to follow the aio-libs Code of Conduct

[webknjaz]

What's the method of installation? Were aiohttp and it's deps installed from the musllinux wheels on PyPI or compiled from sdist?

[webknjaz]

Also, is this reproducible with aiohttp v3.8.4?

[magicse]

I try different methods
Last metod

wget https://files.pythonhosted.org/packages/5a/86/5f63de7a202550269a617a5d57859a2961f3396ecd1739a70b92224766bc/aiohttp-3.8.1.tar.gz
tar -xf aiohttp-3.8.1.tar.gz
cd aiohttp-3.8.1.tar.gz
find ./* -mtime +10950 -exec touch {} +
python setup.py build -j 4 install

[webknjaz]

Using setup.py as a CLI script is not supported. Use pip or pypa/build.

[magicse]

bash-5.1# cd aiohttp-3.8.1
bash-5.1# pip install -e .
Obtaining file:///home/aiohttp-3.8.1
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build editable ... done
  Installing backend dependencies ... done
  Preparing editable metadata (pyproject.toml) ... done
Requirement already satisfied: yarl<2.0,>=1.0 in /usr/local/lib/python3.10/site-packages (from aiohttp==3.8.1) (1.8.1)
Requirement already satisfied: charset-normalizer<3.0,>=2.0 in /usr/local/lib/python3.10/site-packages (from aiohttp==3.8.1) (2.1.1)
Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.10/site-packages (from aiohttp==3.8.1) (1.3.1)
Requirement already satisfied: async-timeout<5.0,>=4.0.0a3 in /usr/local/lib/python3.10/site-packages (from aiohttp==3.8.1) (4.0.2)
Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.10/site-packages (from aiohttp==3.8.1) (22.2.0)
Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.10/site-packages (from aiohttp==3.8.1) (6.0.4)
Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.10/site-packages (from aiohttp==3.8.1) (1.3.3)
Requirement already satisfied: idna>=2.0 in /usr/local/lib/python3.10/site-packages (from yarl<2.0,>=1.0->aiohttp==3.8.1) (3.4)
Building wheels for collected packages: aiohttp
  Building editable for aiohttp (pyproject.toml) ... done
  Created wheel for aiohttp: filename=aiohttp-3.8.1-0.editable-cp310-cp310-linux_armv7l.whl size=5395 sha256=1445b7e2b9fad1209a82d353a0bd51ebc625931585d2cb4f90462b3907794da7
  Stored in directory: /tmp/pip-ephem-wheel-cache-svoy2sy0/wheels/fe/10/79/4b0cb50d2a47f838863e8114f00e29bc379736ab400ca74f1b
Successfully built aiohttp
Installing collected packages: aiohttp
  Attempting uninstall: aiohttp
    Found existing installation: aiohttp 3.8.1
    Uninstalling aiohttp-3.8.1:
      Successfully uninstalled aiohttp-3.8.1
Successfully installed aiohttp-3.8.1
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv

bash-5.1# python
Python 3.10.7 (main, Nov 24 2022, 13:02:43) [GCC 11.2.1 20220219] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import aiohttp
Segmentation fault
bash-5.1# 

[webknjaz]

You don't really need it to be editable or unpack it manually. In fact, there's no testing for building the unpacked sdists. Try installing from PyPI. Also, choose the latest version.

linux_armv7l

I missed that it's also an old ARM arch. I'm guessing it's some like Raspberry? This explains why there's no wheel for your env — the newer envs are better standardized and we ship musllinux wheels for aarch64.

Also, this means that aiohttp was never tested on that platform and having no access to the hardware, it's rather hard to guess what's needed for your case to work.

One thing to change is to try running a reproducer on a clean Python alpine image. We're definitely not going to debug third party software that might've influenced your custom image.

Oh, and as an immediate workaround, you can set AIOHTTP_NO_EXTENSIONS=1 env var to force it to use pure-Python env fallbacks. The segfaulting bit is C-extensions. If you do not use them, they won't fail. It'll be slower, though.

[magicse]

I missed that it's also an old ARM arch. I'm guessing it's some like Raspberry?

This is QNAP TS-231P3 with Home assistant docker image.

@a8e86c47f7cb:/home/aiohttp-3.8.1$ uname -a
Linux a8e86c47f7cb 4.2.8 #2 SMP Thu Jan 12 10:44:50 CST 2023 armv7l Linux
@a8e86c47f7cb:/home/aiohttp-3.8.1$ uname -m
armv7l

bash-5.1# cat /proc/cpuinfo
processor : 0
model name : Annapurna Labs Alpine AL314 Quad-core ARM Cortex-A15 CPU @ 1.70GHz
Speed : 1.7GHz
Features : half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae evtstrm
CPU implementer : 0x41
CPU architecture: 7
CPU variant : 0x2
CPU part : 0xc0f
CPU revision : 4

And after some update home assistant didn't work. That is why I dig for this problem.

[webknjaz]

FWIW try out all the things I pointed out earlier. Also, this issue needs a reproducer that doesn't depend on how someone else (as in HA) built their images. It'd expect a proper repro to use some raw official Python Alpine image demonstrating that this problem exists in isolated. Until then, for what we know, it could be HA breaking musl or smth in the image.

[magicse]

On the original clear Alpine 3.16 docker image, aiohttp is compiled and works just fine. Home Assistant too )) Hah.
Yes you are right they broke smth in wheels or image.

[webknjaz]

I can't think of a way wheels could break musl but I trust that the HA devs can figure it out. It sounds like this issue doesn't need to remain open, now that it's clear that the bug is not coming from us. So I'm going to close it.