Broken in HA Docker container 2023.6.3

[roflcoopter]

Prerequirements
[X] I have connected my mower(s) to the Automower Connect App
[X] I have connected my Husqvanra developers account to the Authentication API
[X] I have connected my Husqvarna developers account to the Automower Connect API

Describe the bug
For some reason the integration stopped working when i tried o update to HA 2023.6.3
Now when i try to downgrade HA it still refuses to load the integration, failing to install deps.
Here is the error log:

2023-06-27 10:33:05.486 ERROR (SyncWorker_5) [homeassistant.util.package] Unable to install package Shapely>=1.8.2: error: subprocess-exited-with-error
  
  � pip subprocess to install build dependencies did not run successfully.
  � exit code: 1
  ��> [278 lines of output]
      Collecting Cython~=0.29
        Using cached Cython-0.29.35-cp310-cp310-musllinux_1_1_x86_64.whl (2.0 MB)
      Collecting oldest-supported-numpy
        Using cached oldest_supported_numpy-2022.11.19-py3-none-any.whl (4.9 kB)
      Collecting setuptools>=61.0.0
        Using cached setuptools-68.0.0-py3-none-any.whl (804 kB)
      Collecting numpy==1.21.6
        Using cached numpy-1.21.6.zip (10.3 MB)
        Installing build dependencies: started
        Installing build dependencies: finished with status 'done'
        Getting requirements to build wheel: started
        Getting requirements to build wheel: finished with status 'done'
        Preparing metadata (pyproject.toml): started
        Preparing metadata (pyproject.toml): finished with status 'done'
      Building wheels for collected packages: numpy
        Building wheel for numpy (pyproject.toml): started
        Building wheel for numpy (pyproject.toml): finished with status 'error'
        error: subprocess-exited-with-error
      
        � Building wheel for numpy (pyproject.toml) did not run successfully.
        � exit code: 1
        ��> [248 lines of output]
            setup.py:63: RuntimeWarning: NumPy 1.21.6 may not yet support Python 3.10.
              warnings.warn(
            Running from numpy source directory.
            Processing numpy/random/_bounded_integers.pxd.in
            Processing numpy/random/_mt19937.pyx
            Processing numpy/random/_generator.pyx
            Processing numpy/random/mtrand.pyx
            Processing numpy/random/bit_generator.pyx
            Processing numpy/random/_common.pyx
            Processing numpy/random/_sfc64.pyx
            Processing numpy/random/_bounded_integers.pyx.in
            Processing numpy/random/_pcg64.pyx
            Processing numpy/random/_philox.pyx
            Cythonizing sources
            blas_opt_info:
            blas_mkl_info:
            customize UnixCCompiler
              libraries mkl_rt not found in ['/usr/local/lib', '/usr/lib']
              NOT AVAILABLE
      
            blis_info:
              libraries blis not found in ['/usr/local/lib', '/usr/lib']
              NOT AVAILABLE
      
            openblas_info:
              libraries openblas not found in ['/usr/local/lib', '/usr/lib']
              NOT AVAILABLE
      
            accelerate_info:
              NOT AVAILABLE
      
            atlas_3_10_blas_threads_info:
            Setting PTATLAS=ATLAS
              libraries tatlas not found in ['/usr/local/lib', '/usr/lib']
              NOT AVAILABLE
      
            atlas_3_10_blas_info:
              libraries satlas not found in ['/usr/local/lib', '/usr/lib']
              NOT AVAILABLE
      
            atlas_blas_threads_info:
            Setting PTATLAS=ATLAS
              libraries ptf77blas,ptcblas,atlas not found in ['/usr/local/lib', '/usr/lib']
              NOT AVAILABLE
      
            atlas_blas_info:
              libraries f77blas,cblas,atlas not found in ['/usr/local/lib', '/usr/lib']
              NOT AVAILABLE
      
            /tmp/pip-install-6s46au9d/numpy_baf82c7692df42b784692c8495005975/numpy/distutils/system_info.py:2026: UserWarning:
                Optimized (vendor) Blas libraries are not found.
                Falls back to netlib Blas library which has worse performance.
                A better performance should be easily gained by switching
                Blas library.
              if self._calc_info(blas):
            blas_info:
              libraries blas not found in ['/usr/local/lib', '/usr/lib']
              NOT AVAILABLE
      
            /tmp/pip-install-6s46au9d/numpy_baf82c7692df42b784692c8495005975/numpy/distutils/system_info.py:2026: UserWarning:
                Blas (http://www.netlib.org/blas/) libraries not found.
                Directories to search for the libraries can be specified in the
                numpy/distutils/site.cfg file (section [blas]) or by setting
                the BLAS environment variable.
              if self._calc_info(blas):
            blas_src_info:
              NOT AVAILABLE
      
            /tmp/pip-install-6s46au9d/numpy_baf82c7692df42b784692c8495005975/numpy/distutils/system_info.py:2026: UserWarning:
                Blas (http://www.netlib.org/blas/) sources not found.
                Directories to search for the sources can be specified in the
                numpy/distutils/site.cfg file (section [blas_src]) or by setting
                the BLAS_SRC environment variable.
              if self._calc_info(blas):
              NOT AVAILABLE
      
            non-existing path in 'numpy/distutils': 'site.cfg'
            lapack_opt_info:
            lapack_mkl_info:
              libraries mkl_rt not found in ['/usr/local/lib', '/usr/lib']
              NOT AVAILABLE
      
            openblas_lapack_info:
              libraries openblas not found in ['/usr/local/lib', '/usr/lib']
              NOT AVAILABLE
      
            openblas_clapack_info:
              libraries openblas,lapack not found in ['/usr/local/lib', '/usr/lib']
              NOT AVAILABLE
      
            flame_info:
              libraries flame not found in ['/usr/local/lib', '/usr/lib']
              NOT AVAILABLE
      
            atlas_3_10_threads_info:
            Setting PTATLAS=ATLAS
              libraries lapack_atlas not found in /usr/local/lib
              libraries tatlas,tatlas not found in /usr/local/lib
              libraries lapack_atlas not found in /usr/lib
              libraries tatlas,tatlas not found in /usr/lib
            <class 'numpy.distutils.system_info.atlas_3_10_threads_info'>
              NOT AVAILABLE
      
            atlas_3_10_info:
              libraries lapack_atlas not found in /usr/local/lib
              libraries satlas,satlas not found in /usr/local/lib
              libraries lapack_atlas not found in /usr/lib
              libraries satlas,satlas not found in /usr/lib
            <class 'numpy.distutils.system_info.atlas_3_10_info'>
              NOT AVAILABLE
      
            atlas_threads_info:
            Setting PTATLAS=ATLAS
              libraries lapack_atlas not found in /usr/local/lib
              libraries ptf77blas,ptcblas,atlas not found in /usr/local/lib
              libraries lapack_atlas not found in /usr/lib
              libraries ptf77blas,ptcblas,atlas not found in /usr/lib
            <class 'numpy.distutils.system_info.atlas_threads_info'>
              NOT AVAILABLE
      
            atlas_info:
              libraries lapack_atlas not found in /usr/local/lib
              libraries f77blas,cblas,atlas not found in /usr/local/lib
              libraries lapack_atlas not found in /usr/lib
              libraries f77blas,cblas,atlas not found in /usr/lib
            <class 'numpy.distutils.system_info.atlas_info'>
              NOT AVAILABLE
      
            lapack_info:
              libraries lapack not found in ['/usr/local/lib', '/usr/lib']
              NOT AVAILABLE
      
            /tmp/pip-install-6s46au9d/numpy_baf82c7692df42b784692c8495005975/numpy/distutils/system_info.py:1858: UserWarning:
                Lapack (http://www.netlib.org/lapack/) libraries not found.
                Directories to search for the libraries can be specified in the
                numpy/distutils/site.cfg file (section [lapack]) or by setting
                the LAPACK environment variable.
              return getattr(self, '_calc_info_{}'.format(name))()
            lapack_src_info:
              NOT AVAILABLE
      
            /tmp/pip-install-6s46au9d/numpy_baf82c7692df42b784692c8495005975/numpy/distutils/system_info.py:1858: UserWarning:
                Lapack (http://www.netlib.org/lapack/) sources not found.
                Directories to search for the sources can be specified in the
                numpy/distutils/site.cfg file (section [lapack_src]) or by setting
                the LAPACK_SRC environment variable.
              return getattr(self, '_calc_info_{}'.format(name))()
              NOT AVAILABLE
      
            numpy_linalg_lapack_lite:
              FOUND:
                language = c
                define_macros = [('HAVE_BLAS_ILP64', None), ('BLAS_SYMBOL_SUFFIX', '64_')]
      
            Warning: attempted relative import with no known parent package
            /tmp/pip-build-env-0p7iv24z/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py:275: UserWarning: Unknown distribution option: 'define_macros'
              warnings.warn(msg)
            running bdist_wheel
            running build
            running config_cc
            unifing config_cc, config, build_clib, build_ext, build commands --compiler options
            running config_fc
            unifing config_fc, config, build_clib, build_ext, build commands --fcompiler options
            running build_src
            build_src
            building py_modules sources
            creating build
            creating build/src.linux-x86_64-3.10
            creating build/src.linux-x86_64-3.10/numpy
            creating build/src.linux-x86_64-3.10/numpy/distutils
            building library "npymath" sources
            Could not locate executable gfortran
            Could not locate executable f95
            Could not locate executable ifort
            Could not locate executable ifc
            Could not locate executable lf95
            Could not locate executable pgfortran
            Could not locate executable nvfortran
            Could not locate executable f90
            Could not locate executable f77
            Could not locate executable fort
            Could not locate executable efort
            Could not locate executable efc
            Could not locate executable g77
            Could not locate executable g95
            Could not locate executable pathf95
            Could not locate executable nagfor
            Could not locate executable frt
            don't know how to compile Fortran code on platform 'posix'
      
      
            [Errno 2] No such file or directory: 'gcc'
      
      
            Traceback (most recent call last):
              File "/tmp/tmp_ub9xlbr_in_process.py", line 363, in <module>
                main()
              File "/tmp/tmp_ub9xlbr_in_process.py", line 345, in main
                json_out['return_val'] = hook(**hook_input['kwargs'])
              File "/tmp/tmp_ub9xlbr_in_process.py", line 261, in build_wheel
                return _build_backend().build_wheel(wheel_directory, config_settings,
              File "/tmp/pip-build-env-0p7iv24z/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 230, in build_wheel
                return self._build_with_temp_dir(['bdist_wheel'], '.whl',
              File "/tmp/pip-build-env-0p7iv24z/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 215, in _build_with_temp_dir
                self.run_setup()
              File "/tmp/pip-build-env-0p7iv24z/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 267, in run_setup
                super(_BuildMetaLegacyBackend,
              File "/tmp/pip-build-env-0p7iv24z/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 158, in run_setup
                exec(compile(code, __file__, 'exec'), locals())
              File "setup.py", line 448, in <module>
                setup_package()
              File "setup.py", line 440, in setup_package
                setup(**metadata)
              File "/tmp/pip-install-6s46au9d/numpy_baf82c7692df42b784692c8495005975/numpy/distutils/core.py", line 169, in setup
                return old_setup(**new_attr)
              File "/tmp/pip-build-env-0p7iv24z/overlay/lib/python3.10/site-packages/setuptools/__init__.py", line 153, in setup
                return distutils.core.setup(**attrs)
              File "/tmp/pip-build-env-0p7iv24z/overlay/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 148, in setup
                dist.run_commands()
              File "/tmp/pip-build-env-0p7iv24z/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 967, in run_commands
                self.run_command(cmd)
              File "/tmp/pip-build-env-0p7iv24z/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 986, in run_command
                cmd_obj.run()
              File "/tmp/pip-build-env-0p7iv24z/overlay/lib/python3.10/site-packages/wheel/bdist_wheel.py", line 299, in run
                self.run_command('build')
              File "/tmp/pip-build-env-0p7iv24z/overlay/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 313, in run_command
                self.distribution.run_command(command)
              File "/tmp/pip-build-env-0p7iv24z/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 986, in run_command
                cmd_obj.run()
              File "/tmp/pip-install-6s46au9d/numpy_baf82c7692df42b784692c8495005975/numpy/distutils/command/build.py", line 61, in run
                old_build.run(self)
              File "/tmp/pip-build-env-0p7iv24z/overlay/lib/python3.10/site-packages/setuptools/_distutils/command/build.py", line 135, in run
                self.run_command(cmd_name)
              File "/tmp/pip-build-env-0p7iv24z/overlay/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 313, in run_command
                self.distribution.run_command(command)
              File "/tmp/pip-build-env-0p7iv24z/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 986, in run_command
                cmd_obj.run()
              File "/tmp/pip-install-6s46au9d/numpy_baf82c7692df42b784692c8495005975/numpy/distutils/command/build_src.py", line 144, in run
                self.build_sources()
              File "/tmp/pip-install-6s46au9d/numpy_baf82c7692df42b784692c8495005975/numpy/distutils/command/build_src.py", line 155, in build_sources
                self.build_library_sources(*libname_info)
              File "/tmp/pip-install-6s46au9d/numpy_baf82c7692df42b784692c8495005975/numpy/distutils/command/build_src.py", line 288, in build_library_sources
                sources = self.generate_sources(sources, (lib_name, build_info))
              File "/tmp/pip-install-6s46au9d/numpy_baf82c7692df42b784692c8495005975/numpy/distutils/command/build_src.py", line 378, in generate_sources
                source = func(extension, build_dir)
              File "/tmp/pip-install-6s46au9d/numpy_baf82c7692df42b784692c8495005975/numpy/core/setup.py", line 666, in get_mathlib_info
                raise RuntimeError("Broken toolchain: cannot link a simple C program")
            RuntimeError: Broken toolchain: cannot link a simple C program
            [end of output]
      
        note: This error originates from a subprocess, and is likely not a problem with pip.
        ERROR: Failed building wheel for numpy
      Failed to build numpy
      ERROR: Could not build wheels for numpy, which is required to install pyproject.toml-based projects
      WARNING: You are using pip version 22.0.4; however, version 23.1.2 is available.
      You should consider upgrading via the '/usr/bin/python3 -m pip install --upgrade pip' command.
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

� pip subprocess to install build dependencies did not run successfully.
� exit code: 1
��> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.
WARNING: You are using pip version 22.0.4; however, version 23.1.2 is available.
You should consider upgrading via the '/usr/bin/python3 -m pip install --upgrade pip' command.
2023-06-27 10:33:05.487 ERROR (MainThread) [homeassistant.setup] Setup failed for custom integration husqvarna_automower: Requirements for husqvarna_automower not found: ['Shapely>=1.8.2'].

Is this something anyone else is experiencing?

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Additional context
Add any other context about the problem here.

[prairiesnpr]

Your build environment is broken in the docker container. You need to install any required build dependencies,. Looks like you are missing gcc to start and I'm sure numpy has others.

[roflcoopter]

Yeah definitely, but i am running the official Home Assistant docker container which makes it difficult to install extra packages

[prairiesnpr]

So we didn't test on docker, as it's unsupported. I've tested on home assistant OS and a core install. Home assistant OS will pull the appropriate python packages and not attempt to build from source like we are seeing here. I would think that the Docker image should pull the Shapely musllinux wheel from https://github.com/home-assistant/wheels-custom-integrations the same as Home assistant OS. But as it installs in Home assistant OS, I don't think it's an issue with the integration.

[prairiesnpr]

So, I was incorrect, container is a supported install method, was thinking you were running supervised. I'll see if I can spin up a container and see if I can replicate.

[roflcoopter]

Great, thanks for taking the time!

[Li0nc0]

Hi all,

Same issue using stable homeassistant docker "stable" release

Logger: homeassistant.setup
Source: setup.py:207
First occurred: 19:01:38 (1 occurrences)
Last logged: 19:01:38

Setup failed for custom integration husqvarna_automower: Requirements for husqvarna_automower not found: ['Shapely>=1.8.2'].

[prairiesnpr]

I just pulled the latest HA stable container (2023.6.3) and installed Husqvarna Automower 2023.6.1, the install authenticated and installed all required packages without issue. Any further ideas on what might be unique about your installs?

For reference I used the following compose template.

version: '3'
services:
  homeassistant:
    container_name: homeassistant
    image: "ghcr.io/home-assistant/home-assistant:stable"
    volumes:
      - /mnt/data/docker/ha/config:/config
      - /etc/localtime:/etc/localtime:ro
    restart: unless-stopped
    privileged: true
    network_mode: host

[pjbaert]

I believe this isn't related to the latest HA version since I got the same error on my 2023.5.3 setup.
The integration isn't working anymore & after checking my configuration in the developer tools > YAML tab, I got this feedback:

Integration error: husqvarna_automower - Requirements for husqvarna_automower not found: ['Shapely>=1.8.2'].

[triumfas]

I've tried to update Husqvarna integration in my HA 2023.6.3 docker and it fails to work. In logs I see:
Setup failed for custom integration husqvarna_automower: Requirements for husqvarna_automower not found: ['Shapely>=1.8.2'].

Restoring backup fixes the issue. Maybe it's for HA 2023.7, but with each integration upgrade I need to play back and forth - copy my coordinates and other info, update, remove integration, add integration and set things back.

Yup - removed the integration and added and did setup again - everything works now.

[prairiesnpr]

I've tried to update Husqvarna integration in my HA 2023.6.3 docker and it fails to work. In logs I see: Setup failed for custom integration husqvarna_automower: Requirements for husqvarna_automower not found: ['Shapely>=1.8.2'].

Restoring backup fixes the issue. Maybe it's for HA 2023.7, but with each integration upgrade I need to play back and forth - copy my coordinates and other info, update, remove integration, add integration and set things back.

Yup - removed the integration and added and did setup again - everything works now.

Can you clarify this for me, when you removed the integration and reinstalled, did you install the current version 2023.6.1 or an older version?

[triumfas]

I have installed 2023.7 version and it worked. But after HA restart it stopped working and I'm trying now to get back to working state without restoring backup. Have issues with configuration page not loading (2023.7 and 2023.6.1).

Update:
Downloaded 2023.6.0, configured successfully and checked that it's working after restart. Updated to 2023.6.1 - getting errors.

`Logger: homeassistant.setup
Source: setup.py:207
First occurred: 16:41:45 (1 occurrences)
Last logged: 16:41:45

Setup failed for custom integration husqvarna_automower: Requirements for husqvarna_automower not found: ['Shapely>=1.8.2'].`

`Logger: homeassistant.util.package
Source: util/package.py:107
First occurred: 16:40:29 (3 occurrences)
Last logged: 16:41:45

Unable to install package Shapely>=1.8.2: error: subprocess-exited-with-error × pip subprocess to install build dependencies did not run successfully. │ exit code: 1 ╰─> [297 lines of output] Looking in links: https://wheels.home-assistant.io/alpine-3.12/amd64/ Collecting Cython~=0.29 Using cached Cython-0.29.36-py2.py3-none-any.whl (988 kB` (just a part - long one)

[prairiesnpr]

Got it, please post logs and we can look into it. I wasn't able to replicate earlier.

[triumfas]

Added to my post above what I see in logs at the moment.

Downgraded integration to 2023.6.0 - everything works and HA restarts fast.
My guess is that Docker container needs to be updated with Shapely >= 1.8.2. Not sure what it is, but was it a dependency all the time just lower build or it was introduced with 2023.6.1?

[prairiesnpr]

Added to my post above what I see in logs at the moment.

Downgraded integration to 2023.6.0 - everything works and HA restarts fast. My guess is that Docker container needs to be updated with Shapely >= 1.8.2. Not sure what it is, but was it a dependency all the time just lower build or it was introduced with 202

Added to my post above what I see in logs at the moment.

Downgraded integration to 2023.6.0 - everything works and HA restarts fast. My guess is that Docker container needs to be updated with Shapely >= 1.8.2. Not sure what it is, but was it a dependency all the time just lower build or it was introduced with 2023.6.1?

Yes, we added Shapely as a dependency in 2023.6.1. I'm not able to replicate it on my Docker host, I've rebooted several times without issue. Would it be possible to spin up a fresh HA container and see if you have the same installation issue?

[triumfas]

I will try.

Is Shapely a part of docker image or it needs to be installed somehow?

[prairiesnpr]

I will try.

Is Shapely a part of docker image or it needs to be installed somehow?

It's compiled and hosted by Home Assistant at https://wheels.home-assistant.io/musllinux/ That's the issue, it should pull that wheel and install it, but it's falling back to trying to build it from source, which fails since the container and HA OS don't have a full build environment.

One note, I see in your logs, that it's looking for cython in https://wheels.home-assistant.io/alpine-3.12/amd64/ that will give a 404, since current version is 3.14, no idea if related.

[triumfas]

Created another container and everything looks good. So something is broken in my setup. Will need to check how to fix as I have loads of integrations set :) Thanks for help!

[prairiesnpr]

Created another container and everything looks good. So something is broken in my setup. Will need to check how to fix as I have loads of integrations set :) Thanks for help!

Any chance your container is out of date, maybe need to pull the latest image?

[triumfas]

It's the latest. And even tried to delete container, delete image, create container and have the same issue.

[andreas-fey]

Very same issue here.

[prairiesnpr]

OK, so I'm at a standstill on this one, since I'm unable to replicate.

If you are having this issue, I would suggest a few things.

1. Make sure your image is "ghcr.io/home-assistant/home-assistant:stable"
2. Ensure you have the latest image, looking at logs, I'm seeing references to python 3.10 where the current image has 3.11.4
3. Ensure you have the correct permissions on your config folder, some reports in core indicated similar issues where subfolders weren't writable.
4. Create a new container with an empty config directory and install the integration, we may be seeing a conflict with another integration.

Beyond that, perhaps there is a bug in core that we need to raise, based on the responses here, it seems pretty limited. Would be good to know if this is only affecting a specific architecture.

[triumfas]

I once again created a new separate container and compared container variables. And found the difference which was causing all of this.

Old and not working variable:
WHEELS_LINKS https://wheels.home-assistant.io/alpine-3.12/amd64/

Working one:
WHEELS_LINKS https://wheels.home-assistant.io/musllinux/

I just updated my original container with new link and tadaaa - integration got loaded.

@prairiesnpr - thank you for pointing to the right direction! Will need to check other variables as well :)

[roflcoopter]

Sorry for being so quiet on this issue.
I actually switched to the linuxserver/homeassistant container which at first gave me some other issues.
It turns out that i had to explicitly pull the amd64 tag linuxserver/homeassistant:amd64-2023.6.3
The multiarch container pulled the wrong image for some reason.

However, that container does not have https://wheels.home-assistant.io/musllinux/ as a pip index URL, so i created a custom init script which runs pip3 install --no-cache-dir --no-index --only-binary=:all: --find-links "https://wheels.home-assistant.io/musllinux/" Shapely>=1.8.2 and the problem is then solved!

[andreas-fey]

@roflcoopter the pip3 command did the trick for me - thanks! I just hope that I do not have to repeat this every time the container starts again...

[Li0nc0]

Hi all, the pip3 command fixed the issue for me too.
I just had to remove the Husqvarna integration and add it again.

Thank you @triumfas and @roflcoopter. :)