Build issue causing raspivid to fail (components not registered)

[ajlennon]

libmmal_vc_client.so makes use of attribute(constructor) to ensure that supplier components (e.g. camera) are loaded when the static library is loaded.

raspivid, and possibly other applications, link against libmmal_vc_client.so, causing the ctor to execute, but there is no needed dependency.

Some build environments (e.g. Yocto/OpenEmbedded) pass the '--no-as-needed' linker flag which removes the dependency on libmmal_vc_client and thus needed components are not registered.

In this situation raspivid then gives an error of the form

root@raspberrypi:~# raspivid -o test
mmal: mmal_component_create_core: could not find component 'vc.ril.camera'
mmal: Failed to create camera component
mmal: main: Failed to create camera component
mmal: Failed to run camera app. Please check for firmware updates

For further details see https://lists.yoctoproject.org/pipermail/yocto/2014-June/019933.html

[Ruffio]

@ajlennon is this still an issue?

[YamSoup]

yes

[ajlennon]

@Ruffio not sure I'm afraid. I submitted a patch to meta-raspberrypi to prevent the --as-needed being used. I I get a chance I'll see if that is working

[PeterDaveHello]

+1

[Ruffio]

@ajlennon is it working?

[JamesH65]

Anyone contributing to the thread have any further comment or possible fixes?

[PeterDaveHello]

Sorry I don't, I'm not using this for a while.

[strongheart1]

I get the same issue building from this repository on a raspberry with Ubuntu 16.04.4 LTS, gcc version 5.4.0 20160609 (Ubuntu/Linaro 5.4.0-6ubuntu1~16.04.9) - so, is this a configuration issue of the build environment?

[6by9]

so, is this a configuration issue of the build environment?

Yes, as covered in some of the linked issues.
If the linker adds the --no-as-needed flag, then it is over-eager in stripping out code from libmmal_vc_client.so to the extent of having no VideoCore components. Ideally there needs to be something extra to signal to the linker that this code is required, but it's not been a priority to resolve.

edit Correction - it's over-eager in determining that the application (raspistill/raspivid etc) doesn't use any functions from libmmal_vc_client.so, therefore drops the advertised dependency. libmmal_vc_client.so is complete with all the expected functionality.

[6by9]

@pelwell from #303 (comment) where you point a similar VPU issue. Having had a play I don't believe that is going to help here.
As I understand it, the issue here is that the linker when processing raspistill/vid has gone through and determined that we don't call any of the libmmal_vc_client.so functions, therefore it isn't needed as a dependency to the app. It misses that the constructor will do stuff that will make a difference.

Your VPU solution (I think) was the equivalent of the compiler for libmmal_vc_client.so determining that none of the functions were exported and therefore it could optimise them out of the .so. Your mod adds in an extra use of the variables to stop it optimising them away.

I've tried various things today, but adding:

SET( CMAKE_EXE_LINKER_FLAGS "-Wl,--no-as-needed" )

to the relevant application CMakeLists.txt seems to be the only solution that actually works.
I fail to see how we can do anything else when the linker with ``--as-needed``` is being so enthusiastic to discard things. I'll create a PR for that and people can object if they so wish.

[luked99]

Looking at the docs for the --as-needed ld flag, it says that this switches off the DT_NEEDED flag on the library concerned.
I wonder if there's a way to add it back in to the mmal shared libraries. It's just an ELF flag.

[luked99]

(i.e. rather than modifying every executable, just change the library).

[pelwell]

Or, as a compromise, add the flag to all library builds.

[6by9]

Thanks guys. I'd been trying to find a suitable flag to set on the library (because that is the nicer thing to do), but had failed. I'm still not convinced that there is a suitable flag on the library.

From man ld

       --as-needed
       --no-as-needed
           This option affects ELF DT_NEEDED tags for dynamic libraries mentioned on the command line after the --as-needed option.  Normally the linker will add a
           DT_NEEDED tag for each dynamic library mentioned on the command line, regardless of whether the library is actually needed or not.  --as-needed causes a
           DT_NEEDED tag to only be emitted for a library that at that point in the link satisfies a non-weak undefined symbol reference from a regular object file
           or, if the library is not found in the DT_NEEDED lists of other needed libraries, a non-weak undefined symbol reference from another needed dynamic
           library.  Object files or libraries appearing on the command line after the library in question do not affect whether the library is seen as needed.
           This is similar to the rules for extraction of object files from archives.  --no-as-needed restores the default behaviour.

As I read that, it is solely at the link stage of the application that it is making up the list of libraries to include. The DT_NEEDED tag itself doesn't exist outside of the linker until the link is complete and the ELF file written.
Without some explicit call into the mmal_vc_client, ld will drop the dependency. We don't want an explicit call as the constructor is the bit that registers the components for general use.

The DT_NEEDED flags on libmmal_vc_client list just the things it depends on, which is as expected:

ubuntu@ubuntu:~/userland$ readelf -d ./build/lib/libmmal_vc_client.so 

Dynamic section at offset 0x6ee8 contains 31 entries:
  Tag        Type                         Name/Value
 0x00000001 (NEEDED)                     Shared library: [libvchiq_arm.so]
 0x00000001 (NEEDED)                     Shared library: [libvcsm.so]
 0x00000001 (NEEDED)                     Shared library: [libvcos.so]
 0x00000001 (NEEDED)                     Shared library: [libpthread.so.0]
 0x00000001 (NEEDED)                     Shared library: [libc.so.6]
 0x00000001 (NEEDED)                     Shared library: [ld-linux-armhf.so.3]
 0x0000000e (SONAME)                     Library soname: [libmmal_vc_client.so]
 0x0000000f (RPATH)                      Library rpath: [/home/ubuntu/userland/build/lib:]

On raspistill, it has totally thrown away libmmal_vc_client.so

ubuntu@ubuntu:~/userland$ readelf -d ./build/bin/raspistill

Dynamic section at offset 0x10ec0 contains 35 entries:
  Tag        Type                         Name/Value
 0x00000001 (NEEDED)                     Shared library: [libmmal_core.so]
 0x00000001 (NEEDED)                     Shared library: [libmmal_util.so]
 0x00000001 (NEEDED)                     Shared library: [libbrcmGLESv2.so]
 0x00000001 (NEEDED)                     Shared library: [libbrcmEGL.so]
 0x00000001 (NEEDED)                     Shared library: [libm.so.6]
 0x00000001 (NEEDED)                     Shared library: [libvcsm.so]
 0x00000001 (NEEDED)                     Shared library: [libbcm_host.so]
 0x00000001 (NEEDED)                     Shared library: [libvcos.so]
 0x00000001 (NEEDED)                     Shared library: [libpthread.so.0]
 0x00000001 (NEEDED)                     Shared library: [libc.so.6]
 0x00000001 (NEEDED)                     Shared library: [ld-linux-armhf.so.3]
 0x0000000f (RPATH)                      Library rpath: [/home/ubuntu/userland/build/lib:]

I guess the other approach is to throw the towel in and make all the components dependencies on mmal_core. mmal_core then needs to be built with --no-as-needed so that ld doesn't throw the dependencies away from that. It bloats sizes though,as we'd have to include all the ARM side components in the list of dependencies too, otherwise the problem still remains for them.
In reality we don't build MMAL for other platforms any more, so don't care about it only being valid for VideoCore.

Thoughts?

[JamesH65]

It would be nice to close this over 4 year old issue before it reaches its next birthday - is there anything that can/should be done?

[JamesH65]

This issue will be closed within 30 days unless further interactions are posted. If you wish this issue to remain open, please add a comment. A closed issue may be reopened if requested.

[6by9]

Closing as should be fixed by #714 due to Bullseye changing the default compiler options.