cyclonedds nightly idlc missing library

[PetervdPerk-NXP]

From cyclonedds-nightly 2023.8.22 and onwards.
idlc fails with following error.

idlc -v
Cannot load generator libcycloneddsidlc.so: libcycloneddsidlc.so: cannot open shared object file: No such file or directory
idlc: cannot load generator c
idlc (Eclipse Cyclone DDS) 0.11.0

[eboasson]

Bummer. That must be because the C backend is now in a separate library and it apparently loads it by default even if you just do idlc -v.

For me (on macOS, master + some fiddling):

DYLD_PRINT_LIBRARIES=1 bin/Debug/idlc -v 2>&1|rg 'idlc|ddsc|py'
dyld[42964]: <E7A243D1-138C-3026-BDE0-C13FBE174688> /Users/erik/cdds/build/bin/Debug/idlc
dyld[42964]: <7BC18725-ACAD-3381-94C9-547C5C9C4A5F> /Users/erik/cdds/build/lib/Debug/libddsc.0.11.0.dylib
dyld[42964]: <E2259F8E-0305-3955-918F-AD6DCE3614ED> /Users/erik/cdds/build/lib/Debug/libcycloneddsidlc.0.11.0.dylib

so that matches the theory that the problem is that it loads the library by default. If instead I add -lpy:

# DYLD_PRINT_LIBRARIES=1 bin/Debug/idlc -lpy -v 2>&1|rg 'idlc|ddsc|py'
dyld[43006]: <E7A243D1-138C-3026-BDE0-C13FBE174688> /Users/erik/cdds/build/bin/Debug/idlc
dyld[43006]: <7BC18725-ACAD-3381-94C9-547C5C9C4A5F> /Users/erik/cdds/build/lib/Debug/libddsc.0.11.0.dylib
dyld[43006]: <4D867984-354A-3147-95D1-4C5C0886D1AE> /usr/lib/system/libcopyfile.dylib
dyld[43006]: <DB35F8A1-AA07-32FF-BB71-EE45C2ACB797> /opt/homebrew/lib/python3.11/site-packages/cyclonedds/_idlpy.cpython-311-darwin.so
idlc (Eclipse Cyclone DDS) 0.11.0

it only loads the Python backend.

I think the most sensible solution is that idlc is changed to load the backend after doing the command-line processing and processing the -v option. Until that's done, idlc -lpy -v would be a workaround. Would you agree?

This way there is no need to update the python packaging because it no longer has a (meaningful) dependency on the C backend ... otherwise it seems it would involve changing idlc_library to idlc_libraries and copying more stuff in the various scripts in https://github.com/eclipse-cyclonedds/cyclonedds-python/blob/1fbd53bae14cd35061f766ae0fa6b3fb17512057/buildhelp, but I wouldn't even know how to test that.

[PetervdPerk-NXP]

I guess idlc -lpy -v would work, I kinda hoped that nightly included the cdrstream-desc option so I can (ab)use that to generate my c definitions. Since no recent release of cyclonedds include the cdrstream-desc option.

[eboasson]

Ah, ok, I see ...

It is not like it intentionally excludes it, it is that I never thought about python wheels when considering the consequences of moving the C backend into its own library.

If you happen to know how to add libcycloneddsidlc.so (or its variants for macOS/Windows) to the Python wheel and test that it works before letting the CI pipeline start pushing out updates to pypi (it's pypi, is it not?) then we can simply do that. Otherwise I can also try to find someone who can help with this.