RFC: consider linking the CLI tool against the shared library #2976
Comments
|
Seems like in git This happens, apparently, because these executables need I have an experimental fix that applies the private symbols more reliably after the shared library symbols get resolved: 1aaa10c Unfortunately:
GCC allows linking to both the shared and static libraries, and it resolves symbols from the shared library first, and then pulls in the implementations of still-missing symbols from the static library. MSVC errors out with: I don't know enough about MSVC to know if you can tell it "yes I know, but given a choice between an import library and a static library, choose the version in the import library and don't complain". I'm not sure whether there is a reliable solution here other than:
|
|
I believe this dependency was added by @yoniko , to implement asynchronous decompression feature in the CLI. I also think we already stated that we don't support linking the CLI to the dynamic library, due to usage of non-stable (experimental) symbols. We don't foresee a situation where we would limit the CLI to stable library symbols only, so this will remain the policy for the foreseeable future. |
|
Maybe it would help if I rephrased the problem a bit. Linking the CLI to the dynamic library (in a build configuration where the library is dynamic, not static) is useful because it reduces code duplication in the installed product. It doesn't imply that only public symbols may be used, but it does mean that symbols which are public will be gotten from the dynamic library instead of included from the static library. I have successfully done that for the most part in the meson.build, and I have a slightly optimized hack to improve its accuracy. However, I don't know how to do it for MSVC, or if it's even possible, so I have also implemented completely disabling the use of dynamic symbols from zstd.dll on MSVC. This could be made properly reliable, if zstd produced two libraries internally:
Inside the build system, the zstd library would:
Then tools could link to both zstd-shared and libcommon-static, and share symbols with the dynamic library where possible, but otherwise behave as they currently do. The straightforward way of implementing such a thing would be to just collect all public symbols into one set of files, and all private symbols into another set of files. Then just always link the tools to both libraries. That is how I'd write it for Meson. |
…f it Partial, Meson-only implementation of facebook#2976 for non-MSVC builds. Due to the prevalence of private symbol reuse, linking to a shared library is simply utterly unreliable, but we still want to defer to the shared library for installable applications. By linking to both, we can share symbols where possible, and statically link where needed. This means we no longer need to manually track every file that needs to be extracted and reused. The flip side is that MSVC completely does not support this, so for MSVC builds we just link to a full static copy even where -Ddefault_library=shared. As a side benefit, by using library inclusion rather than including extra explicit object files, the zstd program shrinks in size slightly (~4kb).
|
The PR I just linked above, fixes building on MSVC (tests still fail, cf. my other PR) by completely disabling the "linking the CLI tool against the shared library" trick that Meson historically did, but only for MSVC. I also anticipate it should entirely prevent future occurrences of "private symbols moved around and now the contributed Meson files fail to build because it does something we don't officially support". I remain convinced that the Meson files should do the "right thing" and link to the shared library, but it will no longer be at the cost of fragility in tracking private symbols. |
…f it Partial, Meson-only implementation of facebook#2976 for non-MSVC builds. Due to the prevalence of private symbol reuse, linking to a shared library is simply utterly unreliable, but we still want to defer to the shared library for installable applications. By linking to both, we can share symbols where possible, and statically link where needed. This means we no longer need to manually track every file that needs to be extracted and reused. The flip side is that MSVC completely does not support this, so for MSVC builds we just link to a full static copy even where -Ddefault_library=shared. As a side benefit, by using library inclusion rather than including extra explicit object files, the zstd program shrinks in size slightly (~4kb).
Split out from #2950
Currently, the official Makefiles statically link the CLI tool
zstdagainst the library objects. This has an immediately crucial purpose: the tool links against some private symbols.It would be interesting to consider optionally linking against the shared library instead. The private symbols can be refactored into an internal utility library called, for the sake of argument,
libcommon.a, and linked to both the shared library and to the program(s) that need it.This has a couple of advantages:
An existing implementation of this concept is already present in meson.build -- and this originally worked because meson didn't implement the
-fvisibilitysettings the Makefile did. I fixed that, and in the process had to unpack the objects from the shared library and link them individually to the CLI tool and to test utilities.Meson's built-in option
-Ddefault_library={static|shared|both}determines whether to build the shared and/or static libraries, and then the program will be linked to whichever the user chose.However, it becomes a bit complicated and undocumented to figure out which object files need to be linked in on top of libzstd itself. Making this officially supported in the Makefile build would be neat.
The text was updated successfully, but these errors were encountered: