vcpkg support

[windelbouwman]

Hi!

I'm trying to setup a CI on github actions to build a rust gtk app. This is not an easy task. I'm using vcpkg with visual studio 2019 as a compiler.

Would it be possible / is it an idea at all to add vcpkg support to this system-deps repository?

Regards,
Windel

[gdesmott]

Hi Windel.

I'm not familiar with Windows development or vcpkg so can't really tell. Currently the recommended way is to use pkg-config on Windows as well, that's what we use for the gtk-rs CI I think.

@sdroege : what do you think about this?

[sdroege]

I'm not very positive about vcpkg as they ship broken GTK since a couple of years without anybody ever fixing it. I expect people to run into problems. If GTK is already that broken for so long, other things will surely be too. Otherwise it would seem like a good solution.

@nirbheek do you have opinions?

[nirbheek]

vcpkg is listed on the gtk.org website, even though it's known to be broken. I'd say vcpkg support can be added if someone is willing to come in and take ownership of that and deal with broken things :)

[windelbouwman]

I looked a bit into vcpkg. As a build system on windows, it works great, but how to extract the various build info remains a bit a mistery to me.

It would be great to fix the issue in gtk with the gtk-3.lib renaming to gtk-3.0.lib etc.. Hopefully vcpkg can somehow report what the proper filename of the gtk lib files is.

The benefit of vcpkg is that you can use the default rust on windows with gtk-rs. This works really well, except for the renaming of dll and lib files. Next item is the resources and settings.ini file, but that's a different story.

[gdesmott]

@ids1024 started working on this on #12

Before considering merging this work we'd need to sort out the following:

• Do we want to support this? I never build anything on Windows so I really don't now what's the added value of vcpkg comparing to pkg-config.
• Should user explicitly ask to use vcpkg instead of pkg-config or should it be transparent? Maybe it could be used as a fallback and we could have an env variable to force it?
• According to Prototype for supporting vcpkg #12 vcpkg does not allow to check for a specific version and looks for a different name than pkg-config (glib vs glib-2.0). What would be the best way to handle that?

[windelbouwman]

I would still encourage to support this. It is true that vcpkg has its own quirks, but I used it to build a gtk-rs app in windows. Windows support is a thing, and I believe it cannot be ignored. The obligatory response: "if you want to use this application, install linux instead of windows" does not make much sense to me.

To me, it appears that this package, system-deps should be a sort of adapter / frontend for many different packaging systems. Packaging systems I know of are pkg-config and vcpkg. There might be more package systems which can install developer headers and libraries. I did not check thoroughly, but for now system-deps does not much more than wrapping pkg-config, and if it fails to find it, it will fallback to some common locations. It would be great to add vcpkg support and for macOS there might be another build system?

[gdesmott]

I would still encourage to support this. It is true that vcpkg has its own quirks, but I used it to build a gtk-rs app in windows. Windows support is a thing, and I believe it cannot be ignored. The obligatory response: "if you want to use this application, install linux instead of windows" does not make much sense to me.

The point here is not to discuss if we want to support Windows or not, of course we do, but what's the added value vcpkg is bringing comparing to pkg-config which is also working on Windows.

[windelbouwman]

* Should user explicitly ask to use `vcpkg` instead of `pkg-config` or should it be transparent? Maybe it could be used as a fallback and we could have an env variable to force it?

This is an interesting design topic. I'm a bit reluctant to automatic detection of either pkg-config or vcpkg, since a change of your PATH setting might surprise users. How about sensible defaults per OS, with the option to override it? So:

• On linux and mac, it will use pkg-config, unless explicitly overridden by an environment variable to use vcpkg. If pkg-config is not found, it will fail (not fall back to heuristics)
• On windows, it will use vcpkg, unless explicitly overridden by an environment variable to use pkg-config

[windelbouwman]

I would still encourage to support this. It is true that vcpkg has its own quirks, but I used it to build a gtk-rs app in windows. Windows support is a thing, and I believe it cannot be ignored. The obligatory response: "if you want to use this application, install linux instead of windows" does not make much sense to me.

The point here is not to discuss if we want to support Windows or not, of course we do, but what's the added value vcpkg is bringing comparing to pkg-config which is also working on Windows.

Okay, I'm sorry for this. Ignore my comment about this 😄 , we agree here.

I did not try pkg-config on windows, I'll give it a shot some day! Does it work well with visual studio? Is this still a discussion point, which compiler on windows we want to support? rust recommends the visual studio backend, while gtk-rs recommends mingw backend. I guess this is a mismatch, and for me a reason to prefer vcpkg.

[sdroege]

It is true that vcpkg has its own quirks, but I used it to build a gtk-rs app in windows.

Also vcpkg is not recommended by the gtk project anymore because they packaging is simply too broken. Everybody I know is doing gtk development on Windows otherwise. Both with mingw and msvc (and personally I'd recommend the latter on Windows).

Generally the quality of software in vcpkg is really bad, I wouldn't make it the default and making it the default would also break existing workflows.

[ids1024]

My experience with Windows development is basically limited to porting software developed primarily on Linux. So it would be nice to hear from someone who knows more but as I understand it:

There are actually two kinds of Rust targets for Windows. <arch>-pc-windows-gnu, and <arch>-pc-windows-msvc. These are ABI incompatible. With the -gnu toolchain, we can use mingw versions of pkg-config and libraries installed with msys2. Which works perfectly well.

But unless I'm mistaken, there isn't an easy way to use pkg-config with the -msvc toochain.

The point here is not to discuss if we want to support Windows or not, of course we do, but what's the added value vcpkg is bringing comparing to pkg-config which is also working on Windows.

If what I said above is accurate, the advantage would be good support for the -msvc toochain. Which it seems like vcpkg is the best way to support.

On windows, it will use vcpkg, unless explicitly overridden by an environment variable to use pkg-config

Probably default to vcpkg on -msvc, but not -gnu Windows toolchains.

[sdroege]

Making vcpkg the default for the msvc toolchain would also break existing workflows.

But unless I'm mistaken, there isn't an easy way to use pkg-config with the -msvc toochain.

It works fine with the msvc toolchain, just not with vcpkg because their packaging is broken.

[ids1024]

Hm. The Gtk docs currently mention gvsbuild. I guess that is currently the best way to use gtk-rs with the msvc toolchain?

The vcpkg crate is a reverse dependency of some fairly significant -sys crates, so they might provide some insight in to how this could/should be handled: https://crates.io/crates/vcpkg/reverse_dependencies

It seems to me it should probably be supported, in some way, even if not by default. I suppose system-deps could allow crates using it to configure whether or not they want to use vcpkg, dependingly on the quality of the packaging there for the library they link.

[sdroege]

Hm. The Gtk docs currently mention gvsbuild. I guess that is currently the best way to use gtk-rs with the msvc toolchain?

That, or cerbero (there's a branch with gtk support), or using meson directly.

The vcpkg crate is a reverse dependency of some fairly significant -sys crates, so they might provide some insight in to how this could/should be handled: https://crates.io/crates/vcpkg/reverse_dependencies

The problem is not so much integration of vcpkg here, but that the vcpkg packaging of gtk is completely broken and it doesn't seem very actively maintained either :)

[...] dependingly on the quality of the packaging there for the library they link.

That would make sense indeed, yes.

[windelbouwman]

The Gtk docs currently mention gvsbuild.

Hmm, this looks interesting! If this is recommended by the gtk team to build with visual studio, maybe we should use gvsbuild instead of vcpkg.

Could gvsbuild be added as a backend to system-deps, or does it also provide pkg-config?

[windelbouwman]

Still I believe that a good approach on windows would be the rust toolchain with the visual studio compiler with a GTK build with visual studio as well. Is there already a wiki page about building gtk on windows specifically for gtk-rs? I thought there was some sort of guide for it?

I keep my own instructions here:

https://github.com/windelbouwman/lognplot/blob/master/lognplotgtk/building.md

[sdroege]

Could gvsbuild be added as a backend to system-deps, or does it also provide pkg-config?

It does. That's the officially supported way for finding GTK and how to link to it.

[windelbouwman]

Hmm, I tried to run gvsbuild, but it failed somewhere with gettext.

This looks like the manual route, which I do not wish to walk: https://wiki.gnome.org/Projects/GTK/Win32/MSVCCompilationOfGTKStack

[ras0219-msft]

Reviving this since it was brought up in gtk-rs/gtk4-rs#865.

Hi everyone! I'm a maintainer on vcpkg. We'd really like to make it easier for rust to consume C & C++ libraries via vcpkg (as well as the reverse in the future).

I'd also like to clarify slightly that vcpkg serves a different role than pkg-config: it should be compared to "apt" instead. For example, we support consuming vcpkg packages via pkg-config, because packages can provide .pc files. Especially in regards to meson and pkg-config, our support for both systems has improved dramatically in the last two years (though we're aware that further improvements remain). It is our intent to supply standard, valid pkg-config files on all platforms for all libraries that have upstream support.

I'm sorry that some users have had bad experiences with gtk via vcpkg; the gnome collection of libraries is much more challenging to package properly than a bog-standard CMake "here's a handful of c files". As with many similar ecosystems, vcpkg relies heavily on the community to invest in maintaining packages that have extensive use. As mentioned above, that resulted in historically poor interactions with meson & pkg-config because they are so under-utilized on Windows.

Setting that aside, I think an ideal place to start would be to consider vcpkg locations for .pc files. Given a vcpkg install tree at $VCPKG_INSTALL, we store the pkg-config files for a target under $VCPKG_INSTALL/<target-triplet>/lib/pkgconfig/*.pc for "release" and $VCPKG_INSTALL/<target-triplet>/debug/lib/pkgconfig/*.pc for "debug". The distinction matters on Windows because the release and debug CRTs have different malloc/free functions, whereas on Linux it's mostly ignorable.

[gdesmott]

Thanks for all these info @ras0219-msft

It is our intent to supply standard, valid pkg-config files on all platforms for all libraries that have upstream support.

I'm happy to read that, this will save us from adding support to another mechanism to check deps.

So what would you suggest as the right road to ease integration between system-deps and vcpkg? As I understood it, the only thing needed would be to set PKG_CONFIG_PATH with the path you mentioned.
Do you have a standard env variable system-deps could get $VCPKG_INSTALL from to build this path?

[ras0219-msft]

Unfortunately it isn't quite as simple to get $VCPKG_INSTALL as reading from an environment variable. vcpkg allows users to have many parallel install trees on their machine, with our newest manifest mode encouraging a unique tree per project.

I think we should follow the conventions set by https://github.com/mcgoo/vcpkg-rs and https://crates.io/crates/cargo-vcpkg to start with (I'm not the author of those, but I advised during their initial implementation). I don't think I could summarize things any better than just copying parts of their docs, so I'll simply link the standard location: https://docs.rs/vcpkg/latest/vcpkg/.

If I recall correctly, those crates attempt to read directly into the vcpkg metadata to provide users integration with all vcpkg packages, not just those providing pkg-config files. Perhaps that's a path worth exploring in the future (system-deps requesting dependency info from cargo-vcpkg), but I think we can get a lot of milage out of just pkg-config for now.

Maybe the right answer is to depend upon that crate and ask it to expose a function to get the current target's install tree location? Then it would be something like (apologies for the poor rust and lack of error handling)

vcpkg::target_installed_dir().unwrap().join("lib").join("pkgconfig")

+@mcgoo for their insight on this

[gdesmott]

Perhaps that's a path worth exploring in the future (system-deps requesting dependency info from cargo-vcpkg), but I think we can get a lot of milage out of just pkg-config for now.

Yes, I'd rather keep using pkg-config as the one and only way to retrieve deps until we have a very good reason to add another system.

Maybe the right answer is to depend upon that crate and ask it to expose a function to get the current target's install tree location?

Looks like there is already a public find_vcpkg_root(). Would this do?

The vcpkg does not have any dep so we could easily depends on it in system-deps.

[ras0219-msft]

find_vcpkg_root() or find_vcpkg_target(...) look pretty good, though skimming through I don't think those have been properly updated to handle manifest mode.

We've fully released manifest mode in the last ~2 years, which changes the location of files from being $VCPKG_ROOT/installed/<triplet>/... to either $PROJECT/vcpkg_installed/<triplet>/... or completely customizable by the user (such as being put into the target-specific build directory). find_vcpkg_root() will get you $VCPKG_ROOT, but that only works for the first case. The additional logic required (in vcpkg-rs) is probably:

1. Check for some env var and use that (e.g. $VCPKGRS_INSTALLED/<triplet>/...)
2. Check for the existence of a vcpkg.json next to the top-level Cargo.toml and if so use $that/vcpkg_installed/<triplet>/....

Obviously in the case that the user is using cargo vcpkg we'd want that to line up as well.

Maybe the next steps are to open an issue on the vcpkg crate cross-referencing this and asking how they'd like to handle this / what api to expose?

Or to just get things moving you could use the find_vcpkg_target() api. I think Config::get_target_triplet is an api to get at the triplet?

[gdesmott]

Maybe the next steps are to open an issue on the vcpkg crate cross-referencing this and asking how they'd like to handle this / what api to expose?

Yes, I'd prefer to have this solved in the vcpkg crate(s) first.

Can you please open such issue? I'd rather have someone actually understanding the problem do it. ;)

[Be-ing]

Hi @ras0219-msft, good to see you in the Rust world. :) I would also like to have better integration of vcpkg and Cargo. This would make it much more practical to use bindings to C & C++ libraries in cross platform Rust applications. I don't like the common practice of Rust bindings to C libraries vendoring the C library. I think Cargo should stick to packaging Rust and not try to take responsibilities for other languages, so integrating vcpkg and Cargo would be great IMO. Having setup vcpkg for two large, complex cross platform C++ applications, I think vcpkg is the most practical way to manage cross platform C & C++ dependencies.

FWIW, just thinking about the difficulty of shipping GTK on Windows and macOS made me not even seriously consider the GTK Rust bindings when considering what GUI library to use for my application (I ended up going with Slint which I'm happy with so far).

I agree with @gdesmott that we should rely on existing mechanisms for locating C & C++ dependencies, particularly pkgconfig. I just did a quick experiment the other day on x86-64 Linux setting RUSTFLAGS="-L /path/to/vcpkg_installed/lib" PKG_CONFIG_PATH="/path/to/vcpkg_installed/lib/pkgconfig" with a crate that uses the pkg_config crate in its build.rs and it worked. It would be great if there was a crate to consume CMake config modules as well (I see you've discussed that here in #3). I don't think it would be a good idea to add vcpkg-specific metadata or code to every C binding crate. That would not scale well across the Rust ecosystem.

IMO Rust crates should just need to say they want to find a C or C++ library with pkgconfig. Like with C & C++, I think it should be Rust applications', not libraries', responsibility to integrate with vcpkg if they want. I would like the Cargo & vcpkg integration to work similarly to CMake & vcpkg integration. That is, add a vcpkg.json manifest and vcpkg as a submodule to the application repository. Then, have cargo build automatically bootstrap vcpkg (if not already done), invoke vcpkg to build the libraries specified in the vcpkg.json manifest file, print cargo:rustc-link-search, and set the PKG_CONFIG_PATH environment variable. I think it would be nice to have vcpkg install inside the target directory like how it installs inside CMake's build directory. The target directory is already in .gitignore for Rust applications.

I'm not sure the existing vcpkg-rs and cargo-vcpkg crates are really useful to implement what I described above for using vcpkg in manifest mode. Those crates predate vcpkg's manifest mode and rely on each individual Rust crate which binds a C library to add their own code to integrate with vcpkg. I think we may need a new crate or a major rewrite of cargo-vcpkg to do this. Ideally it would be as simple for applications as adding one line to build.rs to bootstrap vcpkg and call vcpkg install.

Perhaps this will require some changes in vcpkg-tool as well as new Rust code to make it all work together nicely. It would be great if Microsoft could invest some effort into this. I think some on the vcpkg team know Rust, perhaps you could work on this? Might be worth reaching out to the windows-rs team too.

cc @MrMuetze, @waych, @mcgoo