Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[docs] Tweak package_templates #22325

Closed
wants to merge 19 commits into from

Conversation

valgur
Copy link
Contributor

@valgur valgur commented Jan 14, 2024

Adds notes for a few important but missing details, which I have encountered when creating or updating packages.
Removes a few outdated comments and improves the wording a bit of several others.

I'll add comments for specific changes as review comments.

url:
- "https://mirror1.net/package-1.1.0.tar.gz"
- "https://mirror2.net/package-1.1.0.tar.gz"
url: "https://mirror2.net/package-1.1.0.tar.gz"
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To provide an example for newcomers of the more common non-list format for the URLs.

@@ -22,13 +22,13 @@ class PackageConan(ConanFile):
name = "package"
description = "short description"
# Use short name only, conform to SPDX License List: https://spdx.org/licenses/
# In case not listed there, use "LicenseRef-<license-file-name>"
# In case not listed there, use "DocumentRef-<license-file-name>:LicenseRef-<package-name>"
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The DocumentRef format has been recommended in all recent PRs where a non-standard license has been used.

@@ -62,7 +62,6 @@ def _settings_build(self):
return getattr(self, "settings_build", self.settings)

# no exports_sources attribute, but export_sources(self) method instead
# this allows finer grain exportation of patches per version
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A bit confusing and outdated, imo.

# prefer self.requires method instead of requires attribute
# Prefer self.requires method instead of requires attribute
# Set transitive_headers=True (which usually also requires transitive_libs=True) if
# the dependency is used in any of the packaged header files.
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I considered adding something like "Including a comment with a link to the the public header file using the dependency is encouraged." but it seemed a bit too wordy for me.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As this is a template, I'd try to write that even if wordy, I think it's worth the space.

Maybe have 2 requires one without and one with traits, and leave the traits bit for the second one?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Keeping all the explanation in the same line is more confusing, do it in separated ways. Keep the previous comment, then, add a new dummy requirement that requires transitive flags and add a dummy comment like: "used by foo/baz.hpp:34"

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good idea. Thanks!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

def build_requirements(self):
# only if we have to call autoreconf
self.tool_requires("libtool/x.y.z")
self.tool_requires("libtool/2.4.7")
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While the versions should be always checked and updated, the libtool and automake versions change very rarely, so it does not hurt to include them for convenience.

tc = AutotoolsDeps(self)
tc.generate()
# some recipes might require a workaround for MSVC:
# https://github.com/conan-io/conan-center-index/blob/00ce907b910d0d772f1c73bb699971c141c423c1/recipes/xapian-core/all/conanfile.py#L106-L135
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is an unfortunately rather frequently required workaround. I think it's beneficial to include it as (a) the ./configure error is just a cryptic "cannot create C executables" and (b) the fix is not at all obvious.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please, add a related issue too to be more visible about that idiom and will be done in the future.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

@@ -158,7 +158,7 @@ def generate(self):
env.define("CC", f"{compile_wrapper} cl -nologo")
env.define("CXX", f"{compile_wrapper} cl -nologo")
env.define("LD", "link -nologo")
env.define("AR", f"{ar_wrapper} \"lib -nologo\"")
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The -nologo option saves a few spammy lines from the build logs, but it has caused the build to fail with a "could not determine ar interface" in several recipes for me, so it's safer to not include it.


def source(self):
get(self, **self.conan_data["sources"][self.version], strip_root=True)

def generate(self):
# BUILD_SHARED_LIBS and POSITION_INDEPENDENT_CODE are automatically parsed when self.options.shared or self.options.fPIC exist
# BUILD_SHARED_LIBS and POSITION_INDEPENDENT_CODE are set automatically as tc.variables when self.options.shared or self.options.fPIC exist
# Note that tc.variables require either cmake_minimum_required() >= 3.13 or the CMP0077 policy set to NEW to work correctly.
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is what prompted me to open this PR in the first place. It's a huge pitfall for newcomers, which can cause silent and non-obvious failures if it is not accounted for.

tc = CMakeToolchain(self)
# Boolean values are preferred instead of "ON"/"OFF"
tc.variables["PACKAGE_CUSTOM_DEFINITION"] = True
tc.variables["PACKAGE_BUILD_TESTS"] = False
Copy link
Contributor Author

@valgur valgur Jan 14, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A slightly more relevant example.

def build_requirements(self):
self.tool_requires("tool/x.y.z")
self.tool_requires("cmake/[>=3.16 <4]")
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A slightly more typical example for the CMake recipe.

if self.dependencies["dependency"].options.foobar:
tc.variables["DEPENDENCY_LIBPATH"] = self.dependencies["dependency"].cpp_info.libdirs
# cache_variables should be used sparingly, example setting cmake policies
tc.variables["DEPENDENCY_LIBPATH"] = self.dependencies["dependency"].cpp_info.libdir.replace("\\", "/")
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Need to convert to forward slashes on Windows for CMake not to interpret the backslashes as escaped characters.

Comment on lines -106 to -108
# don't use self.settings.compiler.runtime
tc.variables["USE_MSVC_RUNTIME_LIBRARY_DLL"] = not is_msvc_static_runtime(self)
# deps_cpp_info, deps_env_info and deps_user_info are no longer used
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No longer relevant with Conan 2.0 being commonplace.

Comment on lines +129 to +136
# disable subdirectories by truncating their CMakeLists.txt
save(self, os.path.join(self.source_folder, "tests", "CMakeLists.txt"), "")
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

test and examples directories need to be disabled quite often and this approach requires less maintenance overhead compared to patching CMakeLists.txt.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would not include it. It encourages people doing more patches instead of trying to check it with the upstream an alternative. Remember, patches are the last resource always.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My worry with this as explained elsewhere is that in the case where a new version changes how their tests etc are set up, we won't fail as it would have happened with patches, thus making it easier to sneak tests and the like without noticing. wdyt?

Copy link
Contributor Author

@valgur valgur Jan 15, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not both - use this as a patch and submit a fix upstream as well? 😁

Using plain patches is better in principle, but the add_subdirectory() statements are typically one of the very last ones in a CMakeLists.txt file, which for larger projects means that this needs to be updated for almost every single release due to the line numbers shifting. I can remove this example here, but in general, would a replace_in_file() with strict=True be acceptable instead?

@@ -139,9 +144,7 @@ def package(self):
rmdir(self, os.path.join(self.package_folder, "lib", "pkgconfig"))
rmdir(self, os.path.join(self.package_folder, "lib", "cmake"))
rmdir(self, os.path.join(self.package_folder, "share"))
rm(self, "*.la", os.path.join(self.package_folder, "lib"))
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have never encountered a *.la file in a CMake project.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good catch

Comment on lines 168 to 184
# To export additional CMake variables, such as upper-case variables otherwise set by the project's *-config.cmake,
# you can copy or save a .cmake file under <prefix>/lib/cmake/ with content like
# set(XYZ_VERSION ${${CMAKE_FIND_PACKAGE_NAME}_VERSION})
# set(XYZ_INCLUDE_DIRS ${${CMAKE_FIND_PACKAGE_NAME}_INCLUDE_DIRS})
# ...
# and set the following fields:
self.cpp_info.builddirs.append(os.path.join("lib", "cmake"))
cmake_module = os.path.join("lib", "cmake", "conan-official-variables.cmake")
self.cpp_info.set_property("cmake_build_modules", [cmake_module])
self.cpp_info.build_modules["cmake_find_package"] = [cmake_module]
self.cpp_info.build_modules["cmake_find_package_multi"] = [cmake_module]
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Matching the *-config.cmake output created by the project is often overlooked, imo, and it's difficult to find a good example on how to handle this as a newbie.
${CMAKE_FIND_PACKAGE_NAME} is preferable over a hard-coded package name as it allows the name to be overriden in CMakeDeps without breaking things.

basic_layout(self, src_folder="src")

def requirements(self):
# Prefer self.requires method instead of requires attribute
# Direct dependencies of header only libs are always transitive since they are included in public headers
self.requires("dependency/0.8.1", transitive_headers=True)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

transitive_headers=True is already the default for package_type = "header-library".

platform_toolset = MSBuildToolchain(self).toolset
import_conan_generators = ""
for props_file in ["conantoolchain.props", "conandeps.props"]:
props_path = os.path.join(self.generators_folder, props_file)
if os.path.exists(props_path):
import_conan_generators += f"<Import Project=\"{props_path}\" />"
for vcxproj_file in vcxproj_files:
for vcxproj_file in self.source_path.rglob("*.vcxproj"):
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does not work 100% of the time, but usually does, and is more convenient than explicitly listing all of the vcxproj files.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any example where it does not work?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't remember any, really.

@conan-center-bot

This comment has been minimized.

@AbrilRBS AbrilRBS self-assigned this Jan 15, 2024
Copy link
Member

@uilianries uilianries left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looking good, some observations.

@@ -78,11 +77,12 @@ def configure(self):
self.settings.rm_safe("compiler.libcxx")

def layout(self):
# src_folder must use the same source folder name the project
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please, keep this comment, it explains why do we use src_folder="src"

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A comment would not hurt, but the current wording is kind of broken, imo. Could you provide a better one, maybe?

# prefer self.requires method instead of requires attribute
# Prefer self.requires method instead of requires attribute
# Set transitive_headers=True (which usually also requires transitive_libs=True) if
# the dependency is used in any of the packaged header files.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Keeping all the explanation in the same line is more confusing, do it in separated ways. Keep the previous comment, then, add a new dummy requirement that requires transitive flags and add a dummy comment like: "used by foo/baz.hpp:34"

# only if upstream configure.ac relies on PKG_CHECK_MODULES macro
if not self.conf.get("tools.gnu:pkg_config", check_type=str):
self.tool_requires("pkgconf/x.y.z")
self.tool_requires("pkgconf/2.1.0")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please, revert to x.y.z. Otherwise, people will not check the latest version available when using the template.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok. The Meson recipe template already uses real versions, though. Should I blank these out as well?

@@ -114,20 +114,18 @@ def build_requirements(self):
# for msvc support to get compile & ar-lib scripts (may be avoided if shipped in source code of the library)
# not needed if libtool already in build requirements
if is_msvc(self):
self.tool_requires("automake/x.y.z")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto.

tc = AutotoolsDeps(self)
tc.generate()
# some recipes might require a workaround for MSVC:
# https://github.com/conan-io/conan-center-index/blob/00ce907b910d0d772f1c73bb699971c141c423c1/recipes/xapian-core/all/conanfile.py#L106-L135
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please, add a related issue too to be more visible about that idiom and will be done in the future.


# In case there are dependencies listed under requirements, CMakeDeps should be used
deps = CMakeDeps(self)
# You can override the CMake package and target names if they don't match the names used in the project
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice addition.

Comment on lines +129 to +136
# disable subdirectories by truncating their CMakeLists.txt
save(self, os.path.join(self.source_folder, "tests", "CMakeLists.txt"), "")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would not include it. It encourages people doing more patches instead of trying to check it with the upstream an alternative. Remember, patches are the last resource always.

@@ -139,9 +144,7 @@ def package(self):
rmdir(self, os.path.join(self.package_folder, "lib", "pkgconfig"))
rmdir(self, os.path.join(self.package_folder, "lib", "cmake"))
rmdir(self, os.path.join(self.package_folder, "share"))
rm(self, "*.la", os.path.join(self.package_folder, "lib"))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good catch

Copy link
Member

@AbrilRBS AbrilRBS left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks a lot for this @valgur, we were due taking a look at this. I've some comments on some of the changes, otherwise it's great, thanks!!

# prefer self.requires method instead of requires attribute
# Prefer self.requires method instead of requires attribute
# Set transitive_headers=True (which usually also requires transitive_libs=True) if
# the dependency is used in any of the packaged header files.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As this is a template, I'd try to write that even if wordy, I think it's worth the space.

Maybe have 2 requires one without and one with traits, and leave the traits bit for the second one?

Comment on lines +129 to +136
# disable subdirectories by truncating their CMakeLists.txt
save(self, os.path.join(self.source_folder, "tests", "CMakeLists.txt"), "")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My worry with this as explained elsewhere is that in the case where a new version changes how their tests etc are set up, we won't fail as it would have happened with patches, thus making it easier to sneak tests and the like without noticing. wdyt?

# prefer self.requires method instead of requires attribute
# Prefer self.requires method instead of requires attribute
# Set transitive_headers=True (which usually also requires transitive_libs=True) if
# the dependency is used in any of the packaged header files.
self.requires("dependency/0.8.1")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe add an example of a version range fake dependency, which a comment linking to the FAQ for the allowed ones?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

platform_toolset = MSBuildToolchain(self).toolset
import_conan_generators = ""
for props_file in ["conantoolchain.props", "conandeps.props"]:
props_path = os.path.join(self.generators_folder, props_file)
if os.path.exists(props_path):
import_conan_generators += f"<Import Project=\"{props_path}\" />"
for vcxproj_file in vcxproj_files:
for vcxproj_file in self.source_path.rglob("*.vcxproj"):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any example where it does not work?

@conan-center-bot

This comment has been minimized.

@valgur valgur force-pushed the docs/mention-cmp0077 branch from 07e1cd6 to 7ce0494 Compare February 26, 2024 10:09
@conan-center-bot

This comment has been minimized.

@conan-center-bot

This comment has been minimized.

@conan-center-bot
Copy link
Collaborator

Conan v1 pipeline ❌

Changes not allowed in build 1:

[docs/package_templates/autotools_package/all/conandata.yml, docs/package_templates/autotools_package/all/conanfile.py, docs/package_templates/autotools_package/all/test_package/CMakeLists.txt, docs/package_templates/autotools_package/all/test_package/conanfile.py, docs/package_templates/cmake_package/all/conandata.yml, docs/package_templates/cmake_package/all/conanfile.py, docs/package_templates/cmake_package/all/test_package/CMakeLists.txt, docs/package_templates/cmake_package/all/test_package/conanfile.py, docs/package_templates/header_only/all/conandata.yml, docs/package_templates/header_only/all/conanfile.py, docs/package_templates/header_only/all/test_package/CMakeLists.txt, docs/package_templates/header_only/all/test_package/conanfile.py, docs/package_templates/meson_package/all/conandata.yml, docs/package_templates/meson_package/all/conanfile.py, docs/package_templates/meson_package/all/test_package/conanfile.py, docs/package_templates/msbuild_package/all/conanfile.py, docs/package_templates/msbuild_package/all/test_package/CMakeLists.txt, docs/package_templates/msbuild_package/all/test_package/conanfile.py, docs/package_templates/prebuilt_tool_package/all/conanfile.py, docs/package_templates/prebuilt_tool_package/all/test_package/conanfile.py, recipes/libmemcached/all/conanfile.py]

Changing recipes and configuration/docs files in the same pull-request is not allowed. Please, split changes into several pull-requests.


Conan v2 pipeline ❌

Note: Conan v2 builds are now mandatory. Please read our discussion about it.

The v2 pipeline failed. Please, review the errors and note this is required for pull requests to be merged. In case this recipe is still not ported to Conan 2.x, please, ping @conan-io/barbarians on the PR and we will help you.

See details:

Changes not allowed in build 1:

[docs/package_templates/autotools_package/all/conandata.yml, docs/package_templates/autotools_package/all/conanfile.py, docs/package_templates/autotools_package/all/test_package/CMakeLists.txt, docs/package_templates/autotools_package/all/test_package/conanfile.py, docs/package_templates/cmake_package/all/conandata.yml, docs/package_templates/cmake_package/all/conanfile.py, docs/package_templates/cmake_package/all/test_package/CMakeLists.txt, docs/package_templates/cmake_package/all/test_package/conanfile.py, docs/package_templates/header_only/all/conandata.yml, docs/package_templates/header_only/all/conanfile.py, docs/package_templates/header_only/all/test_package/CMakeLists.txt, docs/package_templates/header_only/all/test_package/conanfile.py, docs/package_templates/meson_package/all/conandata.yml, docs/package_templates/meson_package/all/conanfile.py, docs/package_templates/meson_package/all/test_package/conanfile.py, docs/package_templates/msbuild_package/all/conanfile.py, docs/package_templates/msbuild_package/all/test_package/CMakeLists.txt, docs/package_templates/msbuild_package/all/test_package/conanfile.py, docs/package_templates/prebuilt_tool_package/all/conanfile.py, docs/package_templates/prebuilt_tool_package/all/test_package/conanfile.py, recipes/libmemcached/all/conanfile.py]

Changing recipes and configuration/docs files in the same pull-request is not allowed. Please, split changes into several pull-requests.

valgur added 2 commits July 27, 2024 14:48
…ator props

The CMake and PkgConfig info is the most relevant part to any consumers and should come first, imo.

New recipes should not bother with the legacy generators, especially after test_v1_package is no longer included.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants