From 7c4623962cb362c0b0b038e6b8a78867d31eefda Mon Sep 17 00:00:00 2001 From: Tully Foote Date: Tue, 15 Jun 2021 14:10:25 -0700 Subject: [PATCH 01/13] add verbiage about overriding 3rdparty content The majority of the content is from: https://discourse.ros.org/t/release-package-requiring-custom-version-of-system-dependency/18200/6?u=tfoote Following up: https://github.com/ros/rosdistro/pull/29614#issuecomment-850466694 --- CONTRIBUTING.md | 31 +++++++++++++++++++++++++++++++ 1 file changed, 31 insertions(+) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 855844a2cea9b..182d3af035fb6 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -46,6 +46,37 @@ Additionally, it may be necessary to determine build failures, these jobs are lo If a package continuously fails to build, the ROS Boss for that distribution may choose to revert the pull request that introduced it. Or they may take [other actions](https://github.com/ros-infrastructure/ros_buildfarm/blob/master/doc/ongoing_operations.rst) to avoid repeated failures. +### Releasing third party packages into a ROS distribution + +In general you should not override or replace system dependencies from the system distribution with a package in the ROS distribution. +This is part of the software engineering process of maintaining the distribution that all developers can rely on specific versions of software to be available and remain consistent withing the distribution. +If you need newer versions of underlying libraries the you should either target newer versions with the minimum version available or if that’s not available help and encourage the upstream process to move forward to at least your minimum version for the next release of the distribution. + +Obviously this is a slow process and doesn’t fix things immediately for a short term there are workarounds, however if you want to release packages which build on the workarounds you must make sure not to break other users who are expecting the system version. + +To release a ROS package which will overlay a system dependency it must: + +* Install side by side with the system one, with a different name. +* Not cause compile errors for any packages including downstream ones +* Not cause linking errors or crashes at runtime for yours or any downstream packages, including ones that use the system version on any of our target platforms. + +The release pipeline will only catch the first issue automatically. +Manual review is required for the latter two. +But in general if you want to do this without being disruptive you need to adjust paths to avoid installation collisions. +Change all namespaces to avoid symbol collisions. And change header paths to not collide with the system ones if you export the include directories. + +Embedding a copy inside your package, potentially a statically linking a version into your executables is also a potential way to do it as well. +It’s going to require a non-trivial amount of engineering and modifications to make this possible and most people decide it’s not worth it. + +In many situations a common approach for this is to not actually release it onto the public distro and distribute it separately through a side channel. +This will allow people to opt into the potentially disruptive change. +An example of this is Ubuntu’s PPAs or just instructions for end users to compile from source. + +In the case that a package is released into a ROS distribution that is then later available in upstream system distributions when the ROS distribution rolls forward the ROS packages should be removed in favor of relying on the system version. +To provide a gentle transition a placeholder package can be provided which will just depend on the rosdep key but this should be considered a deprecated package and dependent packages should switch to use the rosdep key. +This transitional package is likely something that should be scoped to be released into rolling only and not be in a subsequent released distribution. +It is up to the judgement of the ROS Distro's release manager to make an exception. + Documentation Indexing ---------------------- From 49bcd5ac3f1809308b070e0071f67a7102fe09aa Mon Sep 17 00:00:00 2001 From: Tully Foote Date: Tue, 15 Jun 2021 14:43:07 -0700 Subject: [PATCH 02/13] Update CONTRIBUTING.md Co-authored-by: Chris Lalancette --- CONTRIBUTING.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 182d3af035fb6..3119a9ebcf9bb 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -65,7 +65,7 @@ Manual review is required for the latter two. But in general if you want to do this without being disruptive you need to adjust paths to avoid installation collisions. Change all namespaces to avoid symbol collisions. And change header paths to not collide with the system ones if you export the include directories. -Embedding a copy inside your package, potentially a statically linking a version into your executables is also a potential way to do it as well. +Embedding a copy inside your package, potentially statically linking a version into your executables is also a potential way to do it as well. It’s going to require a non-trivial amount of engineering and modifications to make this possible and most people decide it’s not worth it. In many situations a common approach for this is to not actually release it onto the public distro and distribute it separately through a side channel. From 6bb86100fedd970eb504b38ad61f409eab85d790 Mon Sep 17 00:00:00 2001 From: Tully Foote Date: Tue, 15 Jun 2021 14:43:30 -0700 Subject: [PATCH 03/13] Update CONTRIBUTING.md Co-authored-by: Chris Lalancette --- CONTRIBUTING.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 3119a9ebcf9bb..7ac7070ee9f5f 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -46,7 +46,7 @@ Additionally, it may be necessary to determine build failures, these jobs are lo If a package continuously fails to build, the ROS Boss for that distribution may choose to revert the pull request that introduced it. Or they may take [other actions](https://github.com/ros-infrastructure/ros_buildfarm/blob/master/doc/ongoing_operations.rst) to avoid repeated failures. -### Releasing third party packages into a ROS distribution +### Releasing third party system packages into a ROS distribution In general you should not override or replace system dependencies from the system distribution with a package in the ROS distribution. This is part of the software engineering process of maintaining the distribution that all developers can rely on specific versions of software to be available and remain consistent withing the distribution. From 5dbbd2920d928f953bbde565968b213975fc41bf Mon Sep 17 00:00:00 2001 From: Tully Foote Date: Thu, 17 Jun 2021 10:11:26 -0700 Subject: [PATCH 04/13] Update CONTRIBUTING.md --- CONTRIBUTING.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 7ac7070ee9f5f..796ecb81b3953 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -52,7 +52,10 @@ In general you should not override or replace system dependencies from the syste This is part of the software engineering process of maintaining the distribution that all developers can rely on specific versions of software to be available and remain consistent withing the distribution. If you need newer versions of underlying libraries the you should either target newer versions with the minimum version available or if that’s not available help and encourage the upstream process to move forward to at least your minimum version for the next release of the distribution. -Obviously this is a slow process and doesn’t fix things immediately for a short term there are workarounds, however if you want to release packages which build on the workarounds you must make sure not to break other users who are expecting the system version. +Obviously this is a slow process and doesn’t fix things immediately for a short term. +There are workarounds such as embedding copies of the libraries into your package or making another package with the newer library version. +This usually can be made to work well in your local development environment. +However if you want to release this style of workarounds you must make sure not to break other users who are expecting the system version either in downstream packages or elsewhere on the system. To release a ROS package which will overlay a system dependency it must: From 8e89679936ab953e410b65ddb57e2b6d7ab44e1d Mon Sep 17 00:00:00 2001 From: Tully Foote Date: Thu, 17 Jun 2021 10:11:41 -0700 Subject: [PATCH 05/13] Update CONTRIBUTING.md Co-authored-by: Jacob Perron --- CONTRIBUTING.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 796ecb81b3953..f67d204bb86e5 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -49,7 +49,7 @@ Or they may take [other actions](https://github.com/ros-infrastructure/ros_build ### Releasing third party system packages into a ROS distribution In general you should not override or replace system dependencies from the system distribution with a package in the ROS distribution. -This is part of the software engineering process of maintaining the distribution that all developers can rely on specific versions of software to be available and remain consistent withing the distribution. +This is part of the software engineering process of maintaining the distribution that all developers can rely on specific versions of software to be available and remain consistent within the distribution. If you need newer versions of underlying libraries the you should either target newer versions with the minimum version available or if that’s not available help and encourage the upstream process to move forward to at least your minimum version for the next release of the distribution. Obviously this is a slow process and doesn’t fix things immediately for a short term. From 2830a4cf796e115cb246c001923386654b2ff9f4 Mon Sep 17 00:00:00 2001 From: Tully Foote Date: Thu, 17 Jun 2021 10:12:05 -0700 Subject: [PATCH 06/13] Update CONTRIBUTING.md Co-authored-by: Jacob Perron --- CONTRIBUTING.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index f67d204bb86e5..b164e26633d74 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -76,7 +76,8 @@ This will allow people to opt into the potentially disruptive change. An example of this is Ubuntu’s PPAs or just instructions for end users to compile from source. In the case that a package is released into a ROS distribution that is then later available in upstream system distributions when the ROS distribution rolls forward the ROS packages should be removed in favor of relying on the system version. -To provide a gentle transition a placeholder package can be provided which will just depend on the rosdep key but this should be considered a deprecated package and dependent packages should switch to use the rosdep key. +To provide a gentle transition, a placeholder package can be provided that only depends on the rosdep key. +This should be considered a deprecated package and dependent packages should switch to use the rosdep key. This transitional package is likely something that should be scoped to be released into rolling only and not be in a subsequent released distribution. It is up to the judgement of the ROS Distro's release manager to make an exception. From a20da04276d9509fdd7623ce3df803f7e766553b Mon Sep 17 00:00:00 2001 From: Tully Foote Date: Thu, 2 Sep 2021 02:45:46 -0700 Subject: [PATCH 07/13] Update CONTRIBUTING.md MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: Steven! Ragnarök --- CONTRIBUTING.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index b164e26633d74..74258b0823fb2 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -52,7 +52,7 @@ In general you should not override or replace system dependencies from the syste This is part of the software engineering process of maintaining the distribution that all developers can rely on specific versions of software to be available and remain consistent within the distribution. If you need newer versions of underlying libraries the you should either target newer versions with the minimum version available or if that’s not available help and encourage the upstream process to move forward to at least your minimum version for the next release of the distribution. -Obviously this is a slow process and doesn’t fix things immediately for a short term. +Obviously this is a slow process and doesn’t fix things immediately. There are workarounds such as embedding copies of the libraries into your package or making another package with the newer library version. This usually can be made to work well in your local development environment. However if you want to release this style of workarounds you must make sure not to break other users who are expecting the system version either in downstream packages or elsewhere on the system. From f7d6b4c0b17ed6ac0ac1d408e2f812c01dac1bb8 Mon Sep 17 00:00:00 2001 From: Tully Foote Date: Thu, 2 Sep 2021 02:46:55 -0700 Subject: [PATCH 08/13] Update CONTRIBUTING.md Co-authored-by: Chris Lalancette --- CONTRIBUTING.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 74258b0823fb2..289a0a9865a59 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -55,7 +55,7 @@ If you need newer versions of underlying libraries the you should either target Obviously this is a slow process and doesn’t fix things immediately. There are workarounds such as embedding copies of the libraries into your package or making another package with the newer library version. This usually can be made to work well in your local development environment. -However if you want to release this style of workarounds you must make sure not to break other users who are expecting the system version either in downstream packages or elsewhere on the system. +However if you want to release this style of workarounds into a ROS distribution you must make sure not to break other users who are expecting the system version either in downstream packages or elsewhere on the system. To release a ROS package which will overlay a system dependency it must: From 0f1d65bf0d64678743d9dd78821bc9d7f9f8c486 Mon Sep 17 00:00:00 2001 From: Tully Foote Date: Thu, 2 Sep 2021 02:47:13 -0700 Subject: [PATCH 09/13] Update CONTRIBUTING.md Co-authored-by: Chris Lalancette --- CONTRIBUTING.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 289a0a9865a59..1e099ab33e9be 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -65,8 +65,10 @@ To release a ROS package which will overlay a system dependency it must: The release pipeline will only catch the first issue automatically. Manual review is required for the latter two. -But in general if you want to do this without being disruptive you need to adjust paths to avoid installation collisions. -Change all namespaces to avoid symbol collisions. And change header paths to not collide with the system ones if you export the include directories. +But in general if you want to do this without being disruptive you need to: +1. Adjust paths to avoid installation collisions. +2. Change all namespaces to avoid symbol collisions +3. Change header paths to not collide with the system ones if you export the include directories Embedding a copy inside your package, potentially statically linking a version into your executables is also a potential way to do it as well. It’s going to require a non-trivial amount of engineering and modifications to make this possible and most people decide it’s not worth it. From 911972b2a5b1f4dc373c395c0a352f4c0153c19d Mon Sep 17 00:00:00 2001 From: Chris Lalancette Date: Tue, 7 Sep 2021 08:40:52 -0400 Subject: [PATCH 10/13] Update CONTRIBUTING.md Co-authored-by: Tully Foote --- CONTRIBUTING.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 1e099ab33e9be..f2c77ed22082b 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -50,7 +50,9 @@ Or they may take [other actions](https://github.com/ros-infrastructure/ros_build In general you should not override or replace system dependencies from the system distribution with a package in the ROS distribution. This is part of the software engineering process of maintaining the distribution that all developers can rely on specific versions of software to be available and remain consistent within the distribution. -If you need newer versions of underlying libraries the you should either target newer versions with the minimum version available or if that’s not available help and encourage the upstream process to move forward to at least your minimum version for the next release of the distribution. +If you truly need a newer versions of underlying libraries the you can consider targeting a newer rosdistro with which will target newer platforms which hopefully will have a higher version of your necessary dependency. +If the necessary version of your dependency isn't available even on the latest platforms, please reach out and encourage the upstream platforms to move forward to at least your minimum version for the next release of the upstream distributions. +By doing this you will make sure that it's available for everyone in the future. Obviously this is a slow process and doesn’t fix things immediately. There are workarounds such as embedding copies of the libraries into your package or making another package with the newer library version. From 3df1cfd28a78d3897ca17a75b726f77143a04c83 Mon Sep 17 00:00:00 2001 From: Chris Lalancette Date: Tue, 7 Sep 2021 08:44:06 -0400 Subject: [PATCH 11/13] Apply suggestions from code review --- CONTRIBUTING.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index f2c77ed22082b..3f9e9d82fc2d5 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -50,19 +50,19 @@ Or they may take [other actions](https://github.com/ros-infrastructure/ros_build In general you should not override or replace system dependencies from the system distribution with a package in the ROS distribution. This is part of the software engineering process of maintaining the distribution that all developers can rely on specific versions of software to be available and remain consistent within the distribution. -If you truly need a newer versions of underlying libraries the you can consider targeting a newer rosdistro with which will target newer platforms which hopefully will have a higher version of your necessary dependency. -If the necessary version of your dependency isn't available even on the latest platforms, please reach out and encourage the upstream platforms to move forward to at least your minimum version for the next release of the upstream distributions. +If you truly need a newer versions of underlying libraries then you can consider targeting a newer ROS distributionthat targets a newer platform which hopefully will have a higher version of your necessary dependency. +If the necessary version of your dependency isn't available even on the latest platforms, please reach out and encourage the upstream operating systems to move forward to at least your minimum version for the next release of the upstream distributions. By doing this you will make sure that it's available for everyone in the future. Obviously this is a slow process and doesn’t fix things immediately. There are workarounds such as embedding copies of the libraries into your package or making another package with the newer library version. This usually can be made to work well in your local development environment. -However if you want to release this style of workarounds into a ROS distribution you must make sure not to break other users who are expecting the system version either in downstream packages or elsewhere on the system. +However if you want to release this style of workaround into a ROS distribution you must make sure not to break other users who are expecting the system version either in downstream packages or elsewhere on the system. To release a ROS package which will overlay a system dependency it must: * Install side by side with the system one, with a different name. -* Not cause compile errors for any packages including downstream ones +* Not cause compile errors for any packages including downstream ones. * Not cause linking errors or crashes at runtime for yours or any downstream packages, including ones that use the system version on any of our target platforms. The release pipeline will only catch the first issue automatically. @@ -72,7 +72,7 @@ But in general if you want to do this without being disruptive you need to: 2. Change all namespaces to avoid symbol collisions 3. Change header paths to not collide with the system ones if you export the include directories -Embedding a copy inside your package, potentially statically linking a version into your executables is also a potential way to do it as well. +Embedding a copy inside your package, potentially statically linking a version into your executables is another way to do it as well. It’s going to require a non-trivial amount of engineering and modifications to make this possible and most people decide it’s not worth it. In many situations a common approach for this is to not actually release it onto the public distro and distribute it separately through a side channel. From d2ebab7198a9ec39cd80b72cb259e3cc6150c802 Mon Sep 17 00:00:00 2001 From: Chris Lalancette Date: Tue, 7 Sep 2021 08:44:33 -0400 Subject: [PATCH 12/13] Update CONTRIBUTING.md --- CONTRIBUTING.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 3f9e9d82fc2d5..f5c896362474e 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -50,7 +50,7 @@ Or they may take [other actions](https://github.com/ros-infrastructure/ros_build In general you should not override or replace system dependencies from the system distribution with a package in the ROS distribution. This is part of the software engineering process of maintaining the distribution that all developers can rely on specific versions of software to be available and remain consistent within the distribution. -If you truly need a newer versions of underlying libraries then you can consider targeting a newer ROS distributionthat targets a newer platform which hopefully will have a higher version of your necessary dependency. +If you truly need newer versions of underlying libraries then you can consider targeting a newer ROS distributionthat targets a newer platform which hopefully will have a higher version of your necessary dependency. If the necessary version of your dependency isn't available even on the latest platforms, please reach out and encourage the upstream operating systems to move forward to at least your minimum version for the next release of the upstream distributions. By doing this you will make sure that it's available for everyone in the future. From ac05b33fc263cb2460a8bdbd2d1c6185965ca465 Mon Sep 17 00:00:00 2001 From: Chris Lalancette Date: Tue, 7 Sep 2021 08:44:56 -0400 Subject: [PATCH 13/13] Update CONTRIBUTING.md --- CONTRIBUTING.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index f5c896362474e..60fff324b0cd6 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -50,7 +50,7 @@ Or they may take [other actions](https://github.com/ros-infrastructure/ros_build In general you should not override or replace system dependencies from the system distribution with a package in the ROS distribution. This is part of the software engineering process of maintaining the distribution that all developers can rely on specific versions of software to be available and remain consistent within the distribution. -If you truly need newer versions of underlying libraries then you can consider targeting a newer ROS distributionthat targets a newer platform which hopefully will have a higher version of your necessary dependency. +If you truly need newer versions of underlying libraries then you can consider targeting a newer ROS distribution that targets a newer platform which hopefully will have a higher version of your necessary dependency. If the necessary version of your dependency isn't available even on the latest platforms, please reach out and encourage the upstream operating systems to move forward to at least your minimum version for the next release of the upstream distributions. By doing this you will make sure that it's available for everyone in the future.