Consider versioning the RMW interface

[eboasson]

Feature request

Feature description

Every now and then there is a change to the RMW interface that necessitates updating all the RMW implementations — for example #187, which gives the application a way to control options offered by the underlying implementation that are not otherwise exposed in the regular RCL interfaces. This then obviously results in the updated RMW implementations no longer being compatible with older versions of ROS2.

Having a version number for the RMW interface allows an implementation to be compatible with a larger range of ROS2 versions using a single source version.

One could also do the same in reverse: let RMW implementations advertise the version they implement, then provide compatibility with older RMW implementations where possible with a small effort. (#187 would have been pretty trivial.) The more different implementations of the RMW interface exist outside the ROS2 sources, the more valuable this becomes.

Implementation considerations

In my view the implementation could be as simple as defining an RMW_VERSION macro in rmw.h, expanding to an integer that gets incremented whenever there is an interface change. That would have a minimal impact on managing the RMW interface but would give RMW implementers options.

While letting RMW implementations advertise the interface version they implement and providing compatibility with older RMW implementations would definitely be very nice for the RMW implementers, it would also require some real effort in maintaining the RMW interface layer. That might be a bit too much to ask for 😄 .

[wjwwood]

Thanks!

[wjwwood]

One could also do the same in reverse: let RMW implementations advertise the version they implement, then provide compatibility with older RMW implementations where possible with a small effort. (#187 would have been pretty trivial.) The more different implementations of the RMW interface exist outside the ROS2 sources, the more valuable this becomes.

This grows complexity quite quickly, and then the question becomes when do you remove such "shims" for older versions.

[wjwwood]

In my view the implementation could be as simple as defining an RMW_VERSION macro in rmw.h, expanding to an integer that gets incremented whenever there is an interface change. That would have a minimal impact on managing the RMW interface but would give RMW implementers options.

I would actually like to have this as a CMake variable instead (which dependent packages could choose to put in as compiler defines), but I guess we could generate a file in rmw, though I generally dislike generating code if it can be avoided.

Maintaining the version in two places (in the package.xml and in a header file) is generally something I would avoid as well.

[wjwwood]

The benefit of using CMake to convey the version and adding syntactic sugar to make it easy to use is that it can immediately be extended to all packages (e.g. via a patch to ament_cmake) and dependent packages could opt into having those compiler definitions, whereas generating a header with the information would require all packages to be touched.

[dirk-thomas]

Lets figure out how we want to do this (a header, a CMake variable, etc.) by collecting the pros and cons of either proposal.

That being said we should land this before the upcoming Eloquent deadline (Oct. 14th) so that RMW implementation can choose to support multiple versions of the changing RMW interface from a single branch.

[dirk-thomas]

After an offline conversation we identified that we need to distinguish that something has changed in the interface compared to the last release. We came up with the following options:

1. Add a flag when any change to the interface lands. The flag needs to be reset for the next upcoming release.

• Ruled out since the revert might be forgotten and downstream packages would need to check a combination of the package version as well as the flag.

2. Bump package version with each breaking change to the interface

• Not tagging / releasing these "in between" versions implies a gap in the changelog, no tag in the repo, etc.
• Ruled out since the side effects are undesired.

3. Bump a to-be-added integer interface version (similar to an .so version) with each breaking change

• One per package or possibly multiple within the same package.
• Either export that number as a CMake variable and/or a C define in a header file.
• Downstream packages just check against the exposed number

4. Export the "upcoming" version (latest version with interface changes)

• The value of that "upcoming" version would be higher that the current package version set in the manifest. Once a new release is being made they should match again (or in case the new version doesn't have any breaking changes the hardcoded upcoming version is older but is being ignored).
• Add a CMake variable with the "upcoming" version.
• Use a function like max("upcoming", actual pkg version) to determine the exported value (which same as in 3. could be exposed through CMake and/or C).

Unrelated: We want to make the package version (from the package.xml file) available in CMake / header to downstream packages independent of this change.

We are looking for any kind of feedback / opinion with the hope that we can get this new option implemented in time for Eloquent.

[dirk-thomas]

And now after the summary my personal opinion:

I think option 4. is less elegant. It requires more logic (combining both versions) and has that "upcoming" version embedded in the CMake code (if it isn't being removed after a future release when not needed anymore, and we don't want to do that hence we rules out option 1.). It will also require a comparison of a X.Y.Z version number in downstream packages (which would probably demand to additionally store the version number in an easy-to-compare combined - e.g. a 6 digit number with two digits per part of the version triplet).

Option 3. is simple and straight forward to implement in a package - either in CMake or as a header-only. It is also coupled with the package version which I think is a good thing. Downstream packages do a simple numeric comparison to toggle between the old and new interface.

[wjwwood]

I think option 3 has a few problems:

• you have to coordinate which pull request will update the version if two are proposed simultaneously (they will always need different values)
  • with option 4 as long as both pull requests agree the next version is the same (e.g. x.y+1.z or similar where x.y.z is the current version) there is no conflict between two racing pr's
• the version number is unrelated to the version number of the package, so if you see #if FOO_API_VERSION > 2 in code, you don't know in which version of the package that was introduced
  • with option 4 you have the package version to use as the comparison, e.g. #if FOO_VERSION_MAJOR > 2 or #if FOO_VERSION_MAJOR == 1 && FOO_VERSION_MINOR > 2 can be directly correlated to a package version (and therefore a git tag)
• you really always want to put this in CMake otherwise you cannot use this information easily in CMake to do things like conditionally include certain source files in a target, and therefore you need to generate a header if you want that as an option as well
  • if you assume you need to put option 3's version in CMake, then implementing it is similar in complexity to option 4
  • the only way option 4 is more complex is that it has to do max(current pkg version, upcoming version) first, and CMake has tools for this comparison already
  • plus we already want to unconditionally expose the package's actual version in CMake and as a header, so adding the information for either option to that is somewhat trivial in my opinion

There are some benefits to option 3:

• you can differentiate each change between the latest version and master
  • however we discussed this offline and I think a few of us agreed that this isn't something that's needed often, and being able to differentiate between the latest version and master is all that really matters
• the comparison as simpler (one number versus major.minor.patch)
  • I see this as a direct trade-off of being able to easily correlate it to a package version, and personally I don't find the major.minor.patch comparisons to be that hard
    • and since we're generating a header we can make macros to easy this comparison, as many package have, e.g. https://doc.qt.io/qt-5/qtglobal.html#QT_VERSION_CHECK
• two libraries in the same package could have different interface version numbers
  • we also discussed this and in my opinion this would be an uncommon use case and probably should be separate packages if you have two decoupled API's that need to be versioned separately, but that's just my inclination and so I don't weigh this use case very highly

Based on the above, I actually prefer option 4, since it doesn't require us to track two separate "versions" and because it's easier to associate logic in downstream packages to an upstream version (and therefore git tag).

For the drawbacks mentioned for option 4:

It will also require a comparison of a X.Y.Z version number in downstream packages (which would probably demand to additionally store the version number in an easy-to-compare combined - e.g. a 6 digit number with two digits per part of the version triplet).

I think this is more work, but not prohibitive. Since we're probably generating code, we can make macros to make this comparison easier, or we can just have the versions broken into a triplet, again we could make this easier to work with. So I do think this is a valid criticism, but not a deal breaker for me.

It requires more logic (combining both versions)

It does require more logic, but not for the end user, it's just one comparison we'd need to place in a cmake macro in a common place.

has that "upcoming" version embedded in the CMake code (if it isn't being removed after a future release when not needed anymore, and we don't want to do that hence we rules out option 1.).

I don't fully follow this completely. The reason (in my opinion) we didn't want to have to undo the flag was because if we forgot then there would be no way to tell between the new version and the master branch that builds on it (because the flag was on at the release on accident), but that's not an issue with option 4. The "upcoming" version could be removed, updated, or left alone after a new release and nothing would be broken.

---

I'd actually like to see this for option 4 (maybe call it option 5), as opposed to "upcoming" you'd just add +dev to the version. Such that any comparisons would work out as 1.0.0 < 1.0.0+dev < 1.0.1 < 1.1.0 < 2.0.0 (this would mean we'd have to use a single number like Qt does, rather than a triplet). This is how version comparison works in Python's distutils.version.LooseVersion, for example, see: https://stackoverflow.com/a/6972866/671658

Then in cmake you'd do something like this:

mark_as_development_if_version_less_than(1.2.3)

and in C++:

// assuming the last release was 1.2.2 and the above cmake line
// FOO_VERSION would be 1.2.2+dev (well, the equivalent hex value)
#if (FOO_VERSION > FOO_VERSION_CHECK(1, 2, 2))
// new thing
#else
// old thing
#endif

We can do the same as Qt and use a hex pattern (https://doc.qt.io/qt-5/qtglobal.html#QT_VERSION), e.g. 0xMMNNPPD where MM is major, NN is minor, PP is patch, and D is 1 for +dev and 0 otherwise. That's just an example, we could add/remove to that if we want.

While this is definitely more complex than a single number, it's also something we only need to implement once and it is easier to interpret when reading downstream code.

After you release 1.2.3 (following the example), FOO_VERSION would become the hex equivalent to 1.2.3 and the cmake line would become a noop and could be removed or not.

[dirk-thomas]

@eboasson Using this change implemented in ament/ament_cmake#204 and #191 you should be able to implement a conditional logic based on the exported version number (likely using rmw_VERSION_MAJOR, rmw_VERSION_MINOR, and rmw_VERSION_PATCH). In Dashing that version is 1.7.x and in Eloquent we use 1.8.x though the latest change which sparked this ticket will only land in the upcoming version 1.8.1.