Skip to content

[Help needed][gazebo] Fix gazebo dependencies#30424

Closed
talregev wants to merge 1 commit intomicrosoft:masterfrom
talregev:TalR/fix_gazebo2
Closed

[Help needed][gazebo] Fix gazebo dependencies#30424
talregev wants to merge 1 commit intomicrosoft:masterfrom
talregev:TalR/fix_gazebo2

Conversation

@talregev
Copy link
Contributor

@talregev talregev commented Mar 25, 2023

This PR is continue from my old PR:
#24969
Thank for @JackBoosY and @dg0yt for helping out.

I made a little progress, but still need to patch gazebo to fit the new ogre lib. Also gazebo using private headers from ogre that is problematic. There is open issue on gazebo that they want to compile gazebo with vcpkg on their side, I hope to catch their attention to fix all the problems with you guys.

Gazebo open issue:
gazebosim/gazebo-classic#3202

  • Changes comply with the maintainer guide
  • SHA512s are updated for each updated download
  • The "supports" clause reflects platforms that may be fixed by this new version
  • Any fixed CI baseline entries are removed from that file.
  • Any patches that are no longer applied are deleted from the port's directory.
  • The version database is fixed by rerunning ./vcpkg x-add-version --all and committing the result.
  • Only one version is added to each modified port's versions file.

Waiting for:

@talregev
Copy link
Contributor Author

talregev commented Mar 25, 2023

On my end, on the ubuntu 22.04 vm I successfully manage to configure gazebo and start build:
config-x64-linux-release-out.log

@talregev talregev force-pushed the TalR/fix_gazebo2 branch 2 times, most recently from 8291256 to 28161a1 Compare March 25, 2023 13:15
@talregev
Copy link
Contributor Author

talregev commented Mar 25, 2023

I am not sure why in the CI, gazebo need intel-mkl port. I insert it, and it still say it missing. What I am missing?

@dg0yt
Copy link
Contributor

dg0yt commented Mar 25, 2023

You actually know that there is a massive baseline regression:
intel-mkl was accidentally picked up by lapack-reference, and now the reuse of the cached lapack-reference artifact breaks downstream builds - even unrelated ones.

@talregev
Copy link
Contributor Author

talregev commented Mar 25, 2023

You actually know that there is a massive baseline regression: intel-mkl was accidentally picked up by lapack-reference, and now the reuse of the cached lapack-reference artifact breaks downstream builds - even unrelated ones.

So I need to wait for the fix? I didn't know that. Thank you for the update.

@talregev talregev force-pushed the TalR/fix_gazebo2 branch 2 times, most recently from 5e8f1c0 to 99bb0aa Compare March 25, 2023 13:35
@dg0yt
Copy link
Contributor

dg0yt commented Mar 25, 2023

You may at least fix your patch. Cf. config log:

CMake Warning (dev) in cmake/FindFreeimage.cmake:
  A logical block opening on the line

    /mnt/vcpkg-ci/buildtrees/gazebo/src/11_11.12.0-96246b10c5.clean/cmake/FindFreeimage.cmake:6 (if)

  closes on the line

    /mnt/vcpkg-ci/buildtrees/gazebo/src/11_11.12.0-96246b10c5.clean/cmake/FindFreeimage.cmake:11 (endif)

  with mis-matching arguments.
Call Stack (most recent call first):
  cmake/SearchForStuff.cmake:6 (include)
  CMakeLists.txt:163 (include)
This warning is for project developers.  Use -Wno-dev to suppress it.

@talregev
Copy link
Contributor Author

You may at least fix your patch. Cf. config log:

CMake Warning (dev) in cmake/FindFreeimage.cmake:
  A logical block opening on the line

    /mnt/vcpkg-ci/buildtrees/gazebo/src/11_11.12.0-96246b10c5.clean/cmake/FindFreeimage.cmake:6 (if)

  closes on the line

    /mnt/vcpkg-ci/buildtrees/gazebo/src/11_11.12.0-96246b10c5.clean/cmake/FindFreeimage.cmake:11 (endif)

  with mis-matching arguments.
Call Stack (most recent call first):
  cmake/SearchForStuff.cmake:6 (include)
  CMakeLists.txt:163 (include)
This warning is for project developers.  Use -Wno-dev to suppress it.

How about now?

@talregev
Copy link
Contributor Author

@dg0yt I need help to patch gazebo into the new ogre lib. Is it interesting you?

@talregev talregev force-pushed the TalR/fix_gazebo2 branch 5 times, most recently from 1f2dfb8 to a937a65 Compare March 26, 2023 02:14
@talregev
Copy link
Contributor Author

I am able to compile gazebo with ogre 1.12.9 in manifest mode.
The first command fall in ode. I manually fix the port, then continue from the second command:

vcpkg install  --x-manifest-root=ports/gazebo/ --x-install-root=installed --triplet=x64-windows-release --host-triplet=x64-windows-release
vcpkg install  gazebo --triplet=x64-windows-release --host-triplet=x64-windows-release

I have linkage error. I feel so close:
Is it qwt is missing? It already install it.

[856/863] cmd.exe /C "cd . && C:\src\vcpkg\downloads\tools\cmake-3.25.1-windows\cmake-3.25.1-windows-i386\bin\cmake.exe -E vs_link_dll --intdir=gazebo\gui\CMakeFiles\gazebo_gui.dir --rc=C:\PROGRA~2\WI3CF2~1\10\bin\100190~1.0\x64\rc.exe --mt=C:\PROGRA~2\WI3CF2~1\10\bin\100190~1.0\x64\mt.exe --manifests  -- C:\PROGRA~1\MIB055~1\2022\COMMUN~1\VC\Tools\MSVC\1435~1.322\bin\Hostx64\x64\link.exe  @CMakeFiles\gazebo_gui.rsp  /out:gazebo\gui\gazebo_gui.dll /implib:gazebo\gui\gazebo_gui.lib /pdb:gazebo\gui\gazebo_gui.pdb /dll /version:11.12 /machine:x64 /nologo /DEBUG /INCREMENTAL:NO /OPT:REF /OPT:ICF  && cd ."
FAILED: gazebo/gui/gazebo_gui.dll gazebo/gui/gazebo_gui.lib 
cmd.exe /C "cd . && C:\src\vcpkg\downloads\tools\cmake-3.25.1-windows\cmake-3.25.1-windows-i386\bin\cmake.exe -E vs_link_dll --intdir=gazebo\gui\CMakeFiles\gazebo_gui.dir --rc=C:\PROGRA~2\WI3CF2~1\10\bin\100190~1.0\x64\rc.exe --mt=C:\PROGRA~2\WI3CF2~1\10\bin\100190~1.0\x64\mt.exe --manifests  -- C:\PROGRA~1\MIB055~1\2022\COMMUN~1\VC\Tools\MSVC\1435~1.322\bin\Hostx64\x64\link.exe  @CMakeFiles\gazebo_gui.rsp  /out:gazebo\gui\gazebo_gui.dll /implib:gazebo\gui\gazebo_gui.lib /pdb:gazebo\gui\gazebo_gui.pdb /dll /version:11.12 /machine:x64 /nologo /DEBUG /INCREMENTAL:NO /OPT:REF /OPT:ICF  && cd ."
LINK: command "C:\PROGRA~1\MIB055~1\2022\COMMUN~1\VC\Tools\MSVC\1435~1.322\bin\Hostx64\x64\link.exe @CMakeFiles\gazebo_gui.rsp /out:gazebo\gui\gazebo_gui.dll /implib:gazebo\gui\gazebo_gui.lib /pdb:gazebo\gui\gazebo_gui.pdb /dll /version:11.12 /machine:x64 /nologo /DEBUG /INCREMENTAL:NO /OPT:REF /OPT:ICF /MANIFEST /MANIFESTFILE:gazebo\gui\gazebo_gui.dll.manifest" failed (exit code 1120) with the following output:
   Creating library gazebo\gui\gazebo_gui.lib and object gazebo\gui\gazebo_gui.exp
PlotCurve.cc.obj : error LNK2019: unresolved external symbol "__declspec(dllimport) public: class QVector<class QPointF> const __cdecl QwtArraySeriesData<class QPointF>::samples(void)const " (__imp_?samples@?$QwtArraySeriesData@VQPointF@@@@QEBA?BV?$QVector@VQPointF@@@@XZ) referenced in function "public: class ignition::math::v6::Vector2<double> __cdecl gazebo::gui::PlotCurve::Point(unsigned int)const " (?Point@PlotCurve@gui@gazebo@@QEBA?AV?$Vector2@N@v6@math@ignition@@I@Z)

QGVScene.cpp.obj : error LNK2001: unresolved external symbol AgMemDisc

QGVScene.cpp.obj : error LNK2001: unresolved external symbol AgIdDisc

QGVScene.cpp.obj : error LNK2001: unresolved external symbol AgIoDisc

QGVScene.cpp.obj : error LNK2001: unresolved external symbol Agdirected

gazebo\gui\gazebo_gui.dll : fatal error LNK1120: 5 unresolved externals

ninja: build stopped: subcommand failed.

@talregev
Copy link
Contributor Author

talregev commented Mar 26, 2023

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not possible in the vcpkg registry.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For now is workaround just to let me compile it. We can discuss it when I successfully compile.
Do you understand the linkage problem?

Comment on lines 141 to 156
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not possible in the vcpkg registry.

@talregev
Copy link
Contributor Author

For linux there more linkage error.
Do I need to add sqlite3 for linux?
gazebo-build-log.txt

Help appreciate.

@talregev
Copy link
Contributor Author

talregev commented Mar 26, 2023

On windows there isn't qwt libs flags when linkage, that led me thinking to go to the direction that the qwt lib is not set correctly in the cmake file.
Any suggestion how to do it correctly?
qwtConfig.cmake is not found at share. I am not sure how to set it.
Do you know how to set it correctly?

install-x64-windows-release-rel-out.log

@talregev
Copy link
Contributor Author

I made a progress in windows compilation. I compile against older qwt 6.1.5 and the linkage of qwt disappear.
Now I left with 4 unresolve symbol, that I search on google and it seems from java. Can someone give me an hint?

[856/863] cmd.exe /C "cd . && C:\src\vcpkg\downloads\tools\cmake-3.25.1-windows\cmake-3.25.1-windows-i386\bin\cmake.exe -E vs_link_dll --intdir=gazebo\gui\CMakeFiles\gazebo_gui.dir --rc=C:\PROGRA~2\WI3CF2~1\10\bin\100190~1.0\x64\rc.exe --mt=C:\PROGRA~2\WI3CF2~1\10\bin\100190~1.0\x64\mt.exe --manifests  -- C:\PROGRA~1\MIB055~1\2022\COMMUN~1\VC\Tools\MSVC\1435~1.322\bin\Hostx64\x64\link.exe  @CMakeFiles\gazebo_gui.rsp  /out:gazebo\gui\gazebo_gui.dll /implib:gazebo\gui\gazebo_gui.lib /pdb:gazebo\gui\gazebo_gui.pdb /dll /version:11.12 /machine:x64 /nologo /DEBUG /INCREMENTAL:NO /OPT:REF /OPT:ICF  && cd ."
FAILED: gazebo/gui/gazebo_gui.dll gazebo/gui/gazebo_gui.lib 
cmd.exe /C "cd . && C:\src\vcpkg\downloads\tools\cmake-3.25.1-windows\cmake-3.25.1-windows-i386\bin\cmake.exe -E vs_link_dll --intdir=gazebo\gui\CMakeFiles\gazebo_gui.dir --rc=C:\PROGRA~2\WI3CF2~1\10\bin\100190~1.0\x64\rc.exe --mt=C:\PROGRA~2\WI3CF2~1\10\bin\100190~1.0\x64\mt.exe --manifests  -- C:\PROGRA~1\MIB055~1\2022\COMMUN~1\VC\Tools\MSVC\1435~1.322\bin\Hostx64\x64\link.exe  @CMakeFiles\gazebo_gui.rsp  /out:gazebo\gui\gazebo_gui.dll /implib:gazebo\gui\gazebo_gui.lib /pdb:gazebo\gui\gazebo_gui.pdb /dll /version:11.12 /machine:x64 /nologo /DEBUG /INCREMENTAL:NO /OPT:REF /OPT:ICF  && cd ."
LINK: command "C:\PROGRA~1\MIB055~1\2022\COMMUN~1\VC\Tools\MSVC\1435~1.322\bin\Hostx64\x64\link.exe @CMakeFiles\gazebo_gui.rsp /out:gazebo\gui\gazebo_gui.dll /implib:gazebo\gui\gazebo_gui.lib /pdb:gazebo\gui\gazebo_gui.pdb /dll /version:11.12 /machine:x64 /nologo /DEBUG /INCREMENTAL:NO /OPT:REF /OPT:ICF /MANIFEST /MANIFESTFILE:gazebo\gui\gazebo_gui.dll.manifest" failed (exit code 1120) with the following output:
   Creating library gazebo\gui\gazebo_gui.lib and object gazebo\gui\gazebo_gui.exp
QGVScene.cpp.obj : error LNK2001: unresolved external symbol AgMemDisc
QGVScene.cpp.obj : error LNK2001: unresolved external symbol AgIdDisc
QGVScene.cpp.obj : error LNK2001: unresolved external symbol AgIoDisc
QGVScene.cpp.obj : error LNK2001: unresolved external symbol Agdirected
gazebo\gui\gazebo_gui.dll : fatal error LNK1120: 4 unresolved externals

@talregev
Copy link
Contributor Author

I am able to understand that this is a graphviz symbols. I will try to compile with older graphviz port.

@LilyWangLL LilyWangLL added category:port-feature The issue is with a library, which is requesting new capabilities that didn’t exist category:port-update The issue is with a library, which is requesting update new revision labels Mar 27, 2023
@talregev
Copy link
Contributor Author

I was able to compile Gazebo on windows with my fix PR with ogre 1.12.9 and qwt 6.1.5 and graphviz 2.49.1.
Please before your compile MAKE SURE your visual studio is the latest update. It have an atomic error bug in the middle versions.
Also I did this for overcome the bug. Not sure if that help:
For compile on windows I did this for atomic error I had:
https://stackoverflow.com/questions/67732065/why-does-vs2019-pro-have-compile-errors-with-xutility-xmemory-and-atomic-when

For compiling:

git clone https://github.com/talregev/vcpkg -b TalR/fix_gazebo2
cd vcpkg
bootstrap-vcpkg.bat
vcpkg install  --x-manifest-root=ports/gazebo/ --x-install-root=installed --triplet=x64-windows-release --host-triplet=x64-windows-release --clean-after-build
vcpkg install gazebo[tools,plugins] --triplet=x64-windows-release --host-triplet=x64-windows-release

@dg0yt I make this PR compile again for Windows.
Are you interesting helping me to compile it also for Linux in this PR, or we try to it on different PR?
Thank you for your input and your help.

@dg0yt
Copy link
Contributor

dg0yt commented Mar 30, 2023

You must make it work with the dependencies as present in vcpkg now. No builtin-baseline, no overrides.

@talregev
Copy link
Contributor Author

talregev commented Mar 30, 2023

You must make it work with the dependencies as present in vcpkg now. No builtin-baseline, no overrides.

It needed old ogre, qwt and graphviz
gazebo port will not keep up with vcpkg update ports and still need old ports.
Different solution is to revive the 3 olds ports and make them that they will not interfere with the new ports.
I can revive the old ports, and I will need an help how to make them non interfere with the new ones.
Are you interesting to help on that?

Maybe there is other solution I didn't think of.

@dg0yt
Copy link
Contributor

dg0yt commented Mar 30, 2023

Are you interesting to help on that?

No.

@talregev
Copy link
Contributor Author

talregev commented Apr 3, 2023

I also able to compile gazebo in windows x86 similar to x64 that I post before.
One of the solution to fix gazebo port is to revive 3 old ports:

  • ogre v1.12.9
  • qwt v6.1.5
  • graphviz 2.49.1

Without intefere the new port.
Is it acceptable solution from official vcpkg?
@Cheney-W is it interesting in helping revive the old ports?
I make it ready for review for start discussion.

@talregev talregev marked this pull request as ready for review April 3, 2023 05:11
@talregev talregev changed the title WIP [gazebo] Fix gazebo dependencies [Help needed][gazebo] Fix gazebo dependencies Apr 3, 2023
@talregev
Copy link
Contributor Author

talregev commented Apr 3, 2023

Help is needed. I can revive the old ports, but I need help to make them not interfere with the new ports. it not trivial task for me.
If vcpkg official will accept that solution, I will update the first comments the tasks is needed to do.
Any other suggestions are also welcome.

@dg0yt
Copy link
Contributor

dg0yt commented Apr 3, 2023

Reviving three outdated versions? Triple no. I tried to make this clear early enough.

@talregev
Copy link
Contributor Author

talregev commented Apr 3, 2023

Reviving three outdated versions? Triple no. I tried to make this clear early enough.

Are you the official vcpkg answer?
I refuse to call it outdated versions.
As other ports needed to older versions of the libraries, For example:

  • qt5
  • gtk3
  • opencv2
  • and more.

As gazebo is large project, you cannot expect it to be up to date with all the libraries it support.
For example qwt.
Current version qwt in vcpkg is: 6.2.0, it compile against qt6.
It mean to update gazebo to compile qwt v6.2.0 in vcpkg, it mean to compile it against qt6.
This is not reasonable request to update gazebo to supprot qt6. That why you still have qt5 in vcpkg.

There always be a ports that needed older versions and cannot update right away and keep up in vcpkg updating system.
That why is needed that some ports needed older versions of the libraries. The solution for that problem that I found
that vcpkg already using is to have multiple versions of the same libraries, that why I suggested this solution again.
I am waiting for official vcpkg respond to my suggestion.
Also you are welcome to offer another solution other then suggest update gazebo to the latest libraries including qt6.

Thank you for your feedback.

"description": "Open source robotics simulator.",
"homepage": "http://gazebosim.org",
"license": "Apache-2.0",
"builtin-baseline": "19af97cba8ca48474e4ad15a24ed50271a9ecdac",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should not be added in to vcpkg.json.

"description": "Open source robotics simulator.",
"homepage": "http://gazebosim.org",
"license": "Apache-2.0",
"builtin-baseline": "19af97cba8ca48474e4ad15a24ed50271a9ecdac",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
"builtin-baseline": "19af97cba8ca48474e4ad15a24ed50271a9ecdac",

@talregev
Copy link
Contributor Author

talregev commented Apr 7, 2023

@LilyWangLL
these lines are temporary solution just to see if it works. I am not intend to merge it like this to the master.
I am suggesting other solution.

"builtin-baseline": "19af97cba8ca48474e4ad15a24ed50271a9ecdac",
...
  "overrides": [
    {
      "name": "ogre",
      "version": "1.12.9",
      "port-version": 10
    },
    {
      "name": "qwt",
      "version": "6.1.5"
    },
    {
      "name": "graphviz",
      "version": "2.49.1",
      "port-version": 4
    }
  ]

Comment on lines +141 to +156
"overrides": [
{
"name": "ogre",
"version": "1.12.9",
"port-version": 10
},
{
"name": "qwt",
"version": "6.1.5"
},
{
"name": "graphviz",
"version": "2.49.1",
"port-version": 4
}
}
]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not possible in the vcpkg registry.

These should not be added in to vcpkg.json.

@talregev
Copy link
Contributor Author

talregev commented Apr 7, 2023

Help is needed. I can revive the old ports, but I need help to make them not interfere with the new ports. it not trivial task for me. If vcpkg official will accept that solution, I will update the first comments the tasks is needed to do. Any other suggestions are also welcome.

As gazebo is large project, you cannot expect it to be up to date with all the libraries it support. For example qwt. Current version qwt in vcpkg is: 6.2.0, it compile against qt6. It mean to update gazebo to compile qwt v6.2.0 in vcpkg, it mean to compile it against qt6. This is not reasonable request to update gazebo to supprot qt6. That why you still have qt5 in vcpkg.

There always be a ports that needed older versions and cannot update right away and keep up in vcpkg updating system. That why is needed that some ports needed older versions of the libraries. The solution for that problem that I found that vcpkg already using is to have multiple versions of the same libraries, that why I suggested this solution again. I am waiting for official vcpkg respond to my suggestion. Also you are welcome to offer another solution other then suggest update gazebo to the latest libraries including qt6.

Thank you for your feedback.

@LilyWangLL Can you read my solution and let me know if it acceptable from vcpkg official team?

@talregev
Copy link
Contributor Author

talregev commented Apr 7, 2023

@LilyWangLL I want to revive 3 old ports to be along with the new ports:

  • ogre v1.12.9
  • qwt v6.1.5 / create new port of qwt-qt5
  • graphviz v2.49.1

@LilyWangLL
Copy link
Contributor

Help is needed. I can revive the old ports, but I need help to make them not interfere with the new ports. it not trivial task for me. If vcpkg official will accept that solution, I will update the first comments the tasks is needed to do. Any other suggestions are also welcome.

As gazebo is large project, you cannot expect it to be up to date with all the libraries it support. For example qwt. Current version qwt in vcpkg is: 6.2.0, it compile against qt6. It mean to update gazebo to compile qwt v6.2.0 in vcpkg, it mean to compile it against qt6. This is not reasonable request to update gazebo to supprot qt6. That why you still have qt5 in vcpkg.

There always be a ports that needed older versions and cannot update right away and keep up in vcpkg updating system. That why is needed that some ports needed older versions of the libraries. The solution for that problem that I found that vcpkg already using is to have multiple versions of the same libraries, that why I suggested this solution again. I am waiting for official vcpkg respond to my suggestion. Also you are welcome to offer another solution other then suggest update gazebo to the latest libraries including qt6.

Thank you for your feedback.

cc @vicroms @BillyONeal This PR need your help. Thanks in advance.

@BillyONeal
Copy link
Member

As gazebo is large project, you cannot expect it to be up to date with all the libraries it support. For example qwt. Current version qwt in vcpkg is: 6.2.0, it compile against qt6. It mean to update gazebo to compile qwt v6.2.0 in vcpkg, it mean to compile it against qt6. This is not reasonable request to update gazebo to supprot qt6.

It doesn't matter how large a project is. No individual project gets to tell the entire ecosystem they must be stuck on any particular version of anything. If something can't use the dependency versions that are currently what you get in the curated registry, our only options are to (1) attempt to patch that project to use what is available, or (2) to delist the project with the incompatible dependency.

This rule is not imposed by vcpkg, it is imposed by C++: in a given program there can only be one owner of a given symbol, so we have to pick one.

That why you still have qt5 in vcpkg.

We have qt5 in vcpkg because the qt project have undertaken the immense effort to make every symbol they publish have the qt major version number be present in the symbol, so a single program can link with both qt5 and qt6 and still be functional. That doesn't mean that everything that transitively wants qt though makes that effort.

The solution for that problem that I found that vcpkg already using is to have multiple versions of the same libraries, that why I suggested this solution again.

In general, multiple versions of the same library aren't link compatible with each other, so we can't do that. We can do it only when the multiple versions are effectively different libraries entirely that don't use any of the same symbol names across version, which is not usual. qt has done that, but I don't know of another project that has.

Even for qt you can't use 5.0 side by side with 5.1, for instance. Only major versions get the side by side treatment.

@talregev
Copy link
Contributor Author

talregev commented Apr 7, 2023

@BillyONeal What is your solution for fixing gazebo port?
I only asking for major version revive ports.

@BillyONeal
Copy link
Member

What is your solution for fixing gazebo port?

It can either be patched to use current dependencies, or it must be delisted.

I only asking for major version revive ports.

Again, we can only do that if those are usable side by side, which is almost never. This is not a vcpkg limitation, this is a C++ limitation.

@talregev
Copy link
Contributor Author

talregev commented Apr 8, 2023

@BillyONeal Thank you for the patience and the detailed explanations.
I am closing this PR for lack to update gazebo to the latest dependencies.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

category:port-feature The issue is with a library, which is requesting new capabilities that didn’t exist category:port-update The issue is with a library, which is requesting update new revision

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants