Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Windows (MSVC) support #76

Open
bchretien opened this issue Oct 7, 2014 · 30 comments
Open

Add Windows (MSVC) support #76

bchretien opened this issue Oct 7, 2014 · 30 comments

Comments

@bchretien
Copy link
Member

Some people are interested in using RobOptim on Windows, so this is worth investigating.

@kayusawa could you make a summary of your tests regarding the compilation of RobOptim on Windows? Thanks! :-)

@francois-keith
Copy link
Member

👍
Some details on the installation of the dependencies would also be appreciated.

@bchretien
Copy link
Member Author

Indeed, we could then make a detailed installation process on the website.

@kayusawa
Copy link
Contributor

kayusawa commented Oct 8, 2014

Dear Dr.Crétien

Thank you for the e-mail.
I could compile RobOpitim on Windows by using VisualStudio 2012 and Intel
Compiler few month ago.
For the compilation, I prepared several windows tools and libraries as
follows:

  • CMake (2.8.12)
  • pkg_config
  • doxgen
  • eigen (3.2.2)
  • boost (1.55)
  • log4cxx (0.10.0)
  • libtool (1.5.26)

Among them, I compiled boost, log4cxx, and libtool myself.
I'm sorry that I don't really remember the details, but a few minor
modifications are required for the compilation of log4cxx and libtool.

After preparing the libraries, the file encoding of RobOptim codes has to
be changed from "utf8-without BOM" to "utf8-with BOM".
(It enables MSVC/Intel to read utf8 characters.)

If Intel Compiler is available, the project files generated by CMake will
work properly without modification.
However, if you plan to use VisualStudio, there may be a few conflicts
about MSVC definitions.
For example, "boost::mpl::size_t" and "unsigned int size_t" will be
ambiguous when calling Eigen sparse matrix.
Actually there are few errors in my case, so I modified Eigen codes
directly.

If you have any question, please contact me.

Best regards,
Ko Ayusawa


Ko AYUSAWA, researcher
CNRS-AIST JRL(Joint Robotics Laboratory), UMI3218/CRT,
Intelligent Systems Research Institute,
National Institute of Advanced Industrial Science and Technology(AIST)
Tsukuba Central 2, 1-1-1 Umezono, Tsukuba, Ibaraki 305-8568 JAPAN
TEL: +81-29-861-3576 FAX: +81-29-862-6507

e-mail: [email protected]

2014-10-07 20:20 GMT+09:00 Benjamin Chrétien [email protected]:

Some people are interested in using RobOptim on Windows, so this is worth
investigating.

@kayusawa https://github.com/kayusawa could you make a summary of your
tests regarding the compilation of RobOptim on Windows? Thanks! :-)


Reply to this email directly or view it on GitHub
#76.

@kayusawa
Copy link
Contributor

kayusawa commented Oct 8, 2014

Dear Dr. Chrétien

I'm so sorry that I misspelled your name.

If you want to need the pre-compiled binaries or modified codes of the
third-party libraries, I can prepare and send you.

Best regards,

Ko AYUSAWA, researcher
CNRS-AIST JRL(Joint Robotics Laboratory), UMI3218/CRT,
Intelligent Systems Research Institute,
National Institute of Advanced Industrial Science and Technology(AIST)
Tsukuba Central 2, 1-1-1 Umezono, Tsukuba, Ibaraki 305-8568 JAPAN
TEL: +81-29-861-3576 FAX: +81-29-862-6507

e-mail: [email protected]

2014-10-08 19:35 GMT+09:00 Ko AYUSAWA [email protected]:

Dear Dr.Crétien

Thank you for the e-mail.
I could compile RobOpitim on Windows by using VisualStudio 2012 and Intel
Compiler few month ago.
For the compilation, I prepared several windows tools and libraries as
follows:

  • CMake (2.8.12)
  • pkg_config
  • doxgen
  • eigen (3.2.2)
  • boost (1.55)
  • log4cxx (0.10.0)
  • libtool (1.5.26)

Among them, I compiled boost, log4cxx, and libtool myself.
I'm sorry that I don't really remember the details, but a few minor
modifications are required for the compilation of log4cxx and libtool.

After preparing the libraries, the file encoding of RobOptim codes has to
be changed from "utf8-without BOM" to "utf8-with BOM".
(It enables MSVC/Intel to read utf8 characters.)

If Intel Compiler is available, the project files generated by CMake will
work properly without modification.
However, if you plan to use VisualStudio, there may be a few conflicts
about MSVC definitions.
For example, "boost::mpl::size_t" and "unsigned int size_t" will be
ambiguous when calling Eigen sparse matrix.
Actually there are few errors in my case, so I modified Eigen codes
directly.

If you have any question, please contact me.

Best regards,
Ko Ayusawa


Ko AYUSAWA, researcher
CNRS-AIST JRL(Joint Robotics Laboratory), UMI3218/CRT,
Intelligent Systems Research Institute,
National Institute of Advanced Industrial Science and Technology(AIST)
Tsukuba Central 2, 1-1-1 Umezono, Tsukuba, Ibaraki 305-8568 JAPAN
TEL: +81-29-861-3576 FAX: +81-29-862-6507

e-mail: [email protected]

2014-10-07 20:20 GMT+09:00 Benjamin Chrétien [email protected]:

Some people are interested in using RobOptim on Windows, so this is worth
investigating.

@kayusawa https://github.com/kayusawa could you make a summary of your
tests regarding the compilation of RobOptim on Windows? Thanks! :-)


Reply to this email directly or view it on GitHub
#76.

@francois-keith
Copy link
Member

@kayusawa Hi, I would be interested in testing roboptim on windows. Could
you please send me (or put on a server) those pre-compiled libraries?

Thanks.
Le 8 oct. 2014 12:43, "kayusawa" [email protected] a écrit :

Dear Dr. Chrétien

I'm so sorry that I misspelled your name.

If you want to need the pre-compiled binaries or modified codes of the
third-party libraries, I can prepare and send you.

Best regards,

Ko AYUSAWA, researcher
CNRS-AIST JRL(Joint Robotics Laboratory), UMI3218/CRT,
Intelligent Systems Research Institute,
National Institute of Advanced Industrial Science and Technology(AIST)
Tsukuba Central 2, 1-1-1 Umezono, Tsukuba, Ibaraki 305-8568 JAPAN
TEL: +81-29-861-3576 FAX: +81-29-862-6507

e-mail: [email protected]

2014-10-08 19:35 GMT+09:00 Ko AYUSAWA [email protected]:

Dear Dr.Crétien

Thank you for the e-mail.
I could compile RobOpitim on Windows by using VisualStudio 2012 and
Intel
Compiler few month ago.
For the compilation, I prepared several windows tools and libraries as
follows:

  • CMake (2.8.12)
  • pkg_config
  • doxgen
  • eigen (3.2.2)
  • boost (1.55)
  • log4cxx (0.10.0)
  • libtool (1.5.26)

Among them, I compiled boost, log4cxx, and libtool myself.
I'm sorry that I don't really remember the details, but a few minor
modifications are required for the compilation of log4cxx and libtool.

After preparing the libraries, the file encoding of RobOptim codes has
to
be changed from "utf8-without BOM" to "utf8-with BOM".
(It enables MSVC/Intel to read utf8 characters.)

If Intel Compiler is available, the project files generated by CMake
will
work properly without modification.
However, if you plan to use VisualStudio, there may be a few conflicts
about MSVC definitions.
For example, "boost::mpl::size_t" and "unsigned int size_t" will be
ambiguous when calling Eigen sparse matrix.
Actually there are few errors in my case, so I modified Eigen codes
directly.

If you have any question, please contact me.

Best regards,
Ko Ayusawa


Ko AYUSAWA, researcher
CNRS-AIST JRL(Joint Robotics Laboratory), UMI3218/CRT,
Intelligent Systems Research Institute,
National Institute of Advanced Industrial Science and Technology(AIST)
Tsukuba Central 2, 1-1-1 Umezono, Tsukuba, Ibaraki 305-8568 JAPAN
TEL: +81-29-861-3576 FAX: +81-29-862-6507

e-mail: [email protected]

2014-10-07 20:20 GMT+09:00 Benjamin Chrétien [email protected]:

Some people are interested in using RobOptim on Windows, so this is
worth
investigating.

@kayusawa https://github.com/kayusawa could you make a summary of
your
tests regarding the compilation of RobOptim on Windows? Thanks! :-)


Reply to this email directly or view it on GitHub
#76.


Reply to this email directly or view it on GitHub
#76 (comment)
.

@bchretien
Copy link
Member Author

Dear Ayusawa-san,

Thanks for the quick answer. Don't worry about my name, I know it's a difficult one :-)

For UTF-8 with BOM, this may be something that can be solved by CMake when generating the MSVC project files. If the compiler is the issue, a fix similar to this one may be what we need, especially if that means we do not need to add a BOM to all the files that contain UTF-8 characters.

For pre-compiled libraries, these could be released for our next major release. For the dependencies, I guess we could provide them (log4cxx/libtool), especially if some patching is involved.

For the Eigen error, is it something that happened in RobOptim or Eigen itself?

@kayusawa
Copy link
Contributor

kayusawa commented Oct 8, 2014

Dear Chrétien-san, Keith-san

For UTF-8 with BOM, this may be something that can be solved by CMake when

generating the MSVC project files. If the compiler is the issue, a fix similar
to this one
https://github.com/chewing/libchewing/blob/master/CMakeLists.txt#L42 may
be what we need, especially if that means we do not need to add a BOM to
all the files that contain UTF-8 characters.

Thank you for the advice.
I agree that we should solve it by the configuration of CMake.

For pre-compiled libraries, these could be released for our next major

release. For the dependencies, I guess we could provide them (log4cxx/
libtool), especially if some patching is involved.

I attached the pre-compiled log4cxx and libtool (both x86 and x64).
Please rename ".lib.dat" to ".lib" inside "install.zip"
(This is because the security problem of AIST mailing service.)
I'm sorry that the path setting of the package config file of log4cxx has
to be modified manually.

I also attached the modified code of them.
Almost all the modification of log4cxx is just the definition of namespace.
The original windows libtool does not call loadlibrary properly on VC2012,
so I tried to modify and create the windows solution file.
I also attach the simple memo of those installation.

For the Eigen error, is it something that happened in RobOptim or Eigen

itself?

I'm sorry that I almost forgot about where the problem occured.
But I guess "boost/mpl/size_t" and "unsigned int size_t" were ambiguous
when creating Eigen matrix array in Function of RobOptim .
When I used Intel Compiler, there are no such kind of problems.
I guess there is a way to solve it by using MSVC macro to control them.
But at that time, I had to compile RobOptim immediately, so I modified
"size_t" to "::size_t" in several places in Eigen codes and planed to
revert later.

If there are problems, please don't hesitate to contact me.

Best regards,
Ko ayusawa


Ko AYUSAWA, researcher
CNRS-AIST JRL(Joint Robotics Laboratory), UMI3218/CRT,
Intelligent Systems Research Institute,
National Institute of Advanced Industrial Science and Technology(AIST)
Tsukuba Central 2, 1-1-1 Umezono, Tsukuba, Ibaraki 305-8568 JAPAN
TEL: +81-29-861-3576 FAX: +81-29-862-6507

e-mail: [email protected]

2014-10-08 21:40 GMT+09:00 Benjamin Chrétien [email protected]:

Dear Ayusawa-san,

Thanks for the quick answer. Don't worry about my name, I know it's a
difficult one :-)

For UTF-8 with BOM, this may be something that can be solved by CMake when
generating the MSVC project files. If the compiler is the issue, a fix similar
to this one
https://github.com/chewing/libchewing/blob/master/CMakeLists.txt#L42
may be what we need, especially if that means we do not need to add a BOM
to all the files that contain UTF-8 characters.

For pre-compiled libraries, these could be released for our next major
release. For the dependencies, I guess we could provide them (log4cxx/
libtool), especially if some patching is involved.

For the Eigen error, is it something that happened in RobOptim or Eigen
itself?


Reply to this email directly or view it on GitHub
#76 (comment)
.

////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////

  1. open VisualStudio comand prompt (x86)
  2. change directory to boost top directory
  3. type as follows:
    $ bootstrap
    $ b2 --build-dir=_build\x86 --stagedir=stage\x86 address-model=32 link=static,shared runtime-link=shared -j 4
  4. if you have intel compiler...
    $ "c:\Program Files (x86)\Intel\Composer XE\bin\compilervars.bat" ia32 vs2012
    $ b2 --toolset="intel" --build-dir=_build\x86 --stagedir=stage\x86 address-model=32 link=static,shared runtime-link=shared -j 4
1. open VisualStudio comand prompt (x64) 2. change directory to boost top directory 3. type as follows: $ bootstrap $ b2 --build-dir=_build\x64 --stagedir=stage\x64 address-model=64 link=static,shared runtime-link=shared -j 4 4. if you have intel compiler... $ "c:\Program Files (x86)\Intel\Composer XE\bin\compilervars.bat" intel64 vs2012 $ b2 --toolset="intel" --build-dir=_build\x64 --stagedir=stage\x64 address-model=64 link=static,shared runtime-link=shared -j 4

////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
<log4cxx(MSVC2012)>

  1. download as follows:

    • apache-log4cxx-0.10.0.zip
    • apr-1.5.1-win32-src.zip
    • apr-iconv-1.2.1-win32-src-r2.zip
    • apr-util-1.5.3-win32-src.zip

    from
    http://logging.apache.org/log4cxx/
    http://ftp.yz.yamagata-u.ac.jp/pub/network/apache/apr/

  2. extract them as follows:
    log4cxx\appatch-log4cxx-0.10.0
    log4cxx\apr
    log4cxx\apr-util
    log4cxx\apr-iconv

  3. call as follows:

    • configure.bat
    • configure-aprutil.bat
  4. open "log4cxx\apache-log4cxx-0.10.0\projects\log4cxx.dsw"

  5. add "log4cxx\apr-iconv\apriconv.dsp" in the solution file.

  6. replace files as follows:

    • loggingevent.h
    • loggingevent.cpp
    • propertiespatternconverter.cpp
    • xmllayout.cpp
  7. build log4cxx

  8. liblog4cxx.pc has to be modified manually.

////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////

1. open libltdl.sln 2. build libltdl

@bchretien
Copy link
Member Author

Note regarding CI support for Windows: I recently noticed that some open-source projects now use AppVeyor, which could very roughly be seen as Travis for Windows.

@phuicy
Copy link

phuicy commented Mar 1, 2016

In the latest stable there seems alot of issues with a windows build, that aren't dependencies.

@bchretien
Copy link
Member Author

Windows support is not available yet, although we are slowly getting there (cc @aescande).

@phuicy
Copy link

phuicy commented Mar 1, 2016

We have worked out quite few issues, but are still working on a lot of templating issues, e.g.:

40>C:\git\roboptim-core\include\roboptim/core/problem.hxx(118): error C2491: 'roboptim::Problem<T>::function' : definition of dllimport function not allowed

Will keep you informed. Is there an old binary of an older commit?

@bchretien
Copy link
Member Author

Thanks for looking into this. Alas I don't think anyone shared any binary yet.

@francois-keith
Copy link
Member

Just putting my 2 cents here.

I have tried to compile roboptim-core on windows very recently (I must be 1-2 commits late wrt master). Here are the errors I found (as I remember them, so this is not as precise as I'd like it to be).

  • The error you're describing definition of dllimport function not allowed is due to the fact that some methods / classes are now templated (but weren't in the last version tested). The error can be suppressed by removing this macro for those classes.
  • Some errors will also pop up because of the macro ROBOPTIM_CORE_**_DEPRECATED, that must be placed *before the function declaration in windows.
  • Some of the unit tests will not compile in VS2008 because of template issues (I cannot tell you which ones right here).
  • The format of the output is not the same as in unix (e.g., infinity is not written inf, but #INF, and the E notation takes an extra 0: 1e03 versus 1e003), so some unit tests may fail.

Those are some leads.
I will push that on my fork (it is still a work in progress) tonight if you want to have a look.

@phuicy
Copy link

phuicy commented Mar 1, 2016

Awesome, this is exactly what we are seeing.
I think to solve the 1st issue the ROBOPTIM_DLLAPI has to moved to a pre-declaration of the class., as:

// function definition
void __declspec(dllimport) funcB() {}   // C2491

// function declaration
void __declspec(dllimport) funcB();   // OK

number 2:
We set

#define ROBOPTIM_CORE_**_DEPRECATED 

@francois-keith
Copy link
Member

I don't think it is required to keep the ROBOPTIM_DLLAPI macro for the templated class (e.g. Problem) since the definition of the method is in the .hxx and those files are installed as well.
So there is no point AFAIK in exporting the symbols.

I compiled the unit tests as well as roboptim/roboptim-core-plugin-eigen without having any symbol issue.

@bchretien
Copy link
Member Author

Indeed, I'll clean this up a bit. I recently added the possibility to precompile most of the templated classes, since they are now only templated on the matrix traits (dense/sparse). However this is disabled by default since explicit template instantiation is a C++11 feature. I guess the error will appear in some other classes.

bchretien added a commit that referenced this issue Mar 1, 2016
@bchretien
Copy link
Member Author

ROBOPTIM_CORE_DEPRECATED (which is equivalent to __declspec (deprecated) on Windows with MSVC) has to be at the beginning of function declarations/typedefs?

@francois-keith
Copy link
Member

Yes, otherwise some compilation issues may raise.

@bchretien
Copy link
Member Author

Ok, we can easily fix that then. I was also considering making a PR to jrl-cmakemodules to have the deprecated macros support version numbers, e.g.:

#define ROBOPTIM_CORE_DEPRECATED(X) __attribute__ ((deprecated))
...
typedef Foo Bar ROBOPTIM_CORE_DEPRECATED(3.2);

They use this for Gazebo, that way you can see exactly when a given function was deprecated when you get the warning while compiling.

@bchretien
Copy link
Member Author

And talking about Gazebo, they had the exact same problem:

#if defined(__GNUC__)
#define GAZEBO_DEPRECATED(version) __attribute__((deprecated))
#define GAZEBO_FORCEINLINE __attribute__((always_inline))
#elif defined(_WIN32)
// GAZEBO_DEPRECATED should be defined as something like
// __declspec(deprecated), but it needs to go *before* the function name,
// and we're putting GAZEBO_DEPRECATED *after* the function.
#define GAZEBO_DEPRECATED(version)
#define GAZEBO_FORCEINLINE __forceinline
#else
#define GAZEBO_DEPRECATED(version) ()
#define GAZEBO_FORCEINLINE
#endif

@phuicy
Copy link

phuicy commented Mar 1, 2016

Bear in mind, getting Gazebo on windows is still not an easy experience. But this looks good.
That commit (992e734) looks very similar to what we got.

@francois-keith
Copy link
Member

@phuicy I pushed on https://github.com/francois-keith/roboptim-core the current state of my windows version.
I tested it on MSVC 2012, with the following deps:

  • boost 1.55
  • libtools 1.5.26
  • eigen 3.2.0

@bchretien No Pull request for now, mainly because of the test issue and the fact that the result file is different between the Win32 and UNIX systems.

@bchretien
Copy link
Member Author

@francois-keith good catch on the bug recently added to boost.cmake (see jrl-umi3218/jrl-cmakemodules#76). As for the differences when printing, there are some ideas there. The _set_output_format trick could be added to our test fixtures, thus it would be automatically applied to all the RobOptim tests (core, plugins...) by simply changing a single file.

@bchretien
Copy link
Member Author

@francois-keith well apparently _set_output_format is obsolete, still any alternative workaround should probably be applied in our test fixtures.

@bchretien
Copy link
Member Author

Complementary information here:

The removal of _set_output_format is by design. The standards-conforming two-digit exponent mode is now the default and the non-conforming, previously-default, three-digit exponent format is no longer supported. This is noted in the draft Visual C++ 2015 breaking changes documentation: https://msdn.microsoft.com/en-us/library/bb531344(v=vs.140).aspx. We have not yet completed the full updates to the documentation for Visual Studio 2015.

So we can just check the Visual C++ version, and use _set_output_format if an old version is detected.

@phuicy
Copy link

phuicy commented Mar 2, 2016

Awesome. Thank you

@bchretien
Copy link
Member Author

Does anyone know how MSVC behaves with explicit template instantiations and symbol exports? The weird symbol exports I had all around the code solved this problem, i.e. gcc and clang behave differently (either one returns a type attributes ignored after type is already defined error, or the other one does not export the symbols).

@francois-keith
Copy link
Member

AFAIK, you can export the specialization of templated methods or methods of templated class.
I've put an example of the syntax in this repo : francois-keith/test_template_export@34e2e12

To be honest, it was much more tricky in my memory (cf stack-of-tasks/dynamic-graph@5721ccb98), so maybe I'm missing the point / the difficulty here.

@bchretien
Copy link
Member Author

The problem is that gcc and clang behave quite differently, so I was wondering if msvc exhibited the same problem.

  • The following version works (at least the export) in clang but not gcc:
// In the header
template <typename T>
class Problem { ... };
...
extern template class ROBOPTIM_CORE_DLLAPI Problem<EigenMatrixDense>;
extern template class ROBOPTIM_CORE_DLLAPI Problem<EigenMatrixSparse>;

// In the implementation cpp
template class Problem<EigenMatrixDense>;
template class Problem<EigenMatrixSparse>;

With clang and the explicit template instantiation:

$ nm -CD src/libroboptim-core.so | grep "W roboptim::Problem<roboptim::EigenMatrixDense>::initialize()"
00000000000f5170 W roboptim::Problem<roboptim::EigenMatrixDense>::initialize()

Without it, the symbol is not exported as expected.

gcc error:

roboptim-core/include/roboptim/core/problem.hxx:656:46: error: type attributes ignored after type is already defined [-Werror=attributes]
   extern template class ROBOPTIM_CORE_DLLAPI Problem<EigenMatrixSparse>;
                                              ^
  • The following version works with both gcc and clang (at least the export), but will fail with msvc:
// In the header
template <typename T>
class ROBOPTIM_CORE_DLLAPI Problem { ... };
...
extern template class Problem<EigenMatrixDense>;
extern template class Problem<EigenMatrixSparse>;

// In the implementation cpp
template class Problem<EigenMatrixDense>;
template class Problem<EigenMatrixSparse>;

With gcc we finally get:

$ nm -CD src/libroboptim-core.so | grep "W roboptim::Problem<roboptim::EigenMatrixDense>::initialize()"
00000000000f9e10 W roboptim::Problem<roboptim::EigenMatrixDense>::initialize()

EDIT: removed ROBOPTIM_CORE_DLLEXPORT from implementation.

@bchretien
Copy link
Member Author

Experimental Windows support is currently available on the dev branch. The next step would be to validate the changes with a solver plugin.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants