Merge pull request #397 from AlexandreG87/feature/python_module

[WIP] feature/python_module : Adding libpointmatcher's Python bindings (#222)

CMakeLists.txt

@@ -1,4 +1,4 @@
-cmake_minimum_required(VERSION 2.8.11)
+cmake_minimum_required(VERSION 2.8.12)
 
 include(CheckSymbolExists)
 include(GNUInstallDirs) # populate CMAKE_INSTALL_{LIB,BIN}DIR
@@ -467,6 +467,15 @@ if (BUILD_TESTS)
 	add_subdirectory(utest)
 endif()
 
+# ========================= Python module =============================
+option(BUILD_PYTHON_MODULE "Build the python module for libpointmatcher" OFF)
+option(USE_SYSTEM_PYBIND11 "Use the system installed pybind11 rather than using a git submodule" ON)
+set(PYTHON_INSTALL_TARGET "" CACHE PATH "Target where to install the python module")
+
+if (BUILD_PYTHON_MODULE)
+	add_subdirectory(python)
+endif()
+
 #=================== allow find_package() =========================
 #
 # the following case be used in an external project requiring libpointmatcher:

doc/CompilationPython.md

@@ -0,0 +1,179 @@
+| [Tutorials Home](index.md) | [Previous](UnitTestDev.md) | [Next](PythonModule.md) |
+| :--- | :---: | ---: |
+
+# Compiling libpointmatcher with Python
+
+This tutorial presents the different steps of compiling *pypointmatcher*, the libpointmatcher's Python module, on Ubuntu and Mac OS X.
+
+## Prerequisites
+
+To get started, you will need the same prerequisites as libpointmatcher, but also some additional dependencies as listed here:
+
+| Name | Version <br> (Tested August 2020 on Ubuntu 18.04) |
+| :--- | :---: |
+| pybind11 | 2.5.0 |
+| Python3 | 3.6.9 |
+|||
+| **Dependencies** | |
+| python3-dev | 3.6.7 |
+| catch | 1.10.0 |
+| pytest | 5.4.3 |
+
+`pytest` needs to be installed with `pip`:
+
+```bash
+pip3 install pytest
+```
+
+But `catch` and `python3-dev` need to be installed with a package manager:
+
+*Ubuntu users:*
+
+```bash
+sudo apt install catch python3-dev
+```
+
+*Mac OS users*:
+
+```bash
+brew install catch2
+```
+
+The rest of this tutorial will guide you through the necessary steps to compile pypointmatcher.
+
+## pybind11
+
+pybind11 is a library used to create Python bindings of existing C++ code and vice versa. So, in order to be able to compile pypointmatcher, you must either install pybind11 on your system or add it as a git submodule in the libpointmatcher's `contrib/` directory. You must then create a symbolic link to this git submodule in the `python/` directory. Go [here](#installing-pybind11) for the installation steps or [here](#adding-pybind11) for the git sudmodule steps.
+
+### Installing pybind11 (recommended) <a name="installing-pybind11"></a>
+
+The very first step is to clone [pybind11](https://github.com/pybind/pybind11) into a directory of your choice.
+
+At the moment, pypointmatcher can only be compiled with **version 2.5.0** of pybind11. To install the right version, run the following commands:
+
+```bash
+cd pybind11
+git checkout v2.5.0
+```
+
+Once this is done, run the following commands:
+
+```bash
+mkdir build && cd build
+cmake ..
+make check -j 4
+```
+
+This will both compile and run pybind11 tests. Next, you can install pybind11 by running this command:
+
+```bash
+sudo make install
+```
+
+Once this is done, return to libpointmatcher's `build/` directory.
+
+You're now ready to proceed to the [configuration step](#configuration).
+
+### Adding pybind11 as a `git` submodule <a name="adding-pybind11"></a>
+
+An alternative to installing pybind11 on your system is to add its repository as a git submodule and create a symbolic link into the `python/` directory. To do this, you will first need to clone the repository as a git submodule by running the following commands in your terminal from the `contrib/` directory.
+
+```bash
+cd contrib
+git submodule add https://github.com/pybind/pybind11.git
+```
+
+This will add pybind11 as a git submodule of libpointmatcher into the `contrib/` directory. Then, still from the `contrib/` directory, run this command to create a symbolic link to pybind11 in the `python/` directory:
+
+```bash
+ln -sr pybind11 ../python/pybind11
+```
+
+At the moment, pypointmatcher can only be compiled with **version 2.5.0** of pybind11. To install the right version, run the following commands:
+
+```bash
+cd pybind11
+git checkout v2.5.0
+```
+
+Finally, tell CMake that you want to use pybind11 as a git submodule by setting the `USE_SYSTEM_PYBIND11` variable to `OFF`:
+
+```bash
+cmake -D USE_SYSTEM_PYBIND11=OFF ..
+```
+
+> ***IMPORTANT:*** When this method is used, it is very important to checkout the version **2.5.0** of pybind11 or it will be impossible to generate the build files.
+
+Once this is done, return to libpointmatcher's `build/` directory.
+
+You're now ready to proceed to the [configuration step](#configuration).
+
+## Configuring the variables <a name="configuration"></a>
+
+> ***Note:*** *It is recommended to create a virtual environment before proceeding with the next steps. For this, you can use the [virtualenv tool](https://virtualenv.pypa.io/en/stable/). If you are not familiar with Python virtual environments, you can [read this tutorial](https://realpython.com/python-virtual-environments-a-primer/), which explains very well the reasons for using a virtual environment, or [watch this video tutorial](https://youtu.be/nnhjvHYRsmM)*
+
+#### Specifying the path
+
+First, you need to specify where you want the module to be installed. To do so, you must provide the path by setting the CMake variable `PYTHON_INSTALL_TARGET` with an absolute path to your Python environment `site-packages` location. This can be achieve manually or automatically.
+
+##### The manual way:
+
+Launch the Python interpreter and run the following commands to find the path to the `site-packages/` directory of your current Python environment:
+
+```bash
+>>> import site
+>>> site.getsitepackages()
+```
+
+> ***Note:*** If you are using the system's Python environment, replace the `getsitepackages()` function call by `getusersitepackages()`.
+
+This will output a list of installation paths for your current Python environment. Now, choose the one that is located in the `python_env_path/lib/python3.x/` directory. The command to run should look like this:
+
+```bash
+cmake -D PYTHON_INSTALL_TARGET=python_env_path/lib/python3.x/site-packages ..
+```
+
+> ***NOTE:*** Replace the `x` with your Python minor version number.
+
+##### The automatic way:
+
+If you don't want to set the path manually, here's a command that should automatically pick the right one for you:
+
+```bash
+cmake -D PYTHON_INSTALL_TARGET=$(python3 -c "import site; print(site.getsitepackages()[0])") ..
+```
+
+> ***Note:*** If you are using the system's Python environment, replace the `site.getsitepackages()[0]` by `site.getusersitepackages()`.
+
+> ***IMPORTANT:*** *This last example is the default behavior if no path has been set before compiling the module.* ***Please, make sure that this corresponds to a valid location or the module will be installed in a wrong location and this will lead to an import error.***
+
+#### Enabling the compilation
+
+By default, pypointmatcher compilation is disabled. In order to compile it, you must set the CMake variable `BUILD_PYTHON_MODULE` to `ON`:
+
+```bash
+cmake -D BUILD_PYTHON_MODULE=ON ..
+```
+
+Everything is now set up to proceed to the compilation and the installation.
+
+## Compilation
+
+Now, to compile pypointmatcher into the `build/` directory, run the following command:
+
+```bash
+make pypointmatcher -j N
+```
+
+where `N` is the number of jobs (or threads) you allow at once on your computer for the compilation. If no argument is passed after `-j`, there will be no limit to the number of jobs.
+
+> ***Note:*** *Depending on your system, the compilation can take quite some time, so consider leaving the `-j` command with no argument in order to speed up this step.*
+
+## Installation
+
+And finally, to install the module on your system, run the following command:
+
+```bash
+sudo make install
+```
+

doc/PythonModule.md

@@ -0,0 +1,67 @@
+| [Tutorials Home](index.md) | [Previous](CompilationPython.md) |
+| :--- | ---: |
+
+# Using libpointmatcher with Python
+
+This tutorial presents the different things to know before using *pypointmatcher*, the libpointmatcher's Python module.
+
+## Differences between the C++ and Python APIs
+
+Despite the fact that pypointmatcher and libpointmatcher have very similar APIs, the fact remains that they differ in some ways. So, why not start by listing these differences.
+
+#### STL containers vs Python data structures
+
+pybind11 provides automatic conversion between C++'s STL containers and their Python equivalent data structures. That is, `std::vector`/`std::deque`/`std::list`/`std::array` are converted into a Python `list`, `std::set`/`std::unordered_set` are converted into a Python `set` and finally `std::map`/`std::unordered_map` are converted into a Python `dict`. 
+
+Although this can be very useful, it comes with some major downsides, [which you can see here](https://pybind11.readthedocs.io/en/latest/advanced/cast/stl.html#automatic-conversion), hence the need to make the classes inheriting from STL containers and some `typedef` *"opaque"*. For the moment, only the `std::vector` and the `std::map` containers are made *"opaque"*, i.e. they keep the same name as the one used in libpointmatcher, but they adopt the behavior of a `list` and a `dict` respectively.
+
+> ***Note:*** This also means that these opaque STL containers must be used in constructors and methods that accept one of these types as parameter.
+ 
+For more information about pybind11 STL containers conversion, visit [this section](https://pybind11.readthedocs.io/en/latest/advanced/cast/stl.html) of the official documentation.
+
+#### Eigen vs Numpy
+
+pybind11 also provides transparent conversion between *Eigen*'s `Matrix`, `Map` and `SparseMatrix` and *numpy*'s `array` and `ndarray` data types. That is, you can seamlessly use *numpy*'s `ndarray` instead of *Eigen*'s `Vector` or `Matrix`.
+
+For more information about pybind11 *Eigen* to *numpy* data type conversion, visit [this section](https://pybind11.readthedocs.io/en/latest/advanced/cast/eigen.html) of the official documentation.
+
+#### Overloaded methods based on constness
+
+In libpointmatcher, more precisely in the `DataPoints` class, some methods are overloaded based on constness, i.e. they will be called with a constant `DataPoints`. So, to avoid ambiguous calls, the suffix `_const` has been appended to the method names. E.g. in the `compute_overlap.py` example, the `getDescriptorViewByName("inliers")` method was calling the `const` version before this fix. For more information on pybind11 overloaded method mechanisms, visit [this section](https://pybind11.readthedocs.io/en/latest/classes.html#overloaded-methods) of the official documentation.
+
+#### Contructors/methods with std::istream or std::ostream as paramater
+
+Some constructors and methods of libpointmatcher have as parameter either an `std::istream` to build an object from a YAML configuration file or an `std::ostream` to dump information. pybind11 doesn't allow to call these constructors/methods with their Python equivalent, i.e. `sys.stdin` and `sys.stdout`. So, to get around this problem, the constructors/methods having a `std::istream` as parameter must be used with a `std::string` instead and those having a `std::ostream` must be used without parameter.
+
+## Structure of pypointmatcher
+
+Before going further, here is the general structure of pypointmatcher to give you a better idea of how to use it.
+
+```textmate
+pypointmatcher # The main module.
+|
+|_ pointmatcher # The submodule containing functions and classes that are dependent on scalar types.
+|
+|_ pointmatchersupport # The submodule containing everything defined in the
+|                       # pointmatchersupport namespace, i.e. functions and classes which are not dependent on scalar types.
+|
+|_ datapointsfilters # The submodule containing the differents DataPointsFilters.
+|
+|_ errorminimizers # The submodule containing the differents ErrorMinimizers.
+```
+
+## General use of pypointmatcher
+
+To help you get familiar with pypointmatcher, some C++ examples have been translated to show you how it is possible to use the Python version of libpointmatcher.
+
+Now that you know the major differences between the C++ and Python API, we suggest newcomers and those who are less familiar with the library to follow, or re-follow, the tutorials in the beginner and advanced sections using the Python examples instead, in order to have a better understanding of how libpointmatcher can be used with Python.
+
+Experienced users of libpointmatcher can, if they wish, take a look at the different Python examples located in the `examples/python/` directory and compare their uses with their C++ counterparts, keeping in mind what makes them different from each other.
+
+The major difference is that they don't require command line arguments like the C++ version. All you have to do is open a terminal and go to the `examples/python/` directory and run one of the examples as a Python module script:
+
+```bash
+python3 icp_simple.py
+```
+
+> ***Note:*** All the information to run a default ICP algorithm is already in the examples, it is up to the user to change these values in order to run a custom ICP algorithm. So take the time to understand what the code does in the examples before making changes.

doc/UnitTestDev.md

@@ -1,5 +1,5 @@
-| [Tutorials Home](index.md) | [Previous](TransformationDev.md) |
-| :--- | ---: |
+| [Tutorials Home](index.md) | [Previous](TransformationDev.md) | [Next](CompilationPython.md) |
+| :--- | :---: | ---: |
 
 # Testing libpointmatcher Modules
 

doc/index.md

@@ -4,15 +4,15 @@
 
 # Tutorials
 
-This page lists the available tutorials for libpointmatcher. The [Beginner Section](#beginner) is aimed at the more casual user and contains high-level information on the various steps of point cloud registration. The [Advanced Section](#advanced) is targeted at those with existing experience with point cloud registration and proficiency in C++ development.  Those who wish to contribute to libpointmatcher can follow the guidelines in the [Developer](#developer) section.
+This page lists the available tutorials for libpointmatcher. The [Beginner](#beginner) section is aimed at the more casual user and contains high-level information on the various steps of point cloud registration. The [Advanced](#advanced) section is targeted at those with existing experience with point cloud registration and proficiency in C++ development.  Those who wish to contribute to libpointmatcher can follow the guidelines in the [Developer](#developer) section. Finally, the [Python](#python) section is intended for those who are interested in using libpointmatcher with *Python*.
 
-## Compilation<a name="compilation"></a>
+## Compilation <a name="compilation"></a>
 
 - [Ubuntu: How to compile libpointmatcher](CompilationUbuntu.md)
 - [Mac OS X: How to compile libpointmatcher](CompilationMac.md)
 - [Windows: How to compile libpointmatcher](CompilationWindows.md)
 
-## Beginner<a name="beginner"></a>
+## Beginner <a name="beginner"></a>
 
 - [What is libpointmatcher about?](Introduction.md)
 - [What can I do with libpointmatcher?](ApplicationsAndPub.md)
@@ -24,7 +24,7 @@ This page lists the available tutorials for libpointmatcher. The [Beginner Secti
 - [Configuring libpointmatcher using YAML](Configuration.md)
 - [Supported file types and importing/exporting point clouds](ImportExport.md)
 
-## Advanced<a name="advanced"></a>
+## Advanced <a name="advanced"></a>
 
 - [How to link a project to libpointmatcher?](LinkingProjects.md)
 - [How to use libpointmatcher in ROS?](UsingInRos.md)
@@ -39,12 +39,17 @@ This page lists the available tutorials for libpointmatcher. The [Beginner Secti
 - How to do a nearest neighbor search between two point clouds without an ICP object? [See the comments here.](https://github.com/ethz-asl/libpointmatcher/issues/193#issuecomment-276093785)
 - How to construct a `DataPoints` from my own point cloud? [See the unit test on `Datapoints` here.](https://github.com/ethz-asl/libpointmatcher/blob/master/utest/ui/DataFilters.cpp#L52)
 
-## Developer<a name="developer"></a>
+## Developer <a name="developer"></a>
 
 - [Creating a DataPointsFilter](DataPointsFilterDev.md)
 - [Creating a Transformation](TransformationDev.md)
 - [Creating unit tests](UnitTestDev.md)
 
+## Python <a name="python"></a>
+
+- [Compiling libpointmatcher with Python](CompilationPython.md)
+- [Using libpointmatcher with Python](PythonModule.md)
+
 **Note**: if you don't find what you need, don't hesitate to propose or participate to new tutorials. 
 
 ---