Ask opencv mat numpy array conversion

[edmBernard]

I try to add an Openv Numpy interface with type_caster
I start from @jampekka comments here

I ended to this :

#include <pybind11/numpy.h>
#include <opencv2/core/core.hpp>
#include <stdexcept>

namespace pybind11 { namespace detail {

template <> struct type_caster<cv::Mat> {
    public:
        /**
         * This macro establishes the name 'inty' in
         * function signatures and declares a local variable
         * 'value' of type inty
         */
        PYBIND11_TYPE_CASTER(cv::Mat, _("numpy.ndarray"));

        /**
         * Conversion part 1 (Python->C++): convert a PyObject into a inty
         * instance or return false upon failure. The second argument
         * indicates whether implicit conversions should be applied.
         */
        bool load(handle src, bool) 
        {
            array b(src, true);
            buffer_info info = b.request();
            int ndims = info.ndim;

            decltype(CV_32F) dtype;
            size_t elemsize; 
            if (info.format == format_descriptor<float>::format()) { 
                dtype = CV_32F;
                elemsize = sizeof(float);
            } else if (info.format == format_descriptor<double>::format()) {
                dtype = CV_64F;
                elemsize = sizeof(double);
            } else if (info.format == format_descriptor<unsigned char>::format()) { 
                dtype = CV_8U;
                elemsize = sizeof(unsigned char);
            } else if (info.format == format_descriptor<unsigned char>::format()) { 
                if (ndims == 3) {
                    dtype = CV_8UC3;
                } else {
                    dtype = CV_8U;
                }
                elemsize = sizeof(unsigned char);
            } else { 
                throw std::logic_error("Unsupported type");
                return false;
            }

            std::vector<int> shape = {info.shape[0], info.shape[1]};
            std::vector<size_t> strides = {(size_t) elemsize * info.strides[0], (size_t) elemsize * info.strides[1], (size_t) elemsize * info.strides[2]};

            value = cv::Mat(2,
                    &shape[0],
                    dtype,
                    info.ptr,
                    &strides[0]);
            return true;
        }

        /**
         * Conversion part 2 (C++ -> Python): convert an inty instance into
         * a Python object. The second and third arguments are used to
         * indicate the return value policy and parent object (for
         * ``return_value_policy::reference_internal``) and are generally
         * ignored by implicit casters.
         */
        static handle cast(const cv::Mat &m, return_value_policy, handle defval) 
        {
            std::cout << "m.cols : " << m.cols << std::endl;
            std::cout << "m.rows : " << m.rows << std::endl;
            std::string format = format_descriptor<unsigned char>::format();
            size_t elemsize = sizeof(unsigned char);
            int dim; 
            switch(m.type()) {
                case CV_8U: 
                    format = format_descriptor<unsigned char>::format();
                    elemsize = sizeof(unsigned char);
                    dim = 2;
                    break; 
                case CV_8UC3: 
                    format = format_descriptor<unsigned char>::format();
                    elemsize = sizeof(unsigned char);
                    dim = 3;
                    break; 
                case CV_32F: 
                    format = format_descriptor<float>::format(); 
                    elemsize = sizeof(float);
                    dim = 2;
                    break;
                case CV_64F: 
                    format = format_descriptor<double>::format();
                    elemsize = sizeof(double);
                    dim = 2;
                    break;
                default: 
                    throw std::logic_error("Unsupported type");
            }

            std::vector<size_t> bufferdim;
            std::vector<size_t> strides;
            if (dim == 2) {
                bufferdim = {(size_t) m.rows, (size_t) m.cols};
                strides = {elemsize * (size_t) m.cols, elemsize};
            } else if (dim == 3) {
                bufferdim = {(size_t) m.rows, (size_t) m.cols, (size_t) 3};
                strides = {(size_t) elemsize * m.cols * 3, (size_t) elemsize * 3, (size_t) elemsize};
            }
            return array(buffer_info(
                m.data,         /* Pointer to buffer */
                elemsize,       /* Size of one scalar */
                format,         /* Python struct-style format descriptor */
                dim,            /* Number of dimensions */
                bufferdim,      /* Buffer dimensions */
                strides         /* Strides (in bytes) for each index */
                )).release();
        }

    };
}} // namespace pybind11::detail

The cast part work, i'm able to transfert cv::Mat from C++ to numpy array in python.
but the load part is completely broken there is buffer issues. it seem strides were bad but I don't find how to fixe them.
have you an idea to fixe that ? thx

[aldanor]

{(size_t) elemsize * info.strides[0], (size_t) elemsize * info.strides[1], (size_t) elemsize * info.strides[2]};

What's the logic here? Strides are already expressed in bytes. That's probably the cause of your errors.

Besides, note that a lot of your code is not outdated / deprecated, there's new/better ways to do these things. Also, you may not need to do a .request() at all, see below.

1. array b(src, true);

Direct constructors from (handle, bool), are deprecated; instead:

if (!isinstance<array>(src))
    return false;
auto arr = reinterpret_borrow<array>(src);

2. std::vector<int> shape = {info.shape[0], info.shape[1]};

const size_t* shape = arr.shape();

3. std::vector<size_t> strides = {(size_t) elemsize * info.strides[0], (size_t) elemsize * info.strides[1], (size_t) elemsize * info.strides[2]};

const size_t* strides = arr.strides();

4. int ndims = info.ndim;

int ndim = arr.ndim();

5. info.ptr

arr.mutable_data()  /* or arr.data() if you only need a const pointer */

[edmBernard]

thx for your answer i'll test it soon.

Just to complete my solution finally it work with small modifications (see below).
My issues come from the python side and mutable objects, when i split channel data here not continuous anymore

a = cv2.imread("my_image.png")
my_function(a)    # worked 
b=a[:,:,0]    
my_function(b)    # don't worked
c=copy.deepcopy(b)
my_function(c)    # worked

PS: I'm aware my code use deprecated things but i don't find enough help on the non deprecated solutions.

working code :

#include <pybind11/numpy.h>
#include <opencv2/core/core.hpp>
#include <stdexcept>

namespace pybind11 { namespace detail {

template <> struct type_caster<cv::Mat> {
    public:
        /**
         * This macro establishes the name 'inty' in
         * function signatures and declares a local variable
         * 'value' of type inty
         */
        PYBIND11_TYPE_CASTER(cv::Mat, _("numpy.ndarray"));

        /**
         * Conversion part 1 (Python->C++): convert a PyObject into a inty
         * instance or return false upon failure. The second argument
         * indicates whether implicit conversions should be applied.
         */
        bool load(handle src, bool) 
        {
            /* Try a default converting into a Python */
            array b(src, true);
            buffer_info info = b.request();

            int ndims = info.ndim;

            decltype(CV_32F) dtype;
            size_t elemsize; 
            if (info.format == format_descriptor<float>::format()) { 
                if (ndims == 3) {
                    dtype = CV_32FC3;
                } else {
                    dtype = CV_32FC1;
                }
                elemsize = sizeof(float);
            } else if (info.format == format_descriptor<double>::format()) {
                if (ndims == 3) {
                    dtype = CV_64FC3;
                } else {
                    dtype = CV_64FC1;
                }
                elemsize = sizeof(double);
            } else if (info.format == format_descriptor<unsigned char>::format()) { 
                if (ndims == 3) {
                    dtype = CV_8UC3;
                } else {
                    dtype = CV_8UC1;
                }
                elemsize = sizeof(unsigned char);
            } else { 
                throw std::logic_error("Unsupported type");
                return false;
            }

            std::vector<int> shape = {info.shape[0], info.shape[1]};

            value = cv::Mat(cv::Size(shape[1], shape[0]), dtype, info.ptr, cv::Mat::AUTO_STEP);
            return true;
        }

        /**
         * Conversion part 2 (C++ -> Python): convert an inty instance into
         * a Python object. The second and third arguments are used to
         * indicate the return value policy and parent object (for
         * ``return_value_policy::reference_internal``) and are generally
         * ignored by implicit casters.
         */
        static handle cast(const cv::Mat &m, return_value_policy, handle defval) 
        {
            std::cout << "m.cols : " << m.cols << std::endl;
            std::cout << "m.rows : " << m.rows << std::endl;
            std::string format = format_descriptor<unsigned char>::format();
            size_t elemsize = sizeof(unsigned char);
            int dim; 
            switch(m.type()) {
                case CV_8U: 
                    format = format_descriptor<unsigned char>::format();
                    elemsize = sizeof(unsigned char);
                    dim = 2;
                    break; 
                case CV_8UC3: 
                    format = format_descriptor<unsigned char>::format();
                    elemsize = sizeof(unsigned char);
                    dim = 3;
                    break; 
                case CV_32F: 
                    format = format_descriptor<float>::format(); 
                    elemsize = sizeof(float);
                    dim = 2;
                    break;
                case CV_64F: 
                    format = format_descriptor<double>::format();
                    elemsize = sizeof(double);
                    dim = 2;
                    break;
                default: 
                    throw std::logic_error("Unsupported type");
            }

            std::vector<size_t> bufferdim;
            std::vector<size_t> strides;
            if (dim == 2) {
                bufferdim = {(size_t) m.rows, (size_t) m.cols};
                strides = {elemsize * (size_t) m.cols, elemsize};
            } else if (dim == 3) {
                bufferdim = {(size_t) m.rows, (size_t) m.cols, (size_t) 3};
                strides = {(size_t) elemsize * m.cols * 3, (size_t) elemsize * 3, (size_t) elemsize};
            }
            return array(buffer_info(
                m.data,         /* Pointer to buffer */
                elemsize,       /* Size of one scalar */
                format,         /* Python struct-style format descriptor */
                dim,            /* Number of dimensions */
                bufferdim,      /* Buffer dimensions */
                strides         /* Strides (in bytes) for each index */
                )).release();
        }

    };
}} // namespace pybind11::detail

[edmBernard]

@aldanor I try your modification but i'm not able to find a equivalent of :
info.format to got the data element type

[aldanor]

Yes, what I think happens is this: the array is non-contiguous, and as such the data pointer stops making much sense. PyArray_Check() works just fine, so the handle is happily accepted as a valid array. Note that array_t has some conversion logic through template-level flags, but array doesn't. As of right now, you probably want to manually check if the array is c-style and contiguous.

@wjakob ^ This is quite closely related to flags rework, how do you think cases like this should be solved, ideally/hypothetically? If instead of array one used e.g. generic_array<array::c_array>, would that help here in load()? (presumably, that would make a copy of data in case it's non-contiguous, but then who would own it once you leave the load() function?)

[aldanor]

@aldanor I try your modification but i'm not able to find a equivalent of :
info.format to got the data element type

@edmBernard There's py::dtype::of<float>() and arr.dtype() which you can compare, which is the same as comparing numpy dtypes (this handles a few more corner cases, like int being the same as long on Windows, but with different format strings, structured types, etc). However, as of right now you would have to use PyArray_EquivTypes_() directly.

Maybe we should properly expose this so that pybind11::dtype either gains an operator==(), or equiv_to(py::dtype) method, or equiv_to<T>(), or something like that. Then you could hypothetically check if (arr.dtype().equiv_to<float>()) which would be quite nice.

[edmBernard]

When i try this api.PyArray_EquivTypes_(arr.dtype(), dtype::of<float>())
to check type, it fail with error: cannot convert ‘pybind11::dtype’ to ‘PyObject* {aka _object*}’ in argument passing

but there is an awfull solution : arr.dtype() == dtype(format_descriptor<unsigned char>::format()) that seem to work ^^ (i don't check if there is border effect)

[edmBernard]

I put my 2 versions on git here :
https://github.com/edmBernard/pybind11_opencv_numpy

[edmBernard]

I close as my solution finally work even if it not really clean
Thx for your help

[virtuald]

I've modified @edmBernard 's sample project and replaced the converter with extracts from https://github.com/yati-sagade/opencv-ndarray-conversion ... which really just copies OpenCV's internal cv::Mat <--> ndarray conversion routines so they can be used separate from the main module.

Ultimately, I'd like to add a capsule to OpenCV that exports it's conversion routines so that anyone can use them, instead of just using it internally. Would make this sort of thing a lot easier. Won't be able to finish that for awhile, but WIP for that at virtuald/opencv@6db6d29

Example project is at https://github.com/virtuald/pybind11_opencv_numpy, check it out, it seems to work so far.

[jampekka]

@virtuald Using OpenCV's code is indeed probably the best approach. When I first started hacking the conversion, that was my main idea, but OpenCV doesn't make using just the conversion too easy. In fact its python bindings are such magic I couldn't even start to guess where to look.

[virtuald]

@jampekka As I said above, I've extracted the necessary bits from OpenCV's python bindings and incorporated them into the example project above, you can find them at https://github.com/virtuald/pybind11_opencv_numpy/blob/master/ndarray_converter.cpp .

[edmBernard]

@virtuald thanks for your work, That's a really better solution.

[bhack]

Is it not better to try to merge this solution in the repository?

[virtuald]

@bhack the real solution is adding a capsule to OpenCV for this. It probably doesn't belong in pybind11.

[bhack]

Capsule?

[virtuald]

See virtuald/opencv@6db6d29 for an initial (crashing) implementation of a capsule, similar to how one imports numpy from a C/C++ project.

[bhack]

@vpisarev What do you think? Could it be upstreamed in opencv?

[bhack]

Gently ping @vpisarev