Using numpy C API with pybind11

[artiit]

My numpy C api related code:

PyObject* GetRet(Ret* ret) {
  npy_intp shape[1];
  shape[0] = 1024;
  PyArray_Descr* descr = PyArray_DescrFromType(NPY_OBJECT);
  PyObject* obj = PyArray_Empty(1, shape, descr, 0);
  PyArrayObject* np_array = reinterpret_cast<PyArrayObject*>(obj);
  PyObject** out = reinterpret_cast<PyObject**>(PyArray_DATA(np_array));
  for (int i = 0; i < ret->size(); ++i) {
      out[i] = PyBytes_FromStringAndSize(ret[i]->data(), ret[i]->size());
    }
  return obj;
}

pybind11 wrapper like:

m.def("get_ret", [](Ret* ret){
//...
return py::interpret_steal<py::object>(GetRet(ret));
});

In this case, should I use py::reinterpret_steal instead of py::reinterpret_borrow and what py::return_value_policy should I take?
I found that the return_value_policy does not matter cause both reference and take_ownership just worked well.

Another question is external call function is much more time consuming than internal function execution. For example, I have two py file.
test.py

import my_pybind11_wrapper_moudule as mm
def foo():
  t1 = time.time()
  res = mm.get_ret(Ret) # wrapper of GetRet
  t2 = time.time()
  time1 = t2-t1
  return res

main.py

t1 = time.time()
res = foo()
t2 = time.time()
time2 = t2-t1

When I use steal, time2 is 30ms when time1 is 25ms, when I use borrow, both are 30ms.

Any suggestions will be appreciate.

Thanks.

[YannickJadoul]

In general, if you just have questions and not really an issue with pybind11, please come to Gitter, please.

In this case, should I use py::reinterpret_steal instead of py::reinterpret_borrow and what py::return_value_policy should I take?

Depends on what the raw CPython C API says. If you need to increase the reference count, you need reinterpret_borrow, otherwise reinterpret_steal works. In numpy.h, reinterpret_steal is also used on newly created PyObject*s.
See also https://pybind11.readthedocs.io/en/stable/reference.html#_CPPv46object

Another question is external call function is much more time consuming than internal function execution.

See e.g., #1227. E.g., no conversions between Python and C++ are happening when you call a pure Python function.