Make a "proper" PDL::OpenCV module

[mohawk2]

Currently there is (from 2013) an XS module https://metacpan.org/pod/Cv, and not on CPAN (and naively generated from header files) https://github.com/dkogan/PDL-OpenCV - it would be good to make a real PDL module, possibly starting from those.

[mohawk2]

It seems @fantasma13 has started to implement this! https://sourceforge.net/p/pdl/mailman/message/37620582/

[fantasma13]

I've started PDL::OCV, which tries to link PDL and a more modern OpenCV. With v4 the C interface was dropped, which complicates matters significantly.

[fantasma13]

Please help me with creating persistent perl or C bindings to opaque C structs holding objects!

[mohawk2]

I will take a look at this very soon! May I suggest renaming the repo (and modules) to PDL-OpenCV? Dima's one was never uploaded to CPAN, so the name is free, and "OCV" would be terribly unclear for people to tell what it is.

[mohawk2]

Did you take a look at Libtmp/GSL/INTERP and its use of the gsl_interp * as recommended for persistent pointer entities?

[fantasma13]

Finally I figured how to use the pointers and references. The GSL example has two pointers wrapped in an array, that got me confused for a while. I dereferenced one level off.
I've pushed a minimal working OpenCV binding, you can create a float grayscale Mat() object from a piddle and query it with mat_at (returning a single value piddle). That's not too exciting yet but it is a start.

[fantasma13]

The module is significantly more complete now. All basic OpenCV data types are supported, video IO works, so PDL has now a way to read videos.
Basic code generation frameworks exist for simple wrapper functions to methods.
The object tracking class should demonstrate how to interface OpenCV modules/classes.

[fantasma13]

Is the issue resolved?

[mohawk2]

Great work! I'd like to leave this open a bit longer, until the module is on CPAN. @fantasma13 Would you mind if I look over the code and make some tweaks?

[mohawk2]

Implied tasks: we need an Alien::OpenCV

[mohawk2]

With this input as headers.txt (which is cut down from the hardcoded debugging-orientated list in hdr_parser.py):

opencv2/core.hpp
opencv2/highgui.hpp
opencv2/imgproc.hpp
opencv2/tracking.hpp
opencv2/videoio.hpp
opencv2/imgcodecs.hpp
opencv2/calib3d.hpp
opencv2/objdetect.hpp
opencv2/features2d.hpp
opencv2/flann/miniflann.hpp
opencv2/video/background_segm.hpp

And a checkout of https://github.com/opencv/opencv in ../opencv, this script:

HEADERS=headers.txt
HEADERS_DIR=/usr/include/opencv4
CVDIR=../opencv
perl -pe 'BEGIN {$dir=shift}s:^:$dir/:' $HEADERS_DIR $HEADERS |
  (cd $CVDIR/modules/python/src2/; python3 -c '
    import hdr_parser, json, sys
    parser = hdr_parser.CppHeaderParser()
    json.dump([[x,parser.parse(x)] for x in sys.stdin.read().splitlines()], sys.stdout)'
  )

produces JSON describing the various enums, Python classes, and methods thereof (with the doc), which can be processed to generate wrappers for the C++ methods.

[fantasma13]

Brilliant idea!

[mohawk2]

I've been conceptually blocked for a while by how to handle more complicated inputs/outputs, particularly std::vectors of data-structures. The C-wrapping of those, together with an obvious API, is easy. Wrapping of a vector of a fixed-size, single-type entity like Size will just be done as an n+1-dimensional ndarray. However, the handling of the I/O from/to the Perl side of non-fixed-size ndarrays, and multi-level vectors, has not been obvious.

I now think the way forward is:

• have OtherPars as the C-wrapped type (e.g. vector_vector_KeyPoint *)
• the Perl PMCode passes in an SV (aka Perl scalar) with undef value
• the typemap INPUT throws an error if the SV had anything other than undef
• the typemap constructs a new C-wrapper object
• XS passes that to the pdl_*_run function
• the PP code passes it to the C wrapper for the OpenCV function as normal (which will populate it)
• the XS code runs typemap OUTPUT code to turn the value into the SV (a new PDL capability), copying values as appropriate (almost certainly to an array-ref), then destroys the C-wrapper object

This preserves the C API concept of #358 so the pdl_*_run doesn't need to know about Perl, while still allowing this more complex capability. Before the capability can be used in the forthcoming PDL::OpenCV, it will need adding to main PDL first, which is why, as I expected, P:OCV will need a very-recent and indeed not-yet-created version of PDL to work.

As part of this, it may be useful to add pp_add_typemaps to PDL::PP, so that it won't be as necessary to have actual typemap files.

[mohawk2]

This will also need adding a modifier [o] to the relevant OtherPars entry, and making that blow up at the PP compile stage if HaveBroadcasting is true.

[mohawk2]

Instead, the broadcast check is runtime.

[mohawk2]

Next steps will be:

• convert the current “comp mode” code over to using output OtherPars
• add the missing fixed-dim types
• OpenCV vector-of-vectors to Perl array-of-arrays in XS (input and output)
• doc-translating from Doxygen to POD
• picking which method to translate if there are several similar ones
• add in the mass of functions, fixing gremlins that show

[mohawk2]

My working notes:

general

"skip" for cv.{h,v}concat, transpose
wrap FileStorage, Algorithm at top
exercise Algorithm with cv.BackgroundSubtractor
handle Ptr_
handle Point2f etc
no need core/mat.hpp
handle defaults: Mat(), noArray()
miniflann wrap IndexParams (is simple hashref), see https://github.com/opencv/opencv/blob/4.x/modules/flann/include/opencv2/flann/all_indices.h
cvflann_flann_distance_t is cvflann::flann_distance_t, see https://github.com/opencv/opencv/blob/4.x/modules/flann/misc/python/pyopencv_flann.hpp
cv cvflann_flann_algorithm_t - all enums
wrap SearchParams, subclass of IndexParams
cv.GeneralizedHough.setTemplate parse default Point(-1,-1)

cv.CLAHE.getTilesGridSize returns a Size, check works

docs

doc enums as =head2 under =head1 ENUMS, values as =head3
# = L</...>
overloaded, pick longest doc - cf cv.integral in imgproc
\f$...\f$ \f(...\f) \f[...\f]
if compmode, add "B.\n\n" at top

Subdiv2D

cv.Subdiv2D.insert vector_Point2f form - picking form more params, else highest-D param(s)
cv.Subdiv2D.getEdgeList vector_Vec4f
cv.Subdiv2D.getLeadingEdgeList vector_int
cv.Subdiv2D.getTriangleList vector_Vec6f
cv.Subdiv2D.getVoronoiFacetList vector_vector_Point2f
cv.Subdiv2D.edgeDst Point2f* /O
cv.createLineSegmentDetector returns Ptr_LineSegmentDetector
cv.erode default morphologyDefaultBorderValue()
cv.remap default Scalar()
cv.pyrUp default Size()
cv.calcHist multiple vector_ inputs - each independent size
cv.createCLAHE returns Ptr_CLAHE, cf Tracker*
cv.wrapperEMD default Ptr(), also output with default Mat()
cv.pyrMeanShiftFiltering TermCriteria (termination) with default
cv.moments returns Moments
cv.minAreaRect returns RotatedRect
cv.fillPoly takes vector_Mat
cv.drawContours maxLevel default INT_MAX, offset default Point()

calib3d

cv.solveP3P has output vector_Mat
struct cv.CirclesGridFinderParameters
cv.findCirclesGrid takes above also Ptr_FeatureDetector (from features2d module, typedef-ed to Feature2D)
cv.rectify3Collinear takes Rect* /O
cv.calibrateHandEye takes HandEyeCalibrationMethod (enum)
cv.StereoBM.create is constructor, ret Ptr_StereoBM, ["/S"]

features2d

cv.Feature2D.detect output vector_vector_KeyPoint
cv.Feature2D.compute variant with vector_vector_KeyPoint,["/O","/IO"]
cv.Feature2D.read variants one with String one FileNode
cv.Feature2D.write takes Ptr_FileStorage
cv.ORB.create takes ORB_ScoreType
cv.MSER.detectRegions returns vector_vector_Point
cv.FastFeatureDetector.setType takes FastFeatureDetector_DetectorType
cv.GFTTDetector.create one variant has more defaults set
struct cv.SimpleBlobDetector.Params
cv.SimpleBlobDetector.Params.Params empty return type, ignore?
cv.SimpleBlobDetector.create takes SimpleBlobDetector_Params
cv.DescriptorMatcher.knnMatch returns vector_vector_DMatch (from core/types.hpp)
cv.DescriptorMatcher.create takes either DescriptorMatcher_MatcherType or String
cv.FlannBasedMatcher.FlannBasedMatcher indexParams typed Ptr_flann_IndexParams default makePtrflann::KDTreeIndexParams()
cv.drawMatches default Scalar::all(-1), also std::vector()

tracker

cv.TrackerGOTURN.create empty params yet "@param parameters GOTURN parameters TrackerGOTURN::Params"
class cv.KalmanFilter has read/write properties

background_segm

cv.createBackgroundSubtractorMOG2 needs changing to cv.BackgroundSubtractorMOG2.create

objdetect

cv.createBackgroundSubtractorKNN ditto
cv.QRCodeDetector.detectAndDecode returns std::string

imgcodecs

cv.imread doc has - for list

[mohawk2]

Update: vector of both primitive types (int, float) and the fixed-dim types (Rect, Point, etc) are now implemented, for both input and output. Those outputs still use what I call "comp mode" when the number of outputs (i.e. one of the dimensions) is unknown.

To describe "comp mode": the relevant output variables are put in the Comp of the generated PDL operation, and the OpenCV call is made not in the Code section as normal (i.e. broadcasting - in the generated readdata function), but in the MakeComp section (in the generated run function). This means no broadcasting is possible for those operations, but RedoDimsCode can then look in the Comp data to get the dimensions of the output, and set the dimensions of the relevant ndarray, letting PDL take care of allocation.

The returning of string values works. There is also partial parsing of Doxygen docs, and a dumb and partial conversion of that to POD. It's comprehensible and includes =for ref for PDLDoc, and I generated a useful =for example just showing a call in PDL style, but could be better.

All the functions and methods are put in appropriate packages under PDL::OpenCV, and none are added to the PDL namespace. Only package-less functions, and class-static methods (a thing in C++ which I chose to just make exported functions rather than $class->method(...)) are exportable.

Remaining is to make the vector of non-fixed dims work (vectors of vectors, and vectors of Mat). There are several options, but I will probably start with having them in the typemap in some way, and an obvious thing is to have OtherPars that use both the [o] option, and also pdl *paramname[]. I wouldn't be surprised if using both doesn't work yet (it would probably only need adding the pass-by-ref of the generated _count variable), but making that work is the way to achieve the aim here, while remaining purely a C interface for the generated _run function without any SV * getting passed around.

I will also be wrapping the more interesting (ideally, nearly all) the remaining classes, especially Subdiv2d.

So far, to make the dev cycle relatively quick, I have limited the number of wrapped functions to what I use in my testing, expanding by one example per added wrapping-feature. However, as I am using the now repo-tracked maint/genlists (which uses the Python binding-generator as outlined in a previous comment), by taking out the restrictions it can wrap a big portion of the available API (in the finite modules I am generating this for so far). When the above work is finished (Real Soon Now™), then a fairly full PDL::OpenCV will be hitting CPAN!

[mohawk2]

Vectors of strings, of Mats, and of other vectors now works. They get input/output as array-refs, of strings/ndarrays. That uses the typemap via e.g. [o] vector_vector_Point2fWrapper * rather than a [] syntax.

Next is to generalise wrapping of other classes, including allowing more fancy constructors via polymorphism. That will also apply to at least VideoCapture.open, to allow both passing a string, and an integer (to allow opening the user's webcam).

[mohawk2]

The wrapping of all the classes in the enabled modules is now complete, and polymorphism in both constructors and other functions/methods is handled in a naive but effective way, by just numbering them: e.g. new, new2, etc.

Remaining steps:

• close the remaining gap of unimplemented core classes, being RNG, TermCriteria, RotatedRect
• cut down the Megamind.avi test file to a representative few frames so it becomes "fair use"
• enable all the modules in at least the Ubuntu-packaging of OpenCV
• release!

[mohawk2]

The first two of these are done. The additional modules are revealing a few extra issues like deeper namespaces (cv::ml etc), but nothing insurmountable.

[mohawk2]

Point 4 was also done (i.e. PDL::OpenCV 0.001 is now on CPAN). Point 3 (add the other modules) is ongoing, subject to the issues of deeper-namespace (now at least partly addressed) and also a more accurate handling of wrapped classes than the current heuristic.