Thursday, May 19, 2011

ODE-ctypes in Blender

ODE-ctypes addon for blender is deprecated, code moved into Pyppet2 core engine.

Tuesday, May 10, 2011

Kinect Hand Tracking

Ctypes and Threads

Check out this ctypes technique to escape the GIL.

Fast Hand Detection

HAAR wavelets are by far the most popular way of detecting hands or other features, the problem is speed, the Haar cascade can easily take a single core to 100% usage, and this is not suitable if we plan to run this hand detector within another program like Blender or RealXtend and maintain good performance.

Another method that is faster is to check for convexity defects of contours, this blog by Andol has a good overview of the techniques. Using the heuristic 4 or more defects is hand, and simply checking for defects among the many contour passes will yield false-positives from noise. The first trick is to filter out this noise on the contour with extreme polygon reduction, using the function cv.ApproxPoly with a factor of 20-30.0 or more. This reduces the head to a few triangles, while keeping the star-shape of the hand.

Saturday, May 7, 2011

Kinect and OpenCV

This research project was made possible by Adminotech. Kinect and OpenCV make a great combination, and now getting started is even easier with this pure-ctypes versions of both Libfreenect and OpenCV2. Python bindings already existed for both Libfreenect and OpenCV, but these must be compiled and linked to a given Python version. This is a pain for developers who need to support multiple platforms and multiple Python versions. Using pure ctypes bindings, only the shared library is needed per-platform, which can even be found pre-compiled. The other benefit of ctypes is threading and escaping the GIL, both Freenect and OpenCV can each run in their own threads.

The current aim of this research is to track the hands with as little CPU resources as possible, and to maintain 30fps read rate from the Kinect device. Streaming both video and depth from the Kinect can have serious effects on performance, so we will need a technique that can work with just the depth buffer. OpenCV is fast enough to sweep through about 12 passes of the depth buffer with increasing thresholds and hold 30fps. This is a good starting point since the contour of the hand is still clearly visible in at least one of the sweeps.

RPythonic Smart Wrappers

Another goal of the project was to make it easy for Python developers who are already familiar with the PyOpenCV API to migrate to the ctypes API. RPythonic now generates much smarter ctypes bindings which hide almost all of the ctypes-broiler-plate so that migration is simply a matter of changing the import header. Callback functions can now be directly passed to wrapped C function, and they will automatically be converted to a PyCFuncPtrType. Also lists and tuples are now valid arguments to wrapped C functions, if the argument type expects an array the tuple or list will be automatically converted. Or, if the argument expects some Structure, the list or tuple is used to initialze the Structure.

There are still some differences with the hand-made PyOpenCV and this automatically generated ctypes version. These mainly concern C functions that are passed a pointer that is then filled with some result, instead of returning a pointer to the result. The hand-made PyOpenCV wrappers deal with this properly and return a pointer without the user having to create it first and pass it to the function.

sample on pastebin