Saturday, May 7, 2011

Kinect and OpenCV

This research project was made possible by Adminotech. Kinect and OpenCV make a great combination, and now getting started is even easier with this pure-ctypes versions of both Libfreenect and OpenCV2. Python bindings already existed for both Libfreenect and OpenCV, but these must be compiled and linked to a given Python version. This is a pain for developers who need to support multiple platforms and multiple Python versions. Using pure ctypes bindings, only the shared library is needed per-platform, which can even be found pre-compiled. The other benefit of ctypes is threading and escaping the GIL, both Freenect and OpenCV can each run in their own threads.

The current aim of this research is to track the hands with as little CPU resources as possible, and to maintain 30fps read rate from the Kinect device. Streaming both video and depth from the Kinect can have serious effects on performance, so we will need a technique that can work with just the depth buffer. OpenCV is fast enough to sweep through about 12 passes of the depth buffer with increasing thresholds and hold 30fps. This is a good starting point since the contour of the hand is still clearly visible in at least one of the sweeps.

RPythonic Smart Wrappers

Another goal of the project was to make it easy for Python developers who are already familiar with the PyOpenCV API to migrate to the ctypes API. RPythonic now generates much smarter ctypes bindings which hide almost all of the ctypes-broiler-plate so that migration is simply a matter of changing the import header. Callback functions can now be directly passed to wrapped C function, and they will automatically be converted to a PyCFuncPtrType. Also lists and tuples are now valid arguments to wrapped C functions, if the argument type expects an array the tuple or list will be automatically converted. Or, if the argument expects some Structure, the list or tuple is used to initialze the Structure.

There are still some differences with the hand-made PyOpenCV and this automatically generated ctypes version. These mainly concern C functions that are passed a pointer that is then filled with some result, instead of returning a pointer to the result. The hand-made PyOpenCV wrappers deal with this properly and return a pointer without the user having to create it first and pass it to the function.

sample on pastebin

No comments:

Post a Comment