Kinect Hand Tracking

The OpenCV-ctypes bindings may have a high overhead per-call because of the thick broiler-plate hiding ctypes, and ctypes itself using calling via libffi. However, given that quad-core CPU's are now standard; ctypes escaping the GIL can more than make up for the overhead, and gain higher performance than the official compiled OpenCV bindings.

The first Kinect/OpenCV test is dual threaded, with libfreenect and OpenCV each running in their own threads. In the graph to the right we can see the CPU usage, the blue and green cores are doing most of the work, and are out of phase - meanwhile the red and orange cores are not being used much. The main Python thread runs 70million operations in 30 seconds, and the frame rate display is low, with only about 100 updates.

Much of the work OpenCV is doing can be done in two passes, so it becomes easy to split the code into two threads. The tripple-threaded version seen to the right showed much better CPU usage. Blue and green cores are now running in-phase, without the blue core spiking. The red and orange cores now do much more work, and are only slightly out of phase. The main Python thread runs 50million operations in 30 seconds, lower than the dual threaded version, but the result is a four times faster display update - with 400 updates.

---

Hand Detection

HAAR wavelets are by far the most popular way of detecting hands or other features, the problem is speed, the Haar cascade can easily take a single core to 100% usage, and this is not suitable if we plan to run this hand detector within another program like Blender or RealXtend and maintain good performance.

Another method that is faster is to check for convexity defects of contours, this blog by Andol has a good overview of the techniques. Using the heuristic 4 or more defects is hand, and simply checking for defects among the many contour passes will yield false-positives from noise. The first trick is to filter out this noise on the contour with extreme polygon reduction, using the function cv.ApproxPoly with a factor of 20-30.0 or more. This reduces the head to a few triangles, while keeping the star-shape of the hand.