Tuesday, September 28, 2010

RPythonic

RPythonic takes in C code and outputs three different formats: 1. pure RPython code, 2. ctypes wrappers, and 3. rffi wrappers. Using the PyPy translation toolchain the pure RPython code can be translated back to C, or C# and Java (allowing your C code to be executed by .Net or Java runtimes). The generated ctypes wrapper can be used from CPython. The rffi wrappers allow you to use C code from RPython. RPythonic is built using PycParser, a powerful C parser written by Eli Bendersky written in Python.

Why not Pygccxml?
Pygccxml is made up of about 50 files and several thousands of lines of code, totalling about 1MB of code; and relies on the external program Gccxml. Gccxml is old and development has stalled. Another automatic binding generator Pybindgen uses Pygccxml, and the interface between Pybindgen and Pygccxml alone is over two thousands lines of code, not pretty!

PycParser is super clean, 10 files, less than 150KB of code; is pure Python and relies on no external programs. While Pygccxml can parse C/C++ code, it can not parse the function bodies. On the other hand, PycParser can fully parse C and function bodies, but not C++. For RPython and ctypes wrappers we are only interested in C; and because PycParser can fully parse C, translation of C to RPython is also possible at the same time the bindings are generated.

RPythonic on google code

Thursday, September 16, 2010

Multiprocessing


OpenCV is multithreaded, and you can set the number of threads it uses by cvSetNumThreads(N). The haar cascades used for face and other feature detection will fully utilize all the threads you give it. Using glib.timeout_add( miliseconds, self.loop ) even with 4 threads you will still get unacceptable lag doing haar detection on a 320x240 image; the GUI will become unresponsive because glib will not update its own events if the timeout is always fully loaded.

The solution is spliting the process in two so that pygtk can have its own core and update without being blocked by the OpenCV/Pygame process. One approach for this is to use the subprocess module, and serialize data from the GUI to the OpenCV/Pygame process over a pipe - but this is rather slow. The fastest approach available in Python is the multiprocessing module that uses ctypes wrapped sharedmemory. If sharedmemory is only read from the OpenCV/Pygame process then there will be no data copy overhead, giving us the maximum possible speed. (copy-on-write is implemented in hardware and supported by modern Linux kernels.)

Ctypes restricts what types can be shared, so its not possible to share a dictionary or instance. Luckly ctypes has a simple interface for using C-level structs, and all the data from the GUI can easily be reformatted from using a dictionary to using structs in an array.

_cfg_ubytes = 'active alpha blur athresh_block_size thresh_min thresh_max'.split()
_cfg_ubytes += 'FXstencil FXblur FXsobel FXathresh FXthresh FXdetect'.split()
class LayerConfig( ctypes.Structure ):
_fields_ = [ ('colorspace',ctypes.c_int) ]
for tag in _cfg_ubytes: _fields_.append( (tag, ctypes.c_ubyte) )
del tag
mysharedmemory = multiprocessing.sharedctypes.Array( LayerConfig, [ (cv.CV_BGR2RGB,) ], lock=False )

source code