Installing BoofCV on Raspberry PI

Before I talk about getting BoofCV (a computer vision library) up and running, let’s talk about what is needed to get other computer vision libraries running on a Raspberry PI. Then I’ll go over building (not needed), installing (PyBoof technically), and running BoofCV.

Building OpenCV and Other Libraries

For the first time I’m doing actual development on a Raspberry PI and not playing video games. I decided to run a QR Code benchmark I wrote on a desktop computer and see how slow everything is on a Raspberry PI for fun and profit. I have a nifty Python script that builds and runs everything ready to go so this should be a 5 minute task right? Unfortunately this has been bringing back memories of the bad old days when nothing built would build the first time because your setup wasn’t exactly the same as the developers.

First task is to build ZBar. This library hasn’t been maintained in a while and can be tricky to build on a regular desktop. Getting it to compile without errors took an hour or so and who knows how many Google searches. Would have been worse if this was my first time building it and I didn’t have a script that patches the code to fix name conflicts already written.

Second task is installing OpenCV version 3.3.1. That exact version or newer is needed because it contains a fix for jpeg images. As far as I can tell only OpenCV 2.4 is available on apt-get or pip but you might be able to obtain other versions elsewhere, e.g. 3.1.0. Everyone seems to be building it from source. Took me four attempts and about 8 hours to get it working. Build failed a few times (unknown reason or small errors on my part) and built the wrong version of OpenCV once. Now that I know what I’m doing it would take 2 to 3 hours. Most of that is compiling the code.

For sake of being fare, not all libraries are this annoying. SimpleCV can be installed in a few lines using just apt-get according to their instructions.

Building BoofCV

There’s no need to build BoofCV. If you build it from source or download it automatically from Maven Central the code works just the same. It should build out of the box if you’re using Raspbian. If you’re running Ubuntu you need to install Oracle’s JDK because OpenJDK has broken encryption keys and Gradle can’t download the dependencies. If you’re dead set on building BoofCV from source make sure you’re connected to the internet and follow these instructions:

git clone -b master --recursive https://github.com/lessthanoptimal/BoofCV.git boofcv
cd boofcv
./gradlew install

That will download the source code and example data. Example data isn’t required but might be fun to play with. See readme for running examples.

Installing BoofCV

BoofCV is a Java library and you don’t really install it. At least not like a shared library or DLL. It’s either already included in the jar you’re running or you’re building a project and you reference the library using Gradle or Maven. Gradle/Maven will do the hard work for you by automatically downloading all the dependencies. Since this is a non-issue when used as a Java library let’s talk about PyBoof. PyBoof is a Python wrapper around BoofCV and you install that using pip, just like any other Python library.

python3 -m venv venv
source venv/bin/activate
pip install wheel

This should look familiar to almost all python developers. It just sets you up in a safe environment. Wheel is installed because it fixed an issue with the version of pip on my Raspberry PI. Then to install PyBoof just type in the command below. Numpy is a dependency and if you don’t have it installed already this will take a 10 minutes or so to build.

pip install PyBoof==0.29.post3

Now let’s open a image, look for a QR code, and print the results to standard out.

wget https://peterabeles.com/blog/wp-content/uploads/2018/05/qrcode.png

That will download the image for you. Next create a python script with the text below. When you run the script it will pause briefly while it processes the image then print out the results. For best performance process more than one image at a time. The first one is always sluggish.

import numpy as np
import pyboof as pb

# This is needed if you want the code to run fast
pb.init_memmap()

# Load an image as gray scale.=
gray = pb.load_single_band("qrcode.png", np.uint8)

# Declare the QR Code detector
detector = pb.FactoryFiducial(np.uint8).qrcode()

# Process the image, detect the QR Codes, and print the results
detector.detect(gray)
print("Detected a total of {} QR Codes".format(len(detector.detections)))
for qr in detector.detections:
    print("Message: "+qr.message)
    print(" at: "+str(qr.bounds))

Conclusion

BoofCV is very easy to get running on Raspberry PI. Large complex C/C++ projects will take several hours and multiple attempts. Giving you plenty of time to write a blog article.

Code from symbolic math using Sage

Abstract

The following article demonstrates how Sage/Python can be used to solve, format, and output auto generated code for complex symbolic equations.   The auto-generated code can be targeted for languages such as C/C++ or Java.  Sage is a computer algebra system (CAS) that uses the Python programming language.  By using Python it is easy to write the code and to take full advantage of Python’s text processing capabilities.

Introduction

It is not uncommon for researchers and engineers to need to solve complex symbolic equations in languages like C/C++ and Java.  While computer algebra systems (CAS), such as Maple or Mathematica, are designed specifically to handle symbolic math with ease, C style languages are not.  A common solution is to first solve the symbolic equations using Maple/Mathematica, then expand out the closed form  solution and convert into compilable code.

This process is often easier said than done.  What was a nice elegant and compact 3 line equation is expanded out into thousands of lines of code.  Hand editing these monstrous equations is not an option.  Another issue is that commercial packages such as Maple and Mathematica are both very expensive.

Recently I needed solve an ugly problem in computer vision and having just learned Python, Sage was the obvious choice.  Sage is CAS which uses Python as its programming language.  Besides being free, Sage is in many ways superior to Maple and Mathematica thanks to Python’s string processing capabilities. The remainder of this article will describe how Sage and Python was used to solve a complex math problem and convert the solution to usable Java code.

The basic for how to use Sage can be found at the following websites:

In this particular application Java code is the final output.  However, it is trivial to convert this example to output C/C++ code.

The Math

To demonstrate these techniques a complex real-world problem is used.  This problem comes from geometric computer vision and understanding all the math involved is not necessary, but you can find the paper it is based on here [1].  Having an understanding of linear algebra will help you understand all the math terminology.  Briefly stated the problem is to solve for a 3×3 Essential matrix given 5 point correspondences.   If that means nothing to you, don’t worry.

The first step in solving this problem is computing the null space of a 5 by 9 linear system.  The null space is then stored in four column vectors X , Y , Z , and W.  The final solution is a linear combination of these four vectors:

E = x \cdot X + y \cdot Y + z \cdot Z + W

where E is a 3 by 3 Essential matrix being computed, (x,y,z) are the scalar variables being solved for.  Note that the column vectors are converted into a matrix as needed.  Next a system, A, with 10 equations and 20 unknowns is constructed by apply the following constrains on E:

det(E) = 0

E \cdot E^T \cdot E-\frac{1}{2}trace(E \cdot E^T) \cdot E=0

Again if you don’t understand why we are doing the math, don’t panic.  Just know that there are some ugly complex equations which need to be solved for in Java.  In summary we need to do the following:

  • In Java compute the null space and extract out X,Y,Z,W   (Not shown)
  • In Sage construct a system of equation for the above constraints
  • In Sage separate out the known and unknown portions of those equations and output Java code
  • In Java solve for unknowns  (Not shown)

This is actually only 2/3 of the solution.  Since the focus of this article is on the process and not about the details of this particular problem we will skip the rest.

Solving with Sage

All code examples below are written in Python and need to be run inside of Sage’s interactive shell.  Do not try to to run the scripts from Python directly, use Sage.  All python files need the following imports:

from sage.all import *
from numpy.core.fromnumeric import var

The first step in solving this problem is creating several 3×3 symbolic matrices.  Each element in the matrix is unique.  I couldn’t find a function built into Sage, so I wrote my own:

def symMatrix( numRows , numCols , letter ):
 A = matrix(SR,numRows,numCols)
 for i in range(0,numRows):
 for j in range(0,numCols):
 A[i,j] = var('%s%d%d' % (letter,i,j) )
return A

The ‘var’ function is used by Sage to define symbolic variables.  Inside of the Sage interactive shell you can do the following:

sage: A = symMatrix(3,3,'a')
sage: A
[a00 a01 a02]
[a10 a11 a12]
[a20 a21 a22]

Now that you can create a symbolic matrix you can solve rest of the problem as follows:

x,y,z = var('x','y','z')
 
X = symMatrix( 3, 3 , 'X')
Y = symMatrix( 3, 3 , 'Y')
Z = symMatrix( 3, 3 , 'Z')
W = symMatrix( 3, 3 , 'W')
 
E = x*X + y*Y + z*Z + 1*W
EE = E*E.T
 
eq1=det(E)
eq2=EE*E-0.5*EE.trace()*E
 
eqs = (eq1,eq2[0,0],eq2[0,1],eq2[0,2],eq2[1,0],eq2[1,1],eq2[1,2],eq2[2,0],eq2[2,1],eq2[2,2])
 
keysA = ('x^3','y^3','x^2*y','x*y^2','x^2*z','x^2','y^2*z','y^2','x*y*z','x*y')
keysB = ('x*z^2','x*z','x','y*z^2','y*z','y','z^3','z^2','z','')
 
# print out machine code for the linear system
printData('A',eqs,keysA)
printData('B',eqs,keysB)

The functions det and trace compute the determinant and trace of a matrix respectively, A.T is the transpose of A.  Most of that is straight forward algebra, but what about that last bit? Well the printData() function massages those equations to produce usable Java output, which is the subject of the next section.

Generating Code

What the Java code is going to do is solve a linear system. To do that the equations above need to reformatted and converted into matrix format. The variable eqs in the above python code contains equations which have a similar appearance to the equation below on the left, but they need to be expanded out so that they look like the equation on the right.

(1 + 5x)(2 - 3x)(8 + 4y) \rightarrow -60x^2y - 120x^2 + 28xy + 56x + 8y + 16

After the equations have been expanded, the knowns and unknowns are separated and put into matrix format. Consider the system of equations below:

$latex \begin{array}{c}
x + 2xy + 3x^2 = 4 \\
5x + 6xy + 7x^2 = 8 \\
9x + 10xy +11x^2 = 12 \\
\end{array} $

After it has been converted into matrix format it will look like:

$latex A=\left[
\begin{array}{ccc}
1 & 2 & 3 \\
5 & 6 & 7 \\
9 & 10 & 11 \\
\end{array}
\right]
\;\;
B=\left[
\begin{array}{c}
4 \\
8 \\
12 \\
\end{array}
\right]$

At this point the unknowns can be solved for using standard linear algebra tools, e.g. Ax=B.

Part of this process is shown below.  First eq1 is expanded out and then the coefficients of x^3 are extracted.

sage: len(str(eq1))         
465
sage: len(str(eq1.expand()))
7065
sage: extractVarEq(eq1.expand(),'x^3')
'X00*X11*X22 - X00*X12*X21 - X01*X10*X22 + X01*X12*X20 + X02*X10*X21 - X02*X11*X20'

The first two statements are just there to show you how long and ugly these equations are.  Below is the python code for extractVarEq:

def extractVarEq( expression , key ):
  chars = set('xyz')
  # expand out and convert into a string
  expression = str(expression.expand())
  # Make sure negative symbols are not stripped and split into multiplicative blocks
  s = expression.replace('- ','-').split(' ')
  # Find blocks multiplied by the key and remove the key from the string
  if len(key) == 0:
    var = [w for w in s if len(w) != 1 and not any( c in chars for c in w )]
  else:
    var = [w[:-(1+len(key))] for w in s if (w.endswith(key) and not any(c in chars for c in w[:-len(key)])) ]
 
  # Expand out power
  var = [expandPower(w) for w in var] 
 
  # construct a string which can be compiled
  ret = var[0]
  for w in var[1:]:
    if w[0] == '-':
      ret += ' - '+w[1:]
    else:
      ret += ' + '+w
 
  return ret

The function expandPower(w) converts statements like a^3 into a*a*a which can be parsed by Java.  After processing all 10 equations most of the hard work is done,   Addition processing is done to simplify the equations, but the code is a bit convoluted and isn’t discussed here.  The final output it sent into a text file and then pasted into a Java application.

Want to see what the final output looks like? Well here is a small sniplet:

A.data[0] = X20*( X01*X12 - X02*X11 ) + X10*( -X01*X22 + X02*X21 ) + X00*( X11*X22 - X12*X21 );
A.data[1] = Y02*( Y10*Y21 - Y11*Y20 ) + Y00*( Y11*Y22 - Y12*Y21 ) + Y01*( -Y10*Y22 + Y12*Y20 );
A.data[2] = X22*( X00*Y11 - X01*Y10 - X10*Y01 + X11*Y00 ) + X20*( X01*Y12 - X02*Y11 - X11*Y02 + X12*Y01 ) + X21*( -X00*Y12 + X02*Y10 + X10*Y02 - X12*Y00 ) + X01*( -X10*Y22 + X12*Y20 ) + Y21*( -X00*X12 + X02*X10 ) + X11*( X00*Y22 - X02*Y20 );

The full source code can be downloaded from:

[1] David Nister “An Efficient Solution to the Five-Point Relative Pose Problem” Pattern Analysis and Machine Intelligence, 2004

Programming in PostScript

 

Calibration target created using PS. Corners are detected and at a known distances apart, allowing intrinsic camera parameters to be determined.

Recently I needed to design and create a calibration grid for calibrating a camera.  The calibration grid was going to be very simple, just composed of squares, but their locations needed to be precise.  I couldn’t just slap something together in power point and call it good enough.

After considering a few options, I decided to give writing my own PostScript (PS) document from scratch a try.  PS is a document standard best known for printers.  Almost all laser printers can print PS documents.  If you have ever looked at a PS document as raw text you know it’s ugly.  Turned out to not be all that bad to work with after all.

PS is in fact a real programming language and is Turing complete.  Someone has even coded up the game of life, which you can run on your favorite or not so favorite printer!  Once you get used to how PS loads everything from the stack, it’s not all that bad of a scripting language.

The two tutorials/documents which I found useful are:

Less than two hours after I started I now know the basics of PS and created the PS document below.  See in line comments for explanation of the code.

%!PS
% paper size
/pagewidth 8.5 72 mul def
/pageheight 10 72 mul def 
 
% ----- Define centimeter
/cm {28.346 mul} def
% ----- square width
/w {3 cm} def
/ww {w 2 mul} def
/r {w 2 div} def
% ----- Define procedure for drawing a box
/box {w 0 rlineto 0 w rlineto -1 w mul 0 rlineto closepath} def
 
% take in account the size of a square
/pagewidth pagewidth ww sub def
/pageheight pageheight w sub def
 
% draw all the boxes it can across a single row
/rowboxes {w ww pagewidth {newpath y moveto box fill} for} def
 
% increments the y variable and draws all the rows
w ww pageheight { /y exch def rowboxes } for
 
showpage