Camera calibration and 3D Reconstruction

2 kinds of parameters

Coordinates linked by

Projection of a P point

However, pixels in image sensor may not be square, so we distinguish $f_x$ and $f_y$ (usually the same though)

Optical center may not match the center of the image coordinate system, $c_x, c_y$​

$\gamma$ is the skew between axis (usually 0)

The intrinsic matrix K is

Goal of camera calibration: find R, t and K

Calibration process

Step 1: define real world coords

Step 2: capture multiple images of the checkerboard from different viewpoints

Step 3: Find 2D coordinates of checkerboard

Step 4: Calibrate camera

Step 5: Undistortion

The goal of Perspective-n-Point is to find the pose of an object when

Estimating the pose is finding 6 degrees of freedom (3 for translation + 3 for rotation)

We need

Direct Linear Transform is used to solve extrinsic matrix, and can be used anytime when the equation is almost linear but is off by an unknown scale.

However, DLT solution does not minimize the correct objective function. Ideally, we want to minimize the reprojection error.

When the pose estimate is incorrect, we can calculate a re-projection error measure — the sum of squared distances between the projected 3D points and 2D facial feature points.

Levenberg-Marquardt optimization allows to iteratively change the values of $R$ and $t$ so that the reprojection error decreases. LM is often used for non linear least square problems.

cv::solvePnPRansac instead of cv::solvePnP because after the matching not all the found correspondences are correct (outliers)

The projection of the different points on OX form a line on right plane (line l′). We call it epiline corresponding to the point

Similarly all points will have its corresponding epilines in the other image. The plane XOO′ is called Epipolar Plane.

O and O′ are the camera centers. Projection of right camera O′ is seen on the left image at the point, e. It is called the epipole

All the epilines pass through its epipole. So to find the location of epipole, we can find many epilines and find their intersection point.

So in this session, we focus on finding epipolar lines and epipoles

But to find epipolar lines, we need two more ingredients, Fundamental Matrix (F) and Essential Matrix (E).

Essential Matrix contains the information about translation and rotation, which describe the location of the second camera relative to the first in global coordinates

Fundamental Matrix contains the same information as Essential Matrix in addition to the information about the intrinsics of both cameras so that we can relate the two cameras in pixel coordinates

Fundamental Matrix F, maps a point in one image to a line (epiline) in the other image

If we are using rectified images and normalize the point by dividing by the focal lengths, F=E

A minimum of 8 such points are required to find the fundamental matrix (while using 8-point algorithm). More points are preferred and use RANSAC to get a more robust result.

The problem with this particular scene is that almost all matched points lie on the same plane in 3d space. This is a known degenerate case for fundamental matrix estimation. Take a look these slides for explanation. I tried to increase 2nd nearest neighbor threshold in this code from 0.7 to 0.8:

In general fundamental matrix estimation is very sensitive to quality of matches and number of outliers, since opencv uses 8-point algorithm for model estimation. Try to work with higher resolution images with rich textures and non-planar scenes.

equivalent vectors, which are related by just a scaling constant, form a class of homogeneous vectors

The set of all equivalent classes, represented by (a,b,c) forms the projective space. 

We use the homogeneous representation of homogeneous coordinates to define elements like points, lines, planes, etc., in projective space.

We use the rules of projective geometry to perform any transformations on these elements in the projective space.

Camera calibration gives us the projection matrix $P_1$ for $C_1$ and $P_2$ for $C_2$​

We define the $[C_1,x_1,X)$ ray as $X(k)=P_1^{-1}x_1+kC_1$, where $P^{-1}$ is the pseudo inverse of $P$​

$k$ is a scaling parameter as we do not know the actual distance C1 → X

Need to find epipolar line $L_2$ to reduce the search space for a pixel in $i_2$ corresponding to pixel $x_1$ in $i_1$​

Hence to calculate $L_2$, we first find two points on ray $R_1$, and project them in image $i_2$ using $P_2$​

Let's use $C_1$ and $P_1^{-1}x_1\;(k=0)$: it yields $P_2C_1$ (the epipole $e_2$) and $P_2P_1^{-1}x_1$​

In projective geometry, a line can be define with the cross product of two points

In projective geometry, if a point x lies on a line L:

This is a necessary condition for the two points x1 and x2 to be corresponding points, and it is also a form of epipolar constraint

Let's use the F matrix to find epilines:

We simplify the dense matching problem even more with stereo disparity

$disparity= x' – x = Bf/Z$, where $B$ is the baseline, $f$ the focal length

Create your own textured 3D model

The application needs an input image of the object to be registered and its 3D mesh.

We have also to provide the intrinsic parameters of the camera with which the input image was taken.

Starts up extracting the ORB features and descriptors from the input image

Uses the mesh along with the Möller–Trumbore intersection algorithm to compute the 3D coordinates of the found features.

Finally, the 3D points and the descriptors are stored in different lists in a file with YAML format which each row is a different point

3D textured model in YAML file

Load Caffe pose estimation module

Load input img into the model by converting into blob

Make prediction and parse keypoint