Image processing

Linear filtering

Non-linear filtering

Feature detection

Feature detectors

1. Point features
2. Aperture problem
3. spatially varying weighting (or window) function: https://en.wikipedia.org/wiki/Window_function
4. auto-correlation function or surface
5. Adaptive non-maximal suppression (ANMS).
6. Measuring repeatability.
7. Scale invariance
8. SIFT

9. Rotational invariance and orientation estimation

   • dominant orientation
10. aggregation window vs detection window
11. Affine invariance

Feature descriptors

Feature descriptors are used to match keypoints retrieved by feature detection algorithms in different images. When we have extracted descriptors from each feature in at least two images, we begin by selecting a matching strategy (determined by the context, e.g. image stitching, object detecting, etc) that determines which correspondences that are qualified to the next stage for further processing. To perform matching as efficiently as possible the second step is to select performant data structures for the corresponding problem.

1. Bias and gain normalization (MOPS).
2. Scale invariant feature transform (SIFT) : https://en.wikipedia.org/wiki/Scale-invariant_feature_transform
3. PCA-SIFT.
4. Gradient location-orientation histogram (GLOH).
5. Steerable filters.

6. ROC curve

   • the use of TP (true positives), FP (false positive), TN (true negatives), and FN (false negatives) when measuring the performance of the matching strategy.
7. Nearest Neighbor Distance Ratio (NNDR)

8. Efficient matching

   • indexing structure can be used (hash maps)
   • multi-dimensional hashing
   • locality sensitive hashing
   • parameter sensitive hashing
   • k-d trees

Verification of matches

When we have gotten matches from the above steps we can use geometric alignment to verify inliers and outliers in the matches.

1. Random sampling (RANSAC)

Feature tracking

Instead of independently finding features in images and then match them with each other, we could find likely feature locations in the first image and search in these locations in the next images. Commonly used in video tracking applications.

Camera

2D point

$$\mathbf{x}=\left[\begin{array}{c}x\\ y\end{array}\right]$$

Homogeneous coordinates

Homogeneous coordinates are given on the form

$$\left[\begin{array}{c}x\\ y\end{array}\right]\to w\cdot \left[\begin{array}{c}x\\ y\\ 1\end{array}\right],w\ne 0$$

De-homogeneous coordinates

$$\left[\begin{array}{c}x\\ y\\ w\end{array}\right]\to \left[\begin{array}{c}x/w\\ y/w\end{array}\right]$$

Basic set of 2D transformations

Basic transformation with homogeneous coordinates.

Translation + rotation

$$\tilde{\mathbf{x}}=\left[\begin{array}{ccc}\cos \theta & -\sin \theta & t_x\\ \sin \theta & \cos \theta & t_y\end{array}\right]\mathbf{x}$$

Scaled rotation

$$\tilde{\mathbf{x}}=\left[\begin{array}{ccc}a& -b& t_x\\ b& a& t_y\end{array}\right]\mathbf{x}$$

Affine

$$\tilde{\mathbf{x}}=\left[\begin{array}{ccc}{a}_{00}& a_{01}& a_{02}\\ a_{10}& a_{11}& a_{12}\end{array}\right]\mathbf{x}$$

Projective

Projective is also known as homography.

Orthography

Contrary to how projective projection, orthographic projection simply drops the z component of a three-dimensional coordinate to obtain the 2D point.

Homography

Homography is the technique of using projection mapping with the homogeneous coordinates to achieve the transformation. It is written like this

$$\left[\begin{array}{c}\tilde{x}\\ \tilde{y}\\ \tilde{w}\end{array}\right]=\left[\begin{array}{ccc}a& b& c\\ d& e& f\\ g& h& i\end{array}\right]\left[\begin{array}{c}x\\ y\\ w\end{array}\right]$$

It could also be written like this

$$\tilde{x}=Hx$$

It is required that

1. H needs to be invertible
2. H has 8 Degrees-of-Freedom (DoF)

To apply the transformation, we first transform the coordinates to homogeneous coordinates. Then we use homography on the coordinates. Then we transform the coordinates back by de-homogenzing them.

General intrinsic camera calibration matrix

$$K=\left[\begin{array}{ccc}f& s& p_x\\ 0& \alpha f& p_y\\ 0& 0& 1\end{array}\right]$$

Relative pose estimation

Epipolar geometry

The essential matrix

The fundamental matrix