Day 78 to 79 – Make my first iPhone app in 16 weeks (Perspective Transform)

This is part of “Make my first iPhone app in 16 weeks” series.

Tristimulus value (RGB color) breakdown

  1. Intensity (brightness) – scalar value
  2. Chromaticity (color) – 2-vector (Red and Green. Blue is thrown away by convention as Blue can be deduced by knowing Red and Green) – R / (R + G + B) + G / (R + G + B) + B / (R + G + B) = 1

Homography matrix – used in warping the image

White paper on how to  estimate the ratio of the paper used -here (Someone else’s implementation)

Different transform explained here

 

 

 

Day 74 to 77 – Make my first iPhone app in 16 weeks (Detecting Corners)

This is part of “Make my first iPhone app in 16 weeks” series.

Functions to find corners:

  1. cornerHarris – most effective edge detector
  2. goodFeaturesToTrack – only finds the most prominent corners using cornerMinEigenVal or cornerHarris
  3. cornerSubPix – refines corners into sub-pixel level
  4. cornerMinEigenVal & cornerEigenValsAndVecs – to make custom corner detector

Day 72 – Make my first iPhone app in 16 weeks (OpenCV Feature Detection)

This is part of “Make my first iPhone app in 16 weeks” series.

FEATURE DETECTION

Harris Corner

1. Use cornerHarris() function to detect corners

cornerHarris(grayscale source image, OutputArray dst, int blockSize, int ksize, double k, int borderType=BORDER_DEFAULT )
  1. src – Input single-channel 8-bit or floating-point image.
  2. dst – Image to store the Harris detector responses. It has the type CV_32FC1 and the same size as src .
  3. blockSize – Neighborhood size (see the details on cornerEigenValsAndVecs() ).
  4. ksize – Aperture parameter for the Sobel() operator.
  5. k – Harris detector free parameter. See the formula below.
  6. borderType – Pixel extrapolation method. See borderInterpolate()

2. Use normalize() and convertScaleAbs() functions to normalize it

normalize( dst, dst_norm, 0, 255, NORM_MINMAX, CV_32FC1, Mat() );
convertScaleAbs( dst_norm, dst_norm_scaled );

Shi-Tomasi corner detector

Use goodFeaturesToTrack() function to detect corners

/// Apply corner detection
  goodFeaturesToTrack( grayscale source image,
               destination corner image,
               maxCorners,
               qualityLevel,
               minDistance,
               Mat(),
               blockSize,
               useHarrisDetector,
               k );

Create your own corner detection

Use cornerEigenValsAndVecs() function to get Eigen Value and Eigen Vector for Harris Corner detection

cornerEigenValsAndVecs( src_gray, myHarris_dst, blockSize, apertureSize, BORDER_DEFAULT );

Use cornerMinEigenVal() function to get minimum Eigen Value for Shi-Tomasi Corner detection

 cornerMinEigenVal( src_gray, myShiTomasi_dst, blockSize, apertureSize, BORDER_DEFAULT );

Detecting corners location in subpixeles

Use cornerSubPix() function to get more precise corner location (more exact than integer pixel)

cornerSubPix( src_gray, corners, winSize, zeroZone, criteria );

Feature Detection

Use FeatureDetector class and its detect() method to find key features

int minHessian = 400;
SurfFeatureDetector detector( minHessian );
std::vector keypoints_1, keypoints_2;
detector.detect( img_1, keypoints_1 );
detector.detect( img_2, keypoints_2 );

Mat img_keypoints_1; Mat img_keypoints_2;
drawKeypoints( img_1, keypoints_1, img_keypoints_1, Scalar::all(-1), DrawMatchesFlags::DEFAULT );
drawKeypoints( img_2, keypoints_2, img_keypoints_2, Scalar::all(-1), DrawMatchesFlags::DEFAULT );

Feature Description

1. Use SurfDescriptorExtractor class and its method compute() to find the feature vector correspondent to the key points (assuming you have key points detected by SurfFeatureDetector class)

SurfDescriptorExtractor extractor;
Mat descriptors_1, descriptors_2;
extractor.compute( img_1, keypoints_1, descriptors_1 );
extractor.compute( img_2, keypoints_2, descriptors_2 );

2. Use BFMatcher class and its method match() to match the key descriptors

BFMatcher matcher(NORM_L2);
std::vector< DMatch > matches;
matcher.match( descriptors_1, descriptors_2, matches );

3. Use drawMatches() to draw matches

Mat img_matches;
drawMatches( img_1, keypoints_1, img_2, keypoints_2, matches, img_matches );

Feature Matching with FLANN

1. Use SurfFeatureDetector class to detect keypoints

int minHessian = 400;
SurfFeatureDetector detector( minHessian );
std::vector keypoints_1, keypoints_2;
detector.detect( img_1, keypoints_1 );
detector.detect( img_2, keypoints_2 );

2. Compute descriptors

SurfDescriptorExtractor extractor;
Mat descriptors_1, descriptors_2;
extractor.compute( img_1, keypoints_1, descriptors_1 );
extractor.compute( img_2, keypoints_2, descriptors_2 );

3. Use FlannBasedMatcher class to match features

FlannBasedMatcher matcher;
std::vector< DMatch > matches;
matcher.match( descriptors_1, descriptors_2, matches );

Homography

Use findHomography() function to find the transform between matched keypoints

Mat H = findHomography( obj, scene, CV_RANSAC );

Use perspectiveTransform() function to map the points

perspectiveTransform( obj_corners, scene_corners, H);

Day 71 – Make my first iPhone app in 16 weeks (OpenCV Moments)

This is part of “Make my first iPhone app in 16 weeks” series.

Moments

An image moment is a certain particular weighted average (moment) of the image pixels’ intensities, or a function of such moments, usually chosen to have some attractive property or interpretation. (from wikipedia.org)

1. Use moments() function to get moments

vector mu(contours.size() );
  for( int i = 0; i < contours.size(); i++ )
     { mu[i] = moments( contours[i], false ); }

2. Use moments() function to get moments

vector mc( contours.size() );
  for( int i = 0; i < contours.size(); i++ )
     { mc[i] = Point2f( mu[i].m10/mu[i].m00 , mu[i].m01/mu[i].m00 ); }

contourArea() function calculates the area of a shape
arcLength() function calculates the perimeter of a shape

Point Polygon Test

The function determines whether the point is inside a contour, outside, or lies on an edge (or coincides with a vertex). It returns positive (inside), negative (outside), or zero (on an edge) value, correspondingly. When measureDist=false , the return value is +1, -1, and 0, respectively. Otherwise, the return value is a signed distance between the point and the nearest contour edge.

1. Use pointPolygonTest() function to get distance to the contour

for( int j = 0; j < src.rows; j++ )
     { for( int i = 0; i < src.cols; i++ )
          { raw_dist.at(j,i) = pointPolygonTest( contours[0], Point2f(i,j), true ); }
     }

Day 70 – Make my first iPhone app in 16 weeks (OpenCV)

This is part of “Make my first iPhone app in 16 weeks” series.

Remap – Translation

This allows a pixel in an image to be remapped to another location in the image.

remap( source image, 
       destination image, 
       map_x, // The mapping function in the x direction
       map_y, // The mapping function in the y direction
       CV_INTER_LINEAR, // The type of interpolation to use for non-integer pixels. This is default.
       BORDER_CONSTANT, // Border type. This is default.
       Scalar(0,0, 0) 
);

Affine Transformation

Affine transformation is linear transformations (rotation, shearing, reflection, contraction, dilation) + translation

Affine Transformation = Linear Transformation Matrix * Coordinate Vector + Translation (View this pdf)

1. Create 3 points on a triangle before and after the transformation you want

srcTri[0] = Point2f( 0,0 );
srcTri[1] = Point2f( src.cols - 1, 0 );
srcTri[2] = Point2f( 0, src.rows - 1 );

dstTri[0] = Point2f( src.cols*0.0, src.rows*0.33 );
dstTri[1] = Point2f( src.cols*0.85, src.rows*0.25 );
dstTri[2] = Point2f( src.cols*0.15, src.rows*0.7 );

2. Use getAffineTransform() function to get affine transformation matrix

warp_matrix = getAffineTransform( srcTri, dstTri );

3. Use warpAffine() function to transform the image

warpAffine( source image, destination image, warp_matrix, destination image.size() );

4. Define center point, angle and scale for rotation

Point center = Point( warp_dst.cols/2, warp_dst.rows/2 );
double angle = -50.0;
double scale = 0.6;

5. Use getRotationMatrix2D() function to rotate the image

rot_mat = getRotationMatrix2D( center, angle, scale );

6. Use warpAffine() function to transform the image

warpAffine( source image, destination image, warp_matrix, destination image.size() );

Continue reading

Day 69 – Make my first iPhone app in 16 weeks (OpenCV Edge & Line Detection)

This is part of “Make my first iPhone app in 16 weeks” series.

Erosion and Dilation

  1. Erosion – Convolving the kernel over the image and replacing the pixel with the local maximum. It makes the brighter area dilated.
  2. Erosion – Convolving the kernel over the image and replacing the pixel with the local minimum. It makes the brighter area eroded (darker part gets dilated)

Erosion

Build Kernel

Mat kern = getStructuringElement(morph_type,
                                 Size(2*erosion_size + 1, 2*erosion_size+1),
                                 Point(erosion_size, erosion_size));

Use erode() function

 erode( src image, destination image, kern );

Dilation

Build Kernel

Mat kern = getStructuringElement(morph_type,
                                 Size(2*dilation_size + 1, 2*dilation_size+1),
                                 Point(dilation_size, dilation_size));

Use dilate() function

 dilate( src image, destination image, kern );

** There are 3 morph types

  1. Rectangular box: MORPH_RECT
  2. Cross: MORPH_CROSS
  3. Ellipse: MORPH_ELLIPSE

morphologyEx

Operations

  1. opening – erode and dilate
  2. closing – dilate and erode
  3. morphological gradient – dilation – erosion
  4. top hat – original image – opening
  5. black hat – closing – original image

Use morphologyEx() function

morphologyEx( src, dst, operation, element );

Image Pyramids

Resizing images – upsampling or downsampling

pyrUp( current image, destination image, Size( tmp.cols*2, tmp.rows*2 )
pyrDown( current image, destination image, Size( tmp.cols/2, tmp.rows/2 )

Threshold

Types of thresholding operations

  1. Binary – if the intensity of the pixel is higher than threshold, it’s set to max value and if it’s lower, it’s set to zero
  2. Binary, Inverted – if the intensity of the pixel is higher than threshold, it’s set to zero and if it’s lower, it’s set to max value
  3. Truncate – if the intensity of the pixel is higher than threshold, it’s set to threshold and if it’s lower, nothing happens
  4. Threshold to Zero – if the intensity of the pixel is higher than threshold, nothing happens and if it’s lower, it’s set to zero
  5. Threshold to Zero, Inverted – if the intensity of the pixel is higher than threshold, it’s set to zero. If it’s lower, nothing happens.
threshold( grayscale original image, destination image, threshold_value, max_BINARY_value, type of thresholding operation );

Linear Filters

Initialize the arguments for the linear filter

anchor = Point( -1, -1 ); // anchor of kernel
delta = 0; // A value to be added to each pixel during the convolution.
ddepth = -1; // depth of destination

Define kernel

kernel_size = 3 + 2*( ind%5 );
kernel = Mat::ones( kernel_size, kernel_size, CV_32F )/ (float)(kernel_size*kernel_size);

Use filter2D() function

filter2D(source image, destination image, ddepth , kernel, anchor, delta, BORDER_DEFAULT );

Make Borders

Border types

  1. BORDER_CONSTANT – a set color for the border
  2. BORDER_REPLICATE – copies over the row or column at the very edge of the original to the extra border.

Define the thickness of top, bottom, left and right borders and the color of the border (if BORDER_CONSTANT)

Use copyMakeBorder() function

copyMakeBorder( src, dst, top, bottom, left, right, borderType, value );

EDGE DETECTION

Sobel and Scharr – Edge detection using first derivative

Sobel

Sobel( grayscale source image, 
       destination x derivative image, 
       ddepth, 
       1, 
       0, 
       3, 
       scale, 
       delta, 
       BORDER_DEFAULT );
convertScaleAbs( grad_x, 
                 abs_grad_x );
Sobel( grayscale source image, 
       destination y derivative image, 
       ddepth, 
       0, 
       1, 
       3, 
       scale, 
       delta, 
       BORDER_DEFAULT );
convertScaleAbs( grad_y, 
                 abs_grad_y );

Scharr – More accurate than Sobel

Scharr( grayscale source image, 
        destination x derivative image, 
        ddepth, 
        1, 
        0, 
        scale, 
        delta, 
        BORDER_DEFAULT ); //ddepth is CV_16S here
convertScaleAbs( destination x derivative image, 
                 abs_grad_x ); // change back to CV_8U
Scharr( grayscale source image, 
        destination y derivative image, 
        ddepth, 
        0, 
        1, 
        scale, 
        delta, 
        BORDER_DEFAULT ); //ddepth is CV_16S here
convertScaleAbs( destination y derivative image, 
                 abs_grad_y ); // change back to CV_8U
/// Approximate total gradient
addWeighted( abs_grad_x, 
             0.5, 
             abs_grad_y, 
             0.5, 
             0, 
             destination image );

Laplace – Edge detection using second derivative

Laplacian( grayscale source image, 
           destination image, 
           ddepth, // CV_16S
           kernel_size, 
           scale, 
           delta, 
           BORDER_DEFAULT );
  convertScaleAbs( dst, 
                   abs_dst );

Canny – Optimal Edge detection using Hysteresis

Canny( blurred image, destination image with edges, lowThreshold, highThreshold, kernel_size );

LINE & SHAPE DETECTION

Standard Hough Line Transform – Straight line

1. First apply Canny

2. Use HoughLines()

vector lines;
HoughLines( resulting grayscale image from Canny, 
            lines, 
            rho, // The resolution of the parameter r in pixels. We use 1 pixel.
            theta, // The resolution of the parameter \theta in radians. We use 1 degree (CV_PI/180)
            threshold, // The minimum number of intersections to “detect” a line
            srn, // Default is zero
            stn // Default is zero
);

Probabilistic Hough Line Transform – Straight line

1. First apply Canny

2. Use HoughLinesP()

vector lines;
HoughLinesP( resulting grayscale image from Canny, 
            lines, 
            rho, // The resolution of the parameter r in pixels. We use 1 pixel.
            theta, // The resolution of the parameter \theta in radians. We use 1 degree (CV_PI/180)
            threshold, // The minimum number of intersections to “detect” a line
            minLineLength, // The minimum number of points that can form a line. Lines with less than this number of points are disregarded.
            maxLineLength // The maximum gap between two points to be considered in the same line.
);

Hough Circle Transform

Mostly the same as straight line detection.

HoughCircles( grayscale source image, 
              destination images, 
              CV_HOUGH_GRADIENT, 
              inverse ratio of resolution, // 1
              min_dist between detected centers, // src_gray.rows/8
              upper threshold for the internal Canny edge detector, 
              threshold for center detection, 
              minimum radius to be detected, // if unknown, put zero
              maximum radius to be detected, // if unknown, put zero
);

Then draw detected circles.

Remapping

Use remap() function

Day 68 – Make my first iPhone app in 16 weeks (OpenCV Documentation)

This is part of “Make my first iPhone app in 16 weeks” series.

Read this documentation first – here

Mat constructor

1. Mat M(row count, column count, CV_[The number of bits per item][Signed or Unsigned][Type Prefix]C[The channel number], scalar );

Mat Constructor

Mat Constructor

2. Mat C = (Mat_(3,3) << 0, -1, 0, -1, 5, -1, 0, -1, 0); // only works in 2-dimensional matrix

Mat Methods

  1. Mat.create()
  2. Mat.clone()
  3. Mat.copyTo()
  4. Mat.isContinuous() – checks if the elements in the matrix are stored continuously without gaps at the end of each row.
  5. Mat.at(row index, col index) – returns a reference to a specific array element
  6. Mat.convertTo(new_image, -1, alpha, beta)

Mat Members

  1. Mat.data – pointer to the first row, first column

Functions that modify Mat objects

  1. LUT(input image, lookup table, output image)
  2. randu(Mat obj, from scalar value, to scalar value) – fill out a matrix with random values
  3. filter2D(Input, output, I.depth(), kernel in matrix) – does convolution

How to measure time taken to perform a function

double t = (double)getTickCount();
// do something ...
t = ((double)getTickCount() - t)/getTickFrequency();
cout << "Times passed in seconds: " << t << endl;

When iterating through each pixel of an image, the most efficient way to do so is using pointer arithmetic of C to go through each element of the matrix. Here is some tutorial on pointer arithmetic.

** Note, if you pass in an array into a function, C automatically changes it to a pointer to the array. Therefore, if you need the size of the array as well, you have to pass the size into the function as a parameter as well. If you try to get the size of the array in the function, it will only give you the size of the pointer to the array.

Kernel and Convolution

Kernel – a fixed size array of numerical coefficients along with an anchor point in that array, which is typically located at the center.

Convolution – operation between every part of an image and an operator (kernel)

What is Mat_?

  • Mat_<destination_type>() constructors to cast the result to the proper type.
Mat kern = (Mat_(3,3) <char> <<  0, -1,  0,
                                -1,  5, -1,
                                 0, -1,  0);

Point and Scalar

  • Point is a position of an element in a matrix
  • Scalar is a 4-element vector that is used to represent a pixel

RNG (Random Number Generator)

  • RNG.uniform(a, b) – Give a randomly uniform distributed number between value a an b

Discrete Fourier Transformation

function to use is dft()

tutorial on Fourier Transformation (here)

 

Day 67 – Make my first iPhone app in 16 weeks (C++ Class Templates)

This is part of “Make my first iPhone app in 16 weeks” series.

Class Templates allow you to defer specifying the data type the class will handle until the instantiation. This makes the  class more abstract and reusable.

Declaration of template class

template <class data_type> class class_name {...};

All member functions of the template class must be template functions

template void class_name<data_type>::function_name(){...}

Explicit instantiation of the template class – telling the compiler to compile the template class with a specific data type so that the class can be used.

template class_name<specific_data_type>

Specialization of the template class – Defining a special override for a specific data type

template<> class_name<specific_data_type>;

Implicit instantiation the template class – true instantiation.

class_name<specific_data_type> class_instance_name

Function Templates

Declaration of the function template

template <typename data_type> void func_name(data_type param) {}

Explicit instantiation of the function template

template void func_name<specific_data_type>(specific_data_type param);

Specialization of the function template

template<> void func_name(specific_data_type a, specific_data_type b);

Implicit instantiation of the function template

func_name<specific_data_type>(param)

Some tutorial on C++ function template is here and here

OpenCV CascadeClassifier – Multi-stage ensemble learning to detect objects

  • CascadeClassifier.load( <xml file to detect whatever you want to detect> )
  • Convert the image to gray scale
  • Create vectors with data type <Rect> to represent faces found
    • A vector is an one-dimensional array which allows fast random access (More on vector)
  • CascadeClassifier.detectMultiScale(const Mat& image, vector<Rect>& objects, double scaleFactor=1.1, int minNeighbors=3, int flags=0, Size minSize=Size(), Size maxSize=Size())
  • Draw the rectangles that you get in return from detectMultiScale function

See this tutorial to know how to train your own Haar Classifier

const in C++

The keyword const serves two purposes: documentation as well as safety. Const reminds you and the users of your code that something should not change. You can make the following “const”

  • Variable
    • Both const int x = 5; and int const x = 5; work
  • Reference pointer
    • const int *x; // the pointer x points to a constant integer. Can change pointer but cannot change what x points to.
    • int * const x; // the constant pointer x points to an integer. Cannot change pointer and therefore cannot change what x points to either.
  • Function
    • Const functions are the only functions that can be called on a const object.
    • Only member methods of a class make sense to be const
    • If a const function returns a member of a class, that member must also be const
    • int funcName() const;

const in objective C

You can use #define or use the following methods.

// header file
FOUNDATION_EXPORT NSString *const MyFirstConstant;
// implementation file
NSString *const MyFirstConstant = @"FirstConstant";

Day 66 – Make my first iPhone app in 16 weeks (CGDataProvider)

This is part of “Make my first iPhone app in 16 weeks” series.

CGDataProvider

CGDataProvider

UIImage (part of UIKit) vs. CGImage (part of Application Services) vs. CIImage (part of Core Image framework)

  • UIImage is a high-level way to represent an image.
  • CGImage represents a bitmap image. Used to do some serious manipulation of the image.
  • CIImage represents an image, which, in conjunction with other Core Image objects, can take advantage of built-in filters.

*** NSImage is for OS X only

OpenCV

  • cv::cvtColor(InputArray src, OutputArray dst, int code, int dstCn=0)
    • converts an image from one color space to another. (For example, from RGB to GrayScale)
  • cv::GaussianBlur(InputArray src, OutputArray dst, Size ksize, double sigmaX, double sigmaY=0, int borderType=BORDER_DEFAULT)
    • Good explanation on GaussianBlur (here)
    • Both kSize, sigmaX and sigmaY have to increase in value to make the blurring effect magnified.
  • cv::Canny(InputArray image, OutputArray edges, double threshold1, double threshold2, int apertureSize=3, bool L2gradient=false)
    • Good explanation on Canny algorithm (here)
    • The resulting image has edges in white on black background
  • cv::setTo() – very good StackOverflow answer
    • if pass a scalar parameter only, every value in the matrix is set to that scalar value
    • if pass a scalar parameter and a matrix, set to the scalar where the value is 1 in the matrix parameter

Day 62 to 65 – Make my first iPhone app in 16 weeks (C++ Crash Course)

This is part of “Make my first iPhone app in 16 weeks” series.

I finished reading C++ Tutorial book

Some facts about C++ and Objective C

  • Both C++ and Objective C were born in the same year of 1983
  • C++ was created by Stroustrup and Objective C by Brad Cox
  • C++ was originally named C with Classes as it added object-oriented features to C
  • Objective C was inspired by Smalltalk and made to be backward compatible with C

Where C++ differs from Objective C

  • C++ supports multiple inheritance and Objective C does not (or chose not to)
  • C++ allows a data member to be “static” and Objective C does not (there is a way to emulate the same effect but pretty janky)
  • In C++ definition, an abstract class is the class that has one or more “pure virtual” functions (cannot be instantiated). In Objective C definition, an abstract class is by convention only.
  • C++ allows operator overloading and Objective C does not