CS 4670 Introduction to Computer Vision (Fall 2010)

Project 2b: Object Matching and Panorama Stitching

Released: Monday, Oct. 18, 2010
Due: Tuesday, Nov. 2, 2010, by 8:59pm
Last Modified: Wednesday, Oct. 20, 2010

Quick Link: Overview Description To Do Running and Testing Phone Checkout Submission Bells and Whistles

Overview

In this two-part project, you will write code to do (a) object matching and (b) panorama stitching on the Nokia N900.

Description

Part I. Object Matching

In this part, you will reuse the code you implemented in Project 2a. You will take a picture and extract its features. You will need to estimate a homography between the database image and the search image using RANSAC, and return the best matched image. You will also write a new feature matcher that uses new data structures to do faster nearest-neighbor matching. Using our provided image database, you will get the following result.



Here are some screenshots on the cell phone.

Part II. Panorama Stitching

In this part you will shoot two photos and make them into a panorama by applying the homography. You will need to implement RANSAC to evaluate feature matches. You will also reuse the homography code to align two images, then use alpha blending and feathering to stitch the images together. Here's a screenshot doing panorama stitching on the cell phone.

Click here to download the skeleton code.

To Do

The only file that you will need to edit is Features.cpp. Where appropriate, you may copy in code from Project 2A if you choose. You are encouraged to use the SURF features (feature type 3 in this code), although you can also try your MOPS features from Project 2A. Here's a list of functions you need to implement.

• void ratioMatchFeatures(FeatureSet &f1, FeatureSet &f2, vector &matches, float &totalScore, Progressable *thread)
This function uses the ratio of the SSD distance of the two best matches and matches a feature in the first image with the closest feature in the second image. It can match multiple features in the first image to the same feature in the second image. Most of this function is implemented for you, but you need to call the FLANN library in order to find approximate nearest neighbors.


• CvMat* computeHomography(vector &f1, vector &f2, vector &matches, CvMat *h)
This function will take in two sets of features, and a list of matches, and return a homography computed using least squares (with the direct linear transformation approximation). Probably the best way to solve this is to use the singular value decomposition, in OpenCV as cvSVD cvSVD, passing in the 2*n by 9 A you compute as part of the least squares problem for the homography. cvSVD returns three output matrices, W (a 2n x 9 matrix), U (a 2n x 2n matrix), and V (a 9 x 9 matrix). This page recommends also giving cvSVD the flags CV_SVD_U_T|CV_SVD_V_T as a fifth option (in which case the transposes of U and V would be returned). The homography is the last row or column of the returned V matrix, depending on whether the CV_SVD_V_T option is set. You may alternately use cvSolve (CV_SVD option) useful for solving the least squares problem. The memory for the output matrix h is allocated by this caller. Note that when calling this function, you will likely want to create a temporary sublist of four or more matches (i.e., you will never pass in the full list of matches, but will construct a subset of matches).


• CvMat* ransacHomography(vector &f1, vector &f2, vector &matches, int numRounds, int inlierThreshold, Progressable *thread)
You will implement RANSAC as described in lecture. This function takes in two sets of features, a list of matches, and returns a 3 × 3 homography matrix, and runs RANSAC with "numRounds" rounds, with an inlier threshold of "inlierThreshold." This function will call computeHomography (which expects a preallocated 3 × 3 matrix of type CV_32F), as well as applyHomography.


• IplImage* constructPanorama(IplImage *img1, IplImage *img2, int featureType, int matchType, Progressable *thread)
This is the entry point for all panorama generation. The output image will be allocated by your code and in particular should be allocated from a call to compositeImages. This function will also depend on ransacHomography in order to compute a best homograph for the pair of images. You should use computeFeatures and matchFeatures when necessary.


• IplImage* compositeImages(IplImage *img1, IplImage *img2, CvMat *h, Progressable *thread)
img1 and img2 are color images that you want to make into a panorama by applying the homography h, to img2. This function needs to determine the size of the output image and allocate the memory for it. You need to implement feathering and alpha blending in your own compositing routine.

Running and testing your code

Part I. Object Matching

After writing your code, you should follow the document to compile it. We provide some testing images, which can be downloaded here. We also provide test images for cell phones. There are instructions for building your own image database inside the tarball. You can use this as your image database and test your code in Vision Lab. This database feature is accessible on the desktop version through a new menu item in the GUI used in Part A of the project.

Part II. Panorama Stitching

You should test your panorama stitcher on these two Yosemite images. Please include your panorama create from this pair in your writeup (see below).

If you would like a larger set to test on, try out this larger set of Yosemite images.

Phone Checkout

First you need to sign up for phone checkout. You are given 5 days for this project. There is a sign up sheet in front of Upson 4144. In general, there are two slots available in a week, Friday to Wednesday and Wednesday to Monday. By Friday, October 22, you should signup for one slot. There's an emergency slot from Monday, 11/1 - Tuesday, 11/2. If you need additional time, you can signup for that slot (not open yet), priority will be given to the team who signs up for Friday to Wednesday slot.

Submission

You should turn in their code, as well as an artifact webpage. The webpage should contain the results of the object matching on the N900, panorama results on our test images, and at least two panoramas that you create yourself, on the N900. Compress the webpage into a zip file called webpage.zip and compress your source codes into code.zip, and upload them to CMS. In addition, you will need to demo the result to the TAs.

Bells and Whistles

Here is a list of suggestions for extending the program for extra credit. You are encouraged to come up with your own extensions as well!

• Speed up your object recognition code to run faster than 0.5 seconds on the phone.
• Recognize multiple objects in a single image.
• Implement spherical warping instead of planar perspective warping.
• Calibrate the N900 camera to remove radial distortion.
• Construct a panorama with more than two images.
• Construct a full 360 panorama.
• Use graph cuts to composite the panorama.
• Use gradient-domain blending to create a more seamless composite image.

Monster Bells

Disclaimer: please consult the course staff before spending any serious time on these. They are quite difficult, and credit can vary depending on the quality of your method and implementation.

Web-scale object matching

Real-time panorama creation