CS 5670: Computer Vision, Spring 2020
Project 3:  Autostitch

[Additional hints] 1) When working with homogeneous coordinates, don't forget to normalize when converting them back to Cartesian coordinates. 2) Watch out for black pixels in the source image when inverse warping. You don't want to include them in the accumulation. 3) When doing inverse warping, use linear interpolation for the source image pixels. 4) First try to work out the code by looping over each pixel. Later you can optimize your code using array instructions and numpy tricks (numpy.meshgrid, cv2.remap). You are not required to do this optimization.

[TODO 11] Normalize the image with the accumulated weight channel. Pay attention not to divide by zero. Remember to set the alpha channel of the resulting panorama to opaque! (normalizeBlend)

[TODO 12] In case of 360 degree panoramas, make the left and right edges have perfect seams. The horizontal extent can be computed in the previous blending routine since the first image occurs at both the left and right end of the stitched sequence (draw the "cut" line halfway through this image).  Use a linear warp to the mosaic to remove any vertical "drift" between the first and last image.  This warp, of the form y' = y + ax, should transform the y coordinates of the mosaic such that the first image has the same y-coordinate on both the left and right end.  Calculate the value of 'a' needed to perform this transformation. (blendImages)

Extra Credit

Please see Extra Credit to know where to implement this functionality.

Using the GUI

The skeleton code that we provide comes with a graphical interface, with the module gui.py, which makes it easy for you to do the following:

1. Visualize a Homography: The first tab in the UI provides you a way to load an image and apply an arbitrary homography to the image. This can be useful while debugging when, for example, you want to visualize the results of both manually and programmatically generated transformation matrices. Note that you do not need to implement any functions to run this.
2. Visualize Spherical Warping: The second tab on the UI lets you spherically warp an image with a given focal length.
3. Align Images: The third tab lets you select two images with overlap and uses RANSAC to compute a homography or translation (selectable) that maps the right image onto the left image.
4. Generating a Panorama: The last tab in the UI lets you generate a panorama. To be able to create a panorama, you need to have a folder with images labeled in such an order that sorting them alphabetically gives you the order the images appear on the panorama from left to right (or from right to left). This ensures that the mappings between all neighboring pairs are computed. Our current code assumes that all images in the panorama have the same width!

Debugging Guidelines

The test.py provides very basic testcases for TODO 1-9. You need to test TODO 10-12 and any extra credit parts with your own testcases.

The supplied tests are very simple and are meant as an aid to help you get started. Passing the supplied test cases does not mean the graded test cases will be passed.

You can also use the GUI visualizations to check whether your program is running correctly.

1. Testing the warping routines:

• In the campus test set, the camera parameters used for these examples are
  • f = 595
  • k1 = -0.15
  • k2 = 0.00
• In the yosemite test set, a few example warped images are provided for test purposes. The camera parameters used for these examples are
  • f = 678
  • k1 = -0.21
  • k2 = 0.26
  See if your program produces the same output. Note that if you use these images with the translation motion model, you might get a bit blurry panoramas in the blending region (as you can also see from the panorama given by us).

2. Testing the alignment routines:

• Note that the campus images are only suitable for the translational motion model! The yosemite images are suitable for both motion models. To test alignPair, load two images in the alignment tab of the GUI. Clicking 'Align Images', displays a pair, the left and right images, with the right image transformed according to the inter-image transformation matrix and overlaid over the left image. This enables visually analyzing the accuracy of the transformation matrix. Note that blending is not performed at this stage.

3. Testing the blending routines:

• An example panorama is included in the yosemite and the campus test set. Compare the resulting panorama with these images. Note that it's important to use the specified f, k1, k2 parameters to get the same image. Use 360 degree panorama to get the same result for campus dataset!

4. Additional notes:

• If you use high resolution images when creating you own panorama, you might run into memory problems. Increasing the allowed memory of the VM might solve these issues.

What to Turn In

First, your source code should be zipped up into an archive called 'code.zip', and uploaded to CMS. In addition, turn in a panorama as JPG as your artifact. In particular, turn in a panorama from a hand-held sequence. This panorama can be either translation-aligned (360 or not), or aligned with homographies (your choice).

Taking Pictures

Take a series of images with a digital camera mounted on a tripod or a handheld camera. For best results, overlap each image by 50% with the previous one, and keep the camera level. You can use your own camera for this. Some cameras have a "stitch assist" mode you can use to overlap your images correctly, which only works in regular landscape mode.  In order to use your camera, you have to estimate the focal length.  The simplest way to do this is through the EXIF tags of the images, as described here.  Alternatively, you can use a camera calibration toolkit to get more precise focal length and radial distortion coefficients.

Downloads

Skeleton code: Available through Github

Test sets: Look inside the resources subdirectory. You will find three datasets: yosemite, campus and melbourne.

Use Python3 for this project.

cs5670_python_env: Tutorial on how to set up cs5670_python_env