AutoStitch

Finding a good image match indicator
After I got the main program working, I wanted to try and create an autostiching program. First, I wanted to see if there was some simple calculation that I could do during the LucasKanade step that would be a good indication of whether two images were a good match or not. The first thing I tried was computing an average of It (the difference between the intensities of the two images) over all the pixels. I called this value ItAvg. My reasoning was that images that match should have pixels with about the same intensity at the same position. The following are the computed ItAvg values for all possible pairs of the four test images:



As you can see, it turns out ItAvg is not a good indication of whether two images are a good match or not. The three image pairs that should match (08-09, 09-10, 10-11) do not have ItAvg values that really stand out from the others. I believe this is because even if two images are a good match, exposure differences can cause one image to be generally lighter or darker than the other image. So ItAvg is more a measure of the exposure differences between two images. But even if two images have different exposure levels, if the images match, then the intensity difference should pretty much be constant from pixel to pixel. This led me to try computing the square of the difference between It and the current ItAvg for each pixel, and taking the average of this value over all pixels. I called this value ItDev. The following are the computed ItDev values for all possible pairs of the test images:



As you can see, ItDev is a much better indication of whether two images match or not. The three image pairs that should match all have much lower ItDev values than the other pairs. This is because in image pairs that match, cooresponding pixels have intensity values that differ by an almost constant amount, while in image pairs that do not match, the pixel intensities will differ by an almost random amount.

Implementaion
Now that I have a good match indicator, creating an autostitch program is simply a matter of implementation details. I decided to write it in Java, since I have more experience with it. The program will look in a given folder for all the tga images, apply cylWarp with given focal length and k1 and k2 radial distortion constants, order the images into one or more panoramas and generate the cooresponding pairlist files, and then stitch the panoramas together. As you can see, most of the work is delegated to the v4gp1 program. To keep it simple, all the images are assumed to have the same focal length and k1 and k2 constants and also about the same horizontal displacement.

To order the images into panoramas, the program starts with a sequence of images that form part of the panorama and a pool of unordered images, and tries to "grow" the panorama by placing one of the unordered images on either the left or the right side on the partial panorama. It chooses the image to add by calculating the ItDev values for all the possible left and right image pairs and then choosing the image with the lowest ItDev. However, if all the image pairs have ItDev values that are above some given cutoff value, then it stops growing the current panorama and starts growing a new panorama with one of the remaining images in the unordered image pool.

Usage
The program runs from the command line and takes 6 arguments:

java AutoStitch folder f k1 k2 h_disp cutoff

folder is the absolute path to a folder that contains the tga images. Note that due to the way v4gp1 parses the pairlist file, the path to the folder and the image files cannot have any spaces in their names.
f k1 k2 are the camera focal length and k1 and k2 radial distortion constants.
h_disp is the initial guess of the horizontal displacement. It is also used as the blend width.
cutoff is a value that determines how many different panoramas are created. A low cutoff value will cause each image to be its own panorama. A high cutoff value will place all images in a single panorama, even if they should actually belong to different panoramas. A good value for this is between 500 and 1000.

Results
To test the program I took 4 images from each of 3 different panoramas along with another image and put them all into a folder. The program correctly managed to generate the 3 panoramas when I used a cutoff value of 600.

Input images:


Output images:


Although it looks like only 2 images got stitched into each output panorama, in fact all 4 of the images were correctly stitched together for each panorama. The problem is that v4gp1 crops away the left and right sides of the generated panorama since it is meant to be used to stitch complete 360 degree panoramas. When stitching 360 degree panoramas, the first and last images will be the same, so the panorama is cropped so the left and right edges fit together. When building panoramas that are not the full 360 degrees, it would be better not to do any cropping so more of the panorama can be seen.

As a second test, I tried running autostitch on all of the images from the 3 panorama sequences all together (62 images total). Together, they should form 3 complete panoramas. I tried running autostitch with different cutoff values to see how this would affect the results. Here are the 62 input images:


Here are the results of running autostitch with a cutoff value of 600: (Autostitch also generated several smaller panoramas, which are not shown)


Here are the results of running autostitch with a cutoff value of 1000:


Here are the results of running autostitch with a cutoff value of 1600:


When the cutoff value was 600, autostitch generated several panoramas that only contained 1 or 2 images (not shown), causing the longer panoramas to contain less images. Most noticably, the dorm room sequence got split up into 2 smaller panoramas. When the cutoff value was 1000, the 3 panoramas were correctly generated, with no stitching together of images from different panoramas. However, some images were still not included in the panoramas, as you can see since the result panoramas are not the full 360 degree panoramas. Increasing the cutoff value to 1600 did not help to include more of the images in each panorama. The same dorm room and red square sequences were generated when the cutoff was 1600 as when the cutoff was 1000. In fact, setting the cutoff to 1600 actually caused one of the dorm room images to be incorrectly blended into the lounge sequence, as you can see if you look carefully at the right side of the lounge sequence.

As you can see from the results, when the cutoff value is too low, autostitch is more likely to give up on growing the current panorama. So there will be more panoramas generated than there should be. On the other hand, when the cutoff value is too high, autostitch is less likely to give up on growing the current panorama. So images from different panoramas may actually be stitched together. The trick is to find the cutoff value that returns the best result. Of course, a cutoff value may not always exist that gives the correct result. This just goes to show that ItDev information alone is not enough to do perfect automatic panorama stitching.