1. Motion Models (Registration for two Images)
   1. Planner Perspective Motion
      1. Only translate and rotate in 2D
      2. Application: photo and video stabilization and merging
   2. Rotational Panoramas
      1. When the camera undergoes a pure rotation
      2. Gap Closing
   3. Cylindrical Coordinates
      1. Align images is to first warp the images into cylindrical coordinates and then use a pure translational model to align them.
      2. This only works if the images are all taken with a level camera or with a known tilt angle.
   4. Spherical Coordinates
      1. It is useful if the final panorama includes a full sphere or hemisphere of views, instead of just a cylindrical strip.
2. Global Alignment (Registration for All Images)
   1. Goal: To find a globally consistent set of alignment parameters that minimize the mis-registration between all pairs of images
   2. Method 1: Add new images to the panorama one at a time, aligning the most recent image with the previous ones already in the collection and discovering, if necessary, which images it overlaps.
      1. Problem: Accumulated error may lead to the presence of a gap (or excessive overlap) between the two ends of the panorama.
      2. Solution: Gap closing
   3. Method 2 (Bundle adjustment): Simultaneously align all the images using a least-squares framework to correctly distribute any mis-registration errors.
      1. disadvantage 1: Features that are observed many times are overweighted in the final solution.
      2. disadvantage 2: The derivatives of x ˜ik with respect to the {(Rj, fj)} are a little cumbersome, although using the incremental correction to Rj makes this more tractable.
      3. disadvantage 3: The disadvantage of full bundle adjustment is that there are more variables to solve for, so each iteration and also the overall convergence may be slower.
3. Composoting
   1. Choosing a Composting Surface (parameterization)
      1. Flat Panorama
         1. It start to look severely distorted once the field of view exceeds 90◦
      2. Cylindrical Panorama
      3. Spherical Panorama
   2. Reference Image (View selection)
      1. Flat Panorama
         1. A reasonable choice is the one that is geometrically most central
      2. Rotational Panorama
         1. We can choose the image whose z-axis is closest to the average z-axis (assuming a reasonable field of view).
      3. Cylindrical and Spherical Panoramas
         1. We can use the same heuristic if a subset of the viewing sphere has been imaged. In the case of full 360◦ panoramas, a better choice might be to choose the middle image from the sequence of inputs, or sometimes the first image, assuming this contains the object of greatest interest.
   3. Coordinate Transformations
      1. Flat
      2. Cylindrical or spherical
      3. Texture-mapped polyhedron
   4. Sampling issues
      1. Final panorama has lower resolution than the input images
         1. Pre-filtering the input images is necessary to avoid aliasing.
      2. Highest visual quality
         1. A higher order (e.g., cubic) interpolator combined with a spatially adaptive pre-filter may be necessary.
      3. Final Panorama should has higher resolution than the input images
         1. Super-resolution process
   5. Pixel selection and weighting (de-ghosting)
      1. Average
         1. Performed in an accumulation buffer (using the A channel as the weight).
         2. Simple averaging usually does not work very well, since exposure differences, mis-registrations, and scene movement are all very visible
      2. Median
         1. Used when rapidly moving objects are the only problem
      3. Center-weighting and minimum likelihood selection
         1. Used to retain multiple copies of a moving object.
      4. Feathered average
         1. Weighted averaging with a distance map
         2. It does a reasonable job of blending over exposure differences. However, blurring and ghosting can still be problems
      5. p-norm
         1. Raise the distance map values to some large power. The weighted averages then become dominated by the larger values.
         2. Provides a reasonable tradeoff between visible exposure differences and blur
      6. Voronoi diagram
         1. Assigns each pixel to the nearest image center in the set.
      7. "Cutting through" moving objects
         1. Place the seams in regions where the images agree, so that transitions from one source to another are not visible.
      8. Weighted ROD cover with feathering
         1. Determine regions of difference (RODs) See Page 401
      9. Graph cut seams with Poisson blending
         1. See Page 402
   6. Blending
      1. Laplacian pyramid blending
      2. Gradient domain blending
      3. Exposure compensation
4. Applications
   1. Digital maps and satellite photos
   2. Create beautiful ultra wide-angle panoramas
   3. Summarize and compress videos taken with a panning camera.
   4. Photomontage
5. Problems
   1. Parallax
   2. Exposure differences
   3. Lens distortion
   4. Scene motion
6. Current Research Topics
   1. The potential presence of large amounts of independent motion, camera zoom, and the desire to visualize dynamic events impose additional challenges.
   2. Create panoramic matte paintings for multi-plane cel animation
   3. Creating stitched images of scenes with parallax
   4. 3D representations of more complex scenes using multiple-center-of-projection images and multi-perspective panoramas
   5. The most difficult part of getting a fully automated stitching algorithm to work is deciding which pairs of images actually correspond to the same parts of the scene.
   6. The use of stereographic projections looking down at the ground (in an outdoor scene) to create “little planet” renderings.
7. Recognizing Panoramas
   1. Bundle adjustment combined with the process of topology inference can be used to automatically assemble a panorama.
   2. first find all pairwise image overlaps using a feature-based method and then find connected components in the overlap graph to “recognize” individual panoramas.
8. Parallax removal
   1. Caused by
      1. Unmodeled radial distortion
         1. Solution: plumb-line method (adjusts radial distortion parameters until slightly curved lines become straight)
      2. 3D parallax (failure to rotate the camera around its optical center)
         1. Solution: Doing a full 3D bundle adjustment
      3. Small scene motions such as waving tree branches
         1. Solution: General 2D motion estimation (optical flow) can be used to perform an appropriate correction before blending using a process called local alignment.
      4. Large-scale scene motions such as people moving in and out of pictures
         1. Solution: Select pixels from only one image at a time as the source for the final composite.
   2. General Note
      1. Local Alignment
9. Types
   1. Direct pixel-based alignment
      1. Disadvantage: They have a limited range of convergence. Even though they can be used in a hierarchical (coarse-to-fine) estimation framework.
   2. Feature-based approaches
      1. Disadvantage: Early feature-based methods would get confused in regions that were either too textured or not textured enough. The features would often be distributed unevenly over the images.
10. Codes
11. Resources
    1. The Photosynth Web site
    2. http://gigapan.org