R: [12,3] G: [5,2]
R: [12,3] G: [5,2]
Sergei Mikhailovich Prokudin-Gorskii traveled across the vast Russian Empire taking photographs of everything he saw. He recorded three exposures of every scene onto a glass plate using a red, a green, and a blue filter. The goal of this project is to automatically colorize the photos by stacking and aligning images from each color channel.
Naive Approach: To colorize the photos we need to first find an optimal alignment between different color channels. I experimented with two different objectives for determining the optimal displacement: normalized cross-correlation (NCC) and sum of squared difference (SSD). I observed similar results using both methods, but found SSD to be faster and selected it as my chosen objective. A naive approach to finding the optimal shift is to perform an exhaustive search over [-16,16] pixels on the x and y axis. This works well for smaller images but becomes computationally expensive for large ones.
Image Pyramid: To address this problem, I constructed an image pyramid on the principle that downsampling images and computing alignments on coarse images create good approximations that only need to be refined as the image is upsampled. I used an iterative approach that first downsamples all the images by a factor of 2 at each step until the pyramid has a height of 4 or the image is smaller than 400x400 pixels. While upsampling, I recompute the alignment at each level on a search window that is half as large (i.e. 4th level is [-16,16], 3rd level is [-8,8], etc.). I add the computed shift at each level to 2 global variables that keep track of total dx and dy displacement, multiplying by 2 to account for the number of pixels doubling when upsampling.
Base Image: I used the blue channel as the base image and try to align red and green channel to it.
Sobel Edge Detection: While the base methods worked well for most images, I found that certain images like "emir.tif" were difficult to align with SSD if I used RGB values as input. Instead of computing SSD on the RGB values of the image directly, I preprocessed the images using a sobel edge detection filter and computed SSD on the filtered image. This led to noticeable improvements visualized below.
