I have discussed the Homography matrix from my previous blog, which is a powerful tool usually used for remote measurements, correcting satellite images, perspective correction, image stitching, calculation of depth, and camera pose estimation. However, before we can compute for the Homography matrix, we need to define manually the source and destination points, which can be tedious. This blog will discuss one way in identifying these points called Template Matching.
Suppose we need to find a point in our image where we already determined what the point should resemble. We can use the Template Matching technique to find the reference image (template image) in a bigger image (source image). It is instrumental in image detection, object tracking, and recognition. But how does Template Matching work? The template is compared to each part of the source image, sliding one pixel at a time. Pixels in the source image that matched the template will appear brighter than others in the resulting image.
Template matching works with colored images, but we will convert the source image to grayscale for simplicity. Suppose that we have an aircraft carrier and we want to find similar aircraft to the template image shown in figures 1 and 2.
from skimage.io import imread, imshow
from skimage.color import rgb2gray
carrier = imread('aircraft_carrier.jpg')
carrier_gray = rgb2gray(carrier)
imshow(carrier_gray);
Fig 1. Aircraft Carrier Gray scaled
template = carrier_gray[648:744,775:838]
imshow(template);
Fig 2. Template Aircraft image
Now that we have a template image, we can match the template to the source image using the match_template function from skimage.feature module. As I have mentioned earlier, we can look for the matching objects by looking at the pixel with the highest value.
from skimage.feature import match_template
result = match_template(carrier_gray, template)
imshow(result, cmap='viridis');
Fig 3. Template Matching result
To locate the objects that match the template, ew can look for the peak values with at least a certain correlation threshold. Based on the figure below, we can see that Template Matching can successfully capture objects in the source image based on the template image.
from skimage.feature import peak_local_max
import matplotlib.pyplot as plt
imshow(carrier_gray)
template_width, template_height = template.shape
for x, y in peak_local_max(result, threshold_abs=0.5):
rect = plt.Rectangle((y, x), template_height, template_width, color='y',
fc='none')
plt.gca().add_patch(rect);
Fig 4. Template Matching result
However, Template Matching does have its limitations.
Scale-variant - Template Matching does not work if you enlarge your template.
from skimage import exposure
from skimage.transform import rescale, resize
fig, (ax0, ax1, ax2) = plt.subplots(1, 3, figsize=(20, 7))
ax0.imshow(carrier)
ax0.set_title('Carrier')
# Enlage the template image
template_enlarge = rescale(template, scale=2)
# Match the template
result_enlarge = match_template(carrier_gray, template_enlarge)
# Show carrier
ax1.imshow(carrier_gray, cmap='gray')
ax1.set_title('Template Matching result (Enlarge template)')
template_width, template_height = template_enlarge.shape
# Plot matched results
for x, y in peak_local_max(result_enlarge, threshold_abs=0.8):
rect = plt.Rectangle((y, x), template_height, template_width, color='y',
fc='none')
ax2.add_patch(rect)
ax2.imshow(template_enlarge, cmap='gray')
ax2.set_title('Template Enlarge');
Fig 5. Template Matching with larger template image
Orientation/Rotation-variant - Template Matching does not work if the template image has a different orientation/rotation.
fig, (ax0, ax1, ax2) = plt.subplots(1, 3, figsize=(20, 7))
ax0.imshow(carrier)
ax0.set_title('Carrier')
# Flip the template image
template_flip = template[:, ::-1]
# Match the template
result_flip = match_template(carrier_gray, template_flip)
# Show carrier
ax1.imshow(carrier_gray, cmap='gray')
ax1.set_title('Template Matching result (Flipped template)')
template_width, template_height = template_flip.shape
# Plot matched results
for x, y in peak_local_max(result_flip, threshold_abs=0.8):
rect = plt.Rectangle((y, x), template_height, template_width, color='y',
fc='none')
ax1.add_patch(rect)
ax2.imshow(template_flip, cmap='gray')
ax2.set_title('Template Flipped');
Fig 6. Template Matching with flipped template image
Intensity variant - Template Matching does not work effectively if the template image has a different intensity range from the source image.
fig, (ax0, ax1, ax2) = plt.subplots(1, 3, figsize=(20, 7))
# Change contrast of carrier
carrier_contrast = exposure.adjust_log(carrier, 1.5)
ax0.imshow(carrier_contrast)
ax0.set_title('Carrier')
carrier_contrast = rgb2gray(carrier_contrast)
# Match the template
result_contrast = match_template(carrier_contrast, template)
# Show carrier
ax1.imshow(carrier_contrast, cmap='gray')
ax1.set_title('Template Matching result (Carrier Adjusted Contrast)')
template_width, template_height = template.shape
# Plot matched results
for x, y in peak_local_max(result_contrast, threshold_abs=0.8):
rect = plt.Rectangle((y, x), template_height, template_width, color='y',
fc='none')
ax1.add_patch(rect)
ax2.imshow(template, cmap='gray')
ax2.set_title('Template');
Fig 7. Template Matching with adjusted source image contrast
In summary, Template Matching is a useful tool for object detection and tracking based on a reference image. However, its limitations will not work effectively if the template image has a different scale, orientation, and intensity from the source image. In my next blog, I will discuss SIFT and RANSAC algorithm, which will solve these limitations.
Comments