Assignment 2: Content-Based Image Retrieval
Due 28 February 2019 (2 weeks)
This project is about matching imagery in a way that makes sense to people. You will be working with the database of images you capture in the first project. There are a number of design decisions you need to make as part of the project, but the overall task concept is straightforward: given an example image, find the closest matches.
The database of images will be available on the courses server in the folder Course Materials/ImageDB/. You may want to download a subset of them onto your local HD for testing purposes.
Read the OpenCV documentation on histograms. You will likely want to use histograms in various color spaces to do matching.
Baseline Matching: write a content-based image retrieval [CBIR] program that takes in a query image and an image database, matches the query image to each database image using a distance metric, then sorts the database images according to their similarity to the query images. The most similar images--smallest distance--should come first in the list. Your program should take in an argument N, which is the number of results to show.
Here is some example code showing you how to get the filenames for a directory in C/C++.
If you write your code intelligently, you should be able to use the same framework for the rest of the tasks, substituting different distance metrics for the various tasks.
To test your system, write a distance metric that takes a 5x5 block of pixels from the center of the two images and finds the sum-squared-distance [SSD] between those blocks. To compute the SSD of two identical image blocks, loop over each pixel in the block and compute the sum of the squares of the differences of the overlapping pixels in each color channel.
Required result 1: show the top five matches in the image DB when you use this image as a query image.
Baseline Histogram Matching: write a version of the CBIR program that uses a whole-image histogram to determine the similarity
between the query image and the DB images. You can choose which histogram comparison metric to use or give the user a choice of
Required result 2: A list of the ten most similar images to this query image and a description of the histogram distance used to obtain the reported result.
Multiple Histogram Matching: write a version of the CBIR program with a distance metric that incorporates multiple (2 or more)
histograms per image, dividing the image into parts. The parts can be overlapping, non-overlapping, regular or irregular.
Required result 3: A list of the ten most similar images to this query image and a description of the distance metric used to obtain the reported result.
Integrating Texture and Color: write a version of the CBIR program with a distance metric that includes both color and texture
information. You can use whole-image histograms or some subdivision of the image into parts.
Required result 4: A list of the ten most similar images to this query image and a description of the distance metric used to obtain the reported result.
Custom Distance Metric: write a version of the CBIR program with a distance metric you design. Pick some query images as an evaluation set and pick some images in the DB you think should match. Try to get your system to bring those matches to the top.
Required result 5: for a query image of your choice, show the top 5 results for each of the distance metrics above and your custom distance metric. Discuss any differences in the results.
- Try out other methods of image matching. Different feature spaces, number of buckets, and spatial arrangements all produce slightly different effects.
- Create a GUI around your CBIR system so you can select a query image, pick a distance metric, and display the results.
- The best system developed for this lab so far was created by Casey Smith, who looked at the spatial variance of the primary colors in the image and matched based on the primary colors and the similarity of the variances. He was able to get the system to return all of the pictures containing bananas using his method with good precision. So the challenge is made.
For this project, make a wiki page that begins by explaining your overall pipeline for CBIR and the task you are trying to solve. Your audience is your fellow CS majors not in the course.
Show the query image and the top matches for each case/distance metric. Explain your distance metric for each case. For example, be specific as to the spatial extent and size of histograms, or how you weighted the different inputs to the distance metric. Your audience in this section is other students in the course.
If you did any extensions, describe the algorithms and show at least one example for each extension.
Give your wiki page the label: cs365s19project02
Put a zip file or tar file of your code in your Private Courses handin directory.