CS 397

Assignment 3: Content-Based Image Retrieval

Due 16 October 2007

Overview

This project is about matching imagery in a way that makes sense to people. We will be working with a database of 1000 images taken by students in prior computer vision courses plus some new ones you will take. There are a number of design decisions you need to make as part of the project, but the overall task concept is straightforward: given an example image, find the closest matches.

Setup

Get a DVD from Prof Maxwell with the existing image database on it. On the DVD there are three subdirectories.

The label files provide some information about the image that may be useful. The label files have the following format.

You may want to copy the data from the DVD onto a local hard drive for processing to speed up access to the data, but it is not necessary.

Download the following files that define a basic histogram structure and IO routines for reading and writing histograms to a file. There are also accessor functions to make setting and getting data simple and robust.

If you have questions about using the C qsort function, take a look at the example of using qsort() to sort a set of records.

Procedure

  1. Take at least 25 images using a digital camera checked out from Prof. Maxwell. At least ten of the images should have a single dominant object in the image. While the object does not have to be identical in each image, it should be of the same type. You can take ten pictures of fire hydrants, for example, or ten pictures of shoes. Try to capture the object in different environments.

    Once you have finished taking the pictures, they will be available here. Once you have created the label files, please send them to Prof. Maxwell as a zip file.

  2. Generate a label file for each image using the format described above.
  3. Create a function that takes in an image and generates a 16x16x16 RGB histogram for the entire image. Use the histogram data structure provided.

  4. Create a function that takes two histograms and determines how similar they are using histogram intersection, which is the sum over all buckets of the minimum value of the two histograms in that bucket.
  5. Create an executable that reads in a list of image filenames (the database), the name of a target image, and returns the 10 images in the database that are most similar to the target in order of similarity, along with a score for each match.
  6. Run your algorithm on the following three target images and include the results in your report. These results provide a baseline performance measure against which you can compare other algorithms. The most similar pictures for 0002 are 0001, 0003, 0006, and 0012.
    1. Picture 0002
    2. Picture 0135
    3. Picture 0289
    4. Picture 0953
  7. Develop at least two more methods of matching images.
    • At least one of the alternative methods must use a different color/feature space than RGB. You can use texture features or other filter responses in addition to color distributions.
    • At least one of the alternative methods must use a different spatial arrangement than a whole image histogram. For example, you could divide the image into a central region and outlier region and produce separate histograms for each.
  8. Compare the results. You may want to have your system return all good matches, where good is defined by some cutoff on the match score. In your writeup, come up with a criteria for judging performance and rate the different systems. Subjective ratings by other people are also acceptable. Can your system find all the pictures of blue recycling buckets?

Extensions

Writeup

Follow the writeup instructions to create a web page for your assignment. Send the instructor an email with the code in a zip or tar file along with instructions on how to compile and run it as well as a pointer to the URL for the writeup.