Title image Spring 2019

Assignment 4: Calibration and Augmented Reality

Due 5 April 2016 (2 weeks)


This project is about learning how to calibrate a camera and then use the calibration to generate virtual objects in a scene. The end result should be a program that can detect a target and then place a virtual object in the scene relative to the target that moves and orients itself correctly given motion of the camera or target.


There will be several checkerboard targets in the lab. You can also use this image as a target. You can display it on a screen, or you could probably put it on your cell phone or on a tablet. This checkerboard pattern, when viewed with the long axis horizontally, has 9 columns and 6 rows of internal corners. You will want to use a pattern like this for both the calibration and VR parts of this project.


  1. The first task is to build a system for detecting a target, and extracting target corners. Get this working cleanly before you move on to the calibration task.

    The calibration documentation for OpenCV is fairly good. The relevant functions for the first task are findChessboardCorners, cornerSubPix, and drawChessboardCorners.

    For the corners argument, I suggest using a std::vector. If you have the definition:

    std::vector corner_set;

    Then the number of corners is given by corner_set.size(), and the corner values for the ith entry in the vector are corner_set[i].x and corner_set[i].y.

    Have your video draw the chessboard corners when it finds them. It can also be helpful to print out how many corners it finds, along with the coordinates of the first corner. Note that the first corner will generally be in the upper left of the checkerboard as viewed in the image.

  2. The next step is to let the user specify that a particular image should be used for the calibration and save the corner locations and the corresponding 3D world points. For example, if the user types 's', then store the vector of corners found by findChessbordCorners into a corner_list. At the same time, create a point_set that specifies the 3D position of the corners in world coordinates. You probably want to use the following definitions for the corner_list, point_set, and point_list.
    	std::vector point_set;
    	std::vector > point_list;
    	std::vector > corner_list;

    To build a point set you can either try to get specific and figure out the size of a checkerboard square in mm, or you can just measure the world in units of checkerboard squares and give the corners coordinates of (0, 0, 0), (1, 0, 0), (2, 0, 0), and so on. Note that if the (0, 0, 0) point is in the upper left corner, then the first point on the next row will be (0, -1, 0) if the Z-axis comes towards the viewer.

    There need to be as many points in the point_set as there are corners in the corner_set. You also need as many point_sets in the point_list as there are corner_sets in the corner_list.

    You may want to store the images themselves that are being used for a calibration. Store at least one for your project report.

  3. If the user has selected enough calibration frames--require at least 5--then let the user run a calibration. Alternatively, you could continuously update the calibration each time the user adds a new calibration image (beyond some minimum number), telling the user the current per pixel error after each calibration.

    Use the cv::calibrateCamera function to generate the calibration. The parameters to the function are the point_list, corner_list (definitions above), the size of the calibration images, the camera_matrix, the distortion_coefficients, the rotations, and the translations. You may also want to use the flag CV_CALIB_FIX_ASPECT_RATIO, which specifies that the pixels are assumed to be square (not a bad assumption these days).

    You will have already generated the point_list and corner_list vectors. Make the camera_matrix a 3x3 cv::Mat of type CV_64FC1. You probably want to initialize the 3x3 camera_matrix to something like.

    [1, 0, frame.cols/2]
    [0, 1, frame.rows/2]
    [0, 0, 1           ]

    Print out the camera matrix and distortion coefficients after the calibration, along with the final re-projection error. The two focal lengths should be the same value, and the u0, v0 values should be close to the initial estimates of the center of the image. Your error should be less than a half-pixel, if everything is working well. Include the error estimate in your report.

    The calibrateCamera function also returns the rotations and translations associated with each calibration image. If you saved the calibration images, you might want to also save these rotations and translations with them.

    Enable the user to write out the intrinsic parameters to a file: both the camera_matrix and the distortion_ceofficients.

  4. For the remaining tasks you can either write a completely separate program or continue to enhance your original one.

    The next task is to write a program that reads the camera calibration parameters, then starts a video loop. For each frame, it tries to detect a chessboard. If found, it grabs the locations of the corners, and then uses solvePNP to get the board's pose (rotation and translation).

    Have your program print out the rotation and translation data in real time, as you are testing this task.

  5. Given the pose estimated in the prior step, have your program use the projectPoints function to project the 3D points corresponding to the four outside corners of the chessboard onto the image plane in real time as the chessboard or camera moves around. Alternatively, put 3D axes on the board attached to the origin. Include at least one image from this step in your report.
  6. Construct a virtual object in 3D world space made out of lines. Then project that virtual object to the image and draw the lines in the image. Make sure the object stays in the right orientaiton as the camera moves around. Note, you may need to create an asymmetrical target in order for this to work properly from all angles. You may want to create an asymmetrical virtual object as well to aid in debugging.

    Have fun. Take some screen shots and/or videos of your system in action for your project report.

  7. Pick a feature (e.g. Harris corners) and write a separate program that shows where the features are in the image in a video stream. Make a pattern of your choice and show where those features show up on the pattern. In your report, explain how you might be able to use those feature points as the basis for putting augmented reality into the image.

    Take some screen shots and/or videos of the system working.



For this project, make a wiki page that begins by explaining your overall pipeline from calibration to place AR objects in the scene. Your audience is your fellow CS majors not in the course.

Explain your calibration system and how it works. Your audience for this section is other students in the course. Provide some kind of proof that the system is providing useful/reasonable information.

Your project report should include images from each stage: chessboard detection, back-projection of the four chessboard corners, and the virtual objects in place.

You should take a video/screen capture sequence.

If you did any extensions, describe the algorithms and show at least one example for each extension.

Give your wiki page the label: cs365s19project04


Put a zip file or tar file of your code in your Private Courses handin directory. Please do not include images with your handin folder.