Once we have the real and virtual camera calibrated, we need to register the virtual image data with the real image to obtain an augmented reality overlay. To do so, we pre-operatively scan the patient with the real calibration object so we obtain an additional virtual calibration object. If we match the calibrated patterned virtual object with the scanned virtual object, we will have registered the patient data with the current real image (showing the real calibration object/pattern). We employed a modified Iterative Closest Point (ICP) algorithm to match the scanned and virtual calibration object. The figure below show overlays for a a surgical stereo microscope.

Stereo window (left and right image separate): left is the microscope image of the left orbit; middle is the overlaid AR image; right is the virtual image from MR data showing the frontal lobe. Note the matching of landmarks. x-y overlay error is < 2mm (worst case).