GPU Accelerated Realtime Stereo for Augmented Reality

Abstract

This paper presents a realtime, robust, and accurate stereo matching algorithm based on a coarse-to-fine architecture. At each pyramid level we use non-centered windows for matching and adaptive upsampling of coarse-level disparities. In order to minimize propagation of disparity errors from coarser to finer levels, we also perform an iterative optimization, at each level, that minimizes a cost function similar to the objective functions used in MRF stereo. On the Middlebury dataset our algorithm’s accuracy is comparable to other real time algorithms. At the same time the crispness of occlusion boundaries and the smoothness of the disparity maps are similar to those produced by global algorithms. We have implemented the proposed algorithm on Compute Unified

Device Architecture (CUDA) available on Nvidia GPUs. Our final algorithm runs at 32 fps on 640480 video with a disparity search range of 256, on a off-the-shelf Nvidia GPU. This is a significant improvement in image size, disparity range, and execution speed over other realtime algorithms. Such rapid and accurate stereo can greatly benefit several applications. In this paper, we show its utility for Augmented Reality, which requires dense depth estimates with accurate 3D boundaries. Through several examples, we demonstrate that the proposed algorithm’s accuracy enables inserting virtual characters into scenes with the correct occlusion effects.