Adaptive Smoothing in Real-Time Image Stabilization

Abstract

When using the conventional fixed smoothing factor to display the stabilized video, we have the issue of large undefined black border regions (BBR) when camera is fast panning and zooming. To minimize the size of BBR and also provide smooth visualization to the display, this paper discusses several novel methods that have demonstrated on a real-time platform. These methods include an IIR filter, a single Kalman filter and an interactive multi-model filter. The fundamentals of these methods are to adapt the smoothing factor to the motion change from time to time to ensure small BBR and least jitters. To further remove the residual BBR, the pixels inside the BBR are composited from the previous frames. To do that, we first store the previous images and their corresponding frame-to-frame (F2F) motions in a FIFO queue, and then start filling the black pixels from valid pixels in the nearest neighbor frame based on the F2F motion. If a matching is found, then the search is stopped and continues to the next pixel. If the search is exhausted, the pixel remains black. These algorithms have been implemented and tested in a TI DM6437 processor.