The location and recognition of text in video imagery is more difficult than many other OCR applications (e.g., reading printed matter) because of small character sizes and nonuniform backgrounds. In addition, the recognition of scene text can be particularly difficult if the text is viewed from an oblique angle, if the text is not in a two-dimensional plane, or if the text is blurred due to motion of either the camera or the text. SRI has developed an approach that involves binarizing individual color video frames and then applying a commercially developed OCR engine. Figure 1 shows the recognition of text overlaid on a map, and Figure 2 shows two examples of the recognition of scene text.

The accuracy of the recognition result can be improved substantially by postprocessing the OCR results with a lexicon of named entities extracted by MAESTRO from the audio or closed caption tracks. Figure 3 shows OCR recognition of a video overlay caption. Without help from a lexicon, the OCR result is "Blakjb" because of the background of the Prime Minister's hand behind the overlay caption. However, with the help of the lexicon of named entities extracted from the speech recognized in the audio track, the system is able to correctly recognize "Blair". In addition to successfully recognizing the text, the semantic content of the video scene itself (in this case, an image of Tony Blair) can be inferred from the text label, because the detection of instances of named entities in the video imagery often correspond to a person, place, or organization depicted in the video scene.