• Skip to primary navigation
  • Skip to main content
SRI logo
  • About
    • Press room
    • Our history
  • Expertise
    • Advanced imaging systems
    • Artificial intelligence
    • Biomedical R&D services
    • Biomedical sciences
    • Computer vision
    • Cyber & formal methods
    • Education and learning
    • Innovation strategy and policy
    • National security
    • Ocean & space
    • Quantum
    • QED-C
    • Robotics, sensors & devices
    • Speech & natural language
    • Video test & measurement
  • Ventures
  • NSIC
  • Careers
  • Contact
  • 日本支社
Search
Close
Cyber & formal methods publications August 1, 1997

Multimodal Interfaces for Internet

Citation

Copy to clipboard


Julia, L., & Cheyer, A. Multimodal Interfaces for Internet.

Abstract

The World Wide Web frightened us: during the first two years of its popularity, we felt that human-computer interaction had been set back 30 years. Fortunately, Java expanded the possibilities for user interface design, providing a way to run complex programs over the net. In
this paper, we present a Java-enabled application with a multimodal (pen and voice) interface over the web.

Our implementation approach was to add Java to the set of languages accepted by the Open Agent Architecture (OAA), a framework for rapidly prototyping complex applications, and particularly suited to those with multimodal interfaces [1]. Given the OAA’s distributed nature, an OAA-based application can be run from a lightweight computer by downloading only the small user interface component, while the core of the application is implemented on a server composed of larger agents (e.g. speech recognition (SR), natural language, database) which cooperate and compete in parallel.

Despite the current lack of APIs for media input in Java, we chose voice entry as the primary input modality for the user [2], first by using a telephone to access a remote Nuance Speech Recognition server [3], and secondly by designing our own Java API for Speech Recognition using external native methods on the client.

ATIS [4], our first prototype application using SR over the telephone and the Java implementation of the OAA, has been publicly available on our web site for more than a year. Given the success of this first experiment (more than 4000 users), our next task was to bring multimodal concepts developed for several map-based applications [5] to the Web. In order to achieve our multimodal objectives, it was necessary to adapt the pen modalities (gestures and handwriting) to an Internet and multiplatform context, i.e. the pen had to be interchangeable with any pointing device. For portability, the gesture recognition algorithms [6] were recoded directly in Java. A trade-off was made for handwriting, integrating a Java-enabled version of JOT, a character by character recognizer developed by CIC [7].

↓ Download

Share this

How can we help?

Once you hit send…

We’ll match your inquiry to the person who can best help you.

Expect a response within 48 hours.

Career call to action image

Make your own mark.

Search jobs

Our work

Case studies

Publications

Timeline of innovation

Areas of expertise

Institute

Leadership

Press room

Media inquiries

Compliance

Careers

Job listings

Contact

SRI Ventures

Our locations

Headquarters

333 Ravenswood Ave
Menlo Park, CA 94025 USA

+1 (650) 859-2000

Subscribe to our newsletter


日本支社
SRI International
  • Contact us
  • Privacy Policy
  • Cookies
  • DMCA
  • Copyright © 2022 SRI International