Designing the User Interface for Multimodal Speech and Pen-Based Gesture Applications

State-of-the-Art Systems and Future Research Directions

Dados Bibliográficos

ID15216144
AutoresSharon Oviatt (0000-0003-4664-1412, Oregon Health & Science University, autor correspondente), Philip R Cohen (0000-0003-1303-064X, Oregon Research Institute), Phil Cohen, Lizhong Wu (Software (Germany)), Lisbeth Duncan, Bernhard Suhm (Boeing (Australia)), Josh Bers, Thomas F Holzman, Thomas Holzman, Terry Winograd (0000-0003-4435-7215), James A Landay (0000-0003-1520-8894, Stanford University), James Landay, Jim Larson (0000-0002-3814-2659, University of California, Berkeley), David Ferro (Intel (United States))
Ano2000
Volume15
Fascículo4
Páginas263-322
Data de publicação2000-12-01
Peer ReviewedSim
Open AccessNão
TipoARTICLE
PeriódicoHuman-Computer Interaction (JOURNAL)
Identificadores do periódicoISSN: 0737-0024 • E-ISSN: 1532-7051
EditoraTaylor & Francis (PUBLISHER • GB)
DOI10.1207/s15327051hci1504_1
OpenAlexW2009803366
IdiomaEN
Citações recebidas3
Referências citadas78

The growing interest in multimodal interface design is inspired in large part by the goals of supporting more transparent, flexible, efficient, and powerfully expressive means of humancomputer interaction than in the past. Multimodal interfaces are expected to support a wider range of diverse applications, to be usable by a broader spectrum of the average population, and to function more reliably under realistic and challenging usage conditions. In this paper, we summarize the emerging architectural approaches for interpreting speech and pen-based gestural input in a robust manner--- including early and late fusion approaches, and the new hybrid symbolic/statistical approach. We also describe a diverse collection of state-of-the-art multimodal systems that process users' spoken and gestural input. These applications range from map-based and virtual reality systems for engaging in simulations and training, to field medic systems for mobile use in noisy environments, to web-based transactions and standard text-editing applications that will reshape daily computing and have a significant commercial impact. To realize successful multimodal systems of the future, many key research challenges remain to be addressed. Among these challenges are the development of cognitive theories to guide multimodal system design, and the development of effective natural language processing, dialogue processing, and error handling techniques. In addition, new multimodal systems will be needed that can function more robustly and adaptively, and with support for collaborative multi-person use. Before this new class of systems can proliferate, toolkits also will be needed to promote software development for both simulated and functioning systems. Multimodal Speech and Gesture Interfaces 3 CONT

Field (mathematics · Function (biology · Gesture · Human–computer interaction · Interface (matter · Multimedia · Multimodal interaction · Multimodality · Process (computing · USable · User interface · World Wide Web · Computer Science · Multi-Agent Systems and Negotiation · Natural Language Processing Techniques · Speech and dialogue systems · Artificial Intelligence