Robust Understanding in Multimodal Interfaces

Multimodal grammars provide an effective mechanism for quickly creating integration and understanding capabilities for interactive systems supporting simultaneous use of multiple input modalities. However, like other approaches based on hand-crafted grammars, multimodal grammars can be brittle with respect to unexpected, erroneous, or disfluent input. In this article, we show how the finite-state approach to multimodal language processing can be extended to support multimodal applications combining speech with complex freehand pen input, and evaluate the approach in the context of a multimodal conversational system (MATCH). We explore a range of different techniques for improving the robustness of multimodal integration and understanding. These include techniques for building effective language models for speech recognition when little or no multimodal training data is available, and techniques for robust multimodal understanding that draw on classification, machine translation, and sequence edit methods. We also explore the use of edit-based methods to overcome mismatches between the gesture stream and the speech stream

Gesture · Human–computer interaction · Machine translation · Modalities · Multimodal interaction · Multimodality · Natural language processing · Robustness (evolution · Rule-based machine translation · Computer Science · Natural Language Processing Techniques · Speech and dialogue systems · Topic Modeling · Artificial Intelligence

• Rhetorical relations in multimodal documents

  Open Access•M Taboada, Christopher Habel•Discourse Studies•2013

  Add to list

• Mulitmodal Interactive Maps

  Sharon Oviatt•Human-Computer Interaction•1997

  Add to list

• A Systematic Comparison of Various Statistical Alignment Models

  Open Access•Franz Josef Och, Hermann Ney•Computational Linguistics•2003

  Add to list