Researchers in broad disciplines such as natural language processing usually conduct their research by breaking the general problem down into many small problems. A researcher can make a career out of solving a few of these small problems. It is rare for many researchers to coordinate their efforts and demonstrate what progress has been made to solve the general problem. This book documents one such effort.

Though this book does not address the whole of natural language processing, it does address a large part of it. We can view the research reported here as an attempt to build an intelligent system that can carry on conversations with humans in which questions are answered, and follow-up questions are answered in context. Communication between humans and the system takes the form of text, speech, and gesturing (mouse cursor movements on the computer screen). The Interactive Multimodal Information Extraction (IMIX) project, initiated by the Netherlands Organization for Scientific Research (NWO), coordinated the project.

The book is a collection of chapters written by various researchers in the IMIX project. It consists of five parts. Part 1 introduces the IMIX project and describes the system implementation called the IMIX Demonstrator. Researchers posed questions to this system by entering text, speaking into a microphone, and pointing at and drawing on images. The system responded in multiple ways to carry on an informative dialogue with the user. Though the Demonstrator was not the primary focus of the IMIX project, it did show how much of the research we can fit together into an integrated system. It encouraged coordination of the many research efforts, inspired interest in them, and, to some extent, helped to reveal problem areas where more research is necessary.

The next three parts of the book cover the major subdisciplines that the book’s title mentions. Part 2 describes how the dialogue between the user and the system is managed. Since the authors desired that the dialogue be “natural,” that is, comparable to the dialogue that might occur between two humans, they collected and studied corpora of human dialogues involving multiple modes of communication. One of the things learned from those studies was that such dialogues manage several goals at the same time.

Part 3 deals with the generation and presentation of complex answers to questions, using combinations of text, images, and speech.

Part 4 (comprising almost half of the book) deals with text analysis for question answering. The five chapters here present approaches for extracting medical term variants from texts that are available in multiple languages, extracting relations for answering questions in open and closed domains, classifying words and other entities by reasoning about constraints, extracting certain semantic relationships between words, and extracting taxonomic information from the discourse structure of texts.

The final part, Part 5, is a short chapter that summarizes the IMIX project, evaluates its accomplishments and approach to coordinating many research efforts, and makes recommendations for doing similar collaborative research in the future.

Anyone who wants to know about the state of the art in question answering will be interested in this book. Though many examples of text in a couple of the chapters are in Dutch, readers can learn the information in this book without knowing that language.