Social signal processing (SSP) is a less-than-ten-year-old interdisciplinary research field that intends to explore human-computer interaction (HCI) mechanisms by going beyond basic textual, verbal, or visual information exchanges between individuals and systems. Social signals are time-dependent streams of subtle behavioral items such as nonverbal audio (for example, prosody or accent), facial expressions, whole body postures, or hand gestures as introduced by interacting participants, be they robots, embodied conversational agents (ECAs), or humans. SSP intends to provide formal frameworks and tools for the modeling, analysis, and generation of such signals because they have been shown to be key elements to foster quality social interactions in dyadic or group settings.

With more than 400 pages, 50-plus authors, and four editors, the four-part opus Social signal processing is clearly positioned as a major contribution to the diffusion of the issues, methods, and techniques dedicated to SSP, covering the whole field. Part 1 describes the advances in the understanding of social signals coming from all relevant social sciences, such as psychology and anthropology. Part 2 focuses on the automatic, computer-based analysis of social signals generated by interacting humans, while Part 3 adopts the reverse point of view, addressing social signal generation processes by robots and ECAs. Part 4 concludes the book by describing six important use cases for SSP, including surveillance and assistive robotics for autistic children. Each part is structured in chapters, and each chapter provides more or less a literature survey of a specific issue and ends with a concluding section outlining its future challenges. Each chapter makes use of dozens of recent and highly relevant bibliographic entries.

Going through the almost 30 chapters of the book would not be appropriate for the space allotted to this review. As I expect most of the readers of this review to be computer scientists or at least computer savvy, I think it is useful to focus here on the first part of the book, dedicated to human social sciences; the other parts describe various algorithmic ways to handle social signals, building mostly on specific feature extraction and data representation issues, machine learning techniques, and dedicated Extensible Markup Language (XML)-based specification languages such as Behavior Markup Language (BML). Part 1 clearly shows how much can be gained from the mostly non-mathematical social science fields when addressing HCI. For instance, and contrary to what might be expected, chapter 3 mentions that experiments suggest that there exist universal human social signals that transcend situations and cultures: warmth and competence. These are key factors when assessing, conscientiously or not, unknown people on a first encounter. On a lighter note, when looking at the issue of verticality in social settings, that is, personality dominance, power roles, or socioeconomic status, one learns in chapter 4 that resource control such as space “ownership” or who enters first in a room have been shown to be important indicators. A more unexpected resource is the ability to deviate from social norms: for instance, it has been observed that high-power individuals leave generally more crumbs when eating; who knew!

Social signal processing is quite clearly a major contribution to the new, exciting, and important research field of SSP. I found the book quite well organized and set up, with very few typographical errors. Despite the implication of many authors and chapters, the reading levels and writing styles are adequately homogeneous. If there are some overlaps and repetition between different chapters, they are few. One may surely congratulate the editors for the high quality of the book’s presentation.

Yet this great work suffers from some shortcomings that will, hopefully, be addressed in future revisions. One is the lack of a global index, a feature I consider key to making such a compendium a really useful working tool. The second one is that, even though the book explicitly targets both beginners and more advanced researchers (this may explain why one doesn’t find any mathematical formulas in the whole book), I doubt the first group of readers will get much from some of the more technical chapters without resorting to serious web searching. If some technical terms related to data representation and machine learning such as support vector machines or cepstral coefficients may, barely, be assumed to be known by all readers, the Bhattacharyya distribution and the Kuncheva stability index clearly are not. Adding some footnotes or framed textboxes providing brief explanations for the more esoteric terms or algorithms would certainly greatly help newcomers to the field.

The lightning-speed development of deep learning techniques and big data will quite surely rapidly transform the burgeoning field of SSP and mandate future editions of this important book. As for now, this up-to-date survey is clearly successful in providing many concise entry points to the academic SSP literature for computer science researchers interested in advancing the state of the art of human-machine interaction.