As sensors become more ubiquitous in smart homes and work environments, the data they provide can offer information about the locations and actions of any occupants. The authors propose using a combination of deep learning and finite state machine (FSM) methods to recognize and model human activities from ambient sensors (such as motion or open/close sensors). While valuable information can also be extracted from cameras, their use is not considered due to privacy considerations.

A human activity can be viewed as a sequence of events or actions, and specified through an FSM: events are mapped onto states, and actions onto the transitions between the states. Since details of an activity like duration, location, or sequence of specific actions may vary, an FSM may be too rigid to capture the activity. In a fuzzy FSM (FFSM), the mapping of events to states is achieved through linguistic variables, with the effect of allowing multiple states to be active, each with a different degree of belonging.

Assuming that a set of activities has been modeled as an FFSM, it is necessary to associate a sequence of sensor data values with an activity. To capture the temporal correlations in the sensor data, the authors propose the use of long short-term memory (LSTM), a type of recurrent neural network that can remember previous information from a stream of input data.

The combination of LSTM and FFSM allows the extraction of temporal relations from the input data, the derivation of fuzzy rules for the transitions between the system states, and the determination of the most suitable current human activity. In a home or work environment scenario, background knowledge can also be used to narrow down the range of activities by identifying relevant system states, providing templates for the fuzzy rules for the state transitions, and selecting linguistic labels for data items like location (for example, rooms), time points (morning), and events (front door opens).

The experimental setup used includes a number of sensors (motion, open/close electricity use, mat pressure) and seven different activities mapped onto states of the FFSM model (sleeping, grooming, kitchen, dining, living room, garden, leaving) for a single user. For a duration of five days, the system identified the correct activity with an accuracy between 92 and 100 percent. While it is unclear how well this approach will work under less constrained conditions, it indicates that recognizing and modeling human activities in a home environment with a moderate amount of ambient sensors is feasible.