Hidden Cognitive States: What Facial Expressions Can Actually Reveal

This study investigates how internal states, such as attention and motivation, influence the behavior of monkeys and mice during a natural task in virtual reality. Using advanced technology to analyze facial expressions, scientists found that both animals show similar facial patterns linked to their internal states. This helps to better understand how different species share ways of expressing emotions and thoughts through their faces.

In nature, mammals, regardless of species, share a series of behaviors that are fundamental to survival. Everyone, from mice to monkeys, needs to forage, rest, reproduce, avoid danger and explore their surroundings. These behaviors are not just automatic responses to environmental stimuli, as if an animal were to react mechanically to the smell of food or the sound of a predator.

In fact, they are guided by dynamic internal states, such as hunger, fatigue, curiosity, fear or attention. These internal states change over time and directly influence the animals’ actions. This raises an intriguing question: are the internal states that motivate these actions similar across species? Could what we call “attention” in a mouse be the same kind of attention we see in a monkey?

Traditionally, scientists have tried to understand these internal states using very restrictive and simplified methods. In many experiments, animals are placed in very controlled situations, with limited and repetitive tasks, such as pressing a button or moving their nose when a signal appears.

Furthermore, the types of tasks used vary greatly depending on the species studied, making it difficult to compare them. For example, to study attention in monkeys, they typically have to stare at a point while observing subtle changes in an object in the periphery of their field of vision.

In mice, on the other hand, studies usually involve a task in which they have to quickly detect which of five holes lights up and go to it. Although the results of these tasks, such as fast reaction times or correct responses, indicate attention in both cases, the tasks are so different that it is difficult to say whether the attention we are measuring in monkeys is comparable to that in mice.

To overcome these limitations, it would be ideal to study the internal states of animals while they behave more naturally, without so many artificial impositions. However, this is a great challenge. To be effective, a good method must be based on behaviors that the animals already perform naturally, without relying on intensive training.

Children as young as three years old recognize 'cuteness' in the faces of people and animals. by University of Lincoln

In addition, scientists need to find ways to identify the internal states of animals without forcing human concepts, such as trying to impose an idea of ​​“attention” or “motivation” that may not make sense for that species.

And finally, the method must be able to track how these states change over time, since they are dynamic by nature. To do this, simply counting how many times an animal presses a button or moves its nose is not enough. It is necessary to observe in detail how the animal moves, moment by moment, to be able to identify internal states from these small behavioral signals.

Recent advances in technology have helped open up new avenues for this type of study. One example is virtual reality environments, which allow the creation of highly controlled scenarios that are at the same time immersive and realistic for animals.

These environments can be adjusted to suit the sensory and behavioral abilities of different species, allowing monkeys and mice, for example, to perform very similar tasks, which facilitates comparisons between them. 

Another innovation comes from the use of deep learning algorithms, which enable automatic and precise tracking of animals’ movements, including subtle changes in posture and facial expression. This allows scientists to monitor the dynamics of behavior in real time and try to deduce which internal states are behind the observed actions.

In this particular study, researchers from Max Planck, Germany, used these technologies to explore and compare the spontaneous internal states of monkeys and mice, two species widely used in neuroscience research. They created a virtual reality task in which the animals had to search for food in a simulated environment, in a naturalistic way.

While the animals performed this task, their behaviors, especially their facial expressions, were recorded on video and analyzed with an advanced deep learning tool.

The information extracted from these expressions was used as the basis for a mathematical model called Markov Switching Linear Regression, which helps identify patterns in the data and infer internal states that change over time.

A key aspect of this work was to avoid the model only capturing movements directly linked to success or failure in the task. For example, paw movements that indicate that the animal is about to get the answer right could easily generate predictions, but this would not reveal anything about the animal’s actual internal states.

For this reason, the focus was on facial expressions, because they can reflect emotions and internal processes in a more subtle way and less linked to direct motor movement. Although many think that facial expressions are relevant only to social and visual species, such as monkeys and humans, recent research has shown that even mice display facial expressions that indicate states such as pleasure, pain or fear.

This suggests that facial expressions may have an evolutionarily conserved role in communicating and reflecting emotions.

In addition, different aspects of animal faces, such as pupil size, eye movements or whisker movements, have already been used in several studies to investigate processes such as attention, arousal and decision-making. However, the researchers in this study went further and sought to capture a more complete picture of facial expressions, rather than focusing on isolated features.

This image shows how scientists used the facial expressions of monkeys and mice to understand what was going on in the animals' minds during a task. In panel A, spider-shaped graphs indicate which parts of the face (such as the eyes, pupils, nose, ears, and whiskers) helped most to predict whether the animal would get the task right, wrong, or not. In panel B, we see that, when comparing only the traits that the two animals have in common, there is a similar pattern between the species, that is, monkeys and mice use similar parts of the face to express their internal states. Panel C shows how much each facial feature changes from one state to the other. Finally, panel D illustrates how the model can guess, in a single attempt, the animal's internal state based on the face, comparing the current facial expression with an average of the typical expressions of each state. All of this reinforces that the animals' faces reveal a lot about what they are feeling or thinking during a task. By doing so, they hoped to map in a richer and more detailed way the set of internal states that naturally arise during animal behavior, allowing direct comparisons between species in a more objective way and free from human assumptions.

Ultimately, the study showed that internal states inferred from facial expressions were able to reliably predict when animals would respond to stimuli presented in the task.

These states were also linked to animal performance and showed similar facial patterns between monkeys and mice, reinforcing the idea that facial expressions are an important and shared manifestation of internal cognitive states across species.

This innovative approach provides us with a new window into understanding how internal states guide animal behavior and how they can be meaningfully compared across species.

READ MORE:

Inferring internal states across mice and monkeys using facial features

Alejandro Tlaie, Muad Y. Abd El Hay, Berkutay Mert, Robert Taylor, Pierre-Antoine Ferracci, Katharine Shapcott, Mina Glukhova, Jonathan W. Pillow, Martha N. Havenith, and Marieke L. Schölvinck 

Nature Communications. volume 16, Article number: 5168 (2025) 

DOI https://doi.org/10.1038/s41467-025-60296-1

Abstract: 

Animal behaviour is shaped to a large degree by internal cognitive states, but it is unknown whether these states are similar across species. To address this question, here we develop a virtual reality setup in which male mice and macaques engage in the same naturalistic visual foraging task. We exploit the richness of a wide range of facial features extracted from video recordings during the task, to train a Markov-Switching Linear Regression (MSLR). By doing so, we identify, on a single-trial basis, a set of internal states that reliably predicts when the animals are going to react to the presented stimuli. Even though the model is trained purely on reaction times, it can also predict task outcome, supporting the behavioural relevance of the inferred states. The relationship of the identified states to task performance is comparable between mice and monkeys. Furthermore, each state corresponds to a characteristic pattern of facial features that partially overlaps between species, highlighting the importance of facial expressions as manifestations of internal cognitive states across species.