Researchers are developing AI-powered smartphone apps to detect signs of depression noninvasively. One system, PupilSense, monitors pupillary reflexes to identify potential depressive episodes with 76% accuracy. Another tool, FacePsy, analyzes facial expressions and head movements to detect subtle mood changes, with unexpected findings such as increased smiling potentially linked to depression. These tools offer an affordable and privacy-friendly way to identify depression early, leveraging everyday smartphone use.
Mental health encompasses emotional, psychological, and social well-being, affecting how we think, feel, and act. Mental health issues such as depression and anxiety are a leading cause of disability worldwide, affecting millions of people. These issues often begin in early adulthood, and if left untreated, can impair academic performance, work, and relationships.
During the COVID-19 pandemic, social distancing has led many to seek mental health support online, such as telepsychiatry. In 2020, anxiety and depression levels peaked, but have started to decline in 2021. Despite this, many still suffer from these mental health issues and chronic pain.
Studies suggest that depression can be identified by nonverbal cues, such as facial expressions and head movements. A new system, called FacePsy, attempts to use smartphones to detect these cues automatically, without invading users’ privacy.
It collects information such as eye status (open or closed), smiles and head position to predict episodes of depression. Although promising, the system is still being tested to ensure that it works well in real-life situations; we’ll give you more details later.
In general, these studies show that depressed people tend to display fewer happy facial expressions and less head movement, with reduced facial expressiveness. Specifically, it is common for patients with depression to display fewer signs of happiness and fewer facial movements overall. In addition, other studies indicate that these people have weaker pupillary responses.
However, the relationship between depression and negative facial expressions is still a matter of debate. While some studies associate depression with an increase in negative facial expressions, others suggest that, in certain cases, depressed individuals may even display positive expressions more frequently.
To study these variations, analyzing facial expressions using affective computing technology has been a promising approach. Researchers have used nonverbal cues such as facial action units (AUs) and facial modeling techniques to detect patterns associated with depression. For example, expressions such as the “dimple” (AU14) and lowering of the corner of the lip (AU15) have been useful in predicting depression severity levels in controlled studies.
More recently, algorithms and sensors from mobile cameras have been used to capture and analyze facial expressions and pupils, as in the MoodCapture study, which automatically collects facial images to predict depression levels. These approaches still raise privacy concerns, since most systems process data on external servers. The latest study proposes a solution where all processing occurs on the user’s device, ensuring greater security and privacy.
These studies pave the way for automated tools capable of predicting depression through facial expressions, but implementation in uncontrolled environments and respecting privacy remain major challenges. Advances in depression detection through mobile devices, with a focus on systems that monitor behavioral and physiological signals.
Examples of research include Chikersal et al., who used the AWARE framework to track data from smartphones and wearable devices, achieving over 85% accuracy in detecting depression in students. Asare et al. also used this framework to track sleep, physical activity, and phone usage data, resulting in an accuracy of 81.43%. However, these approaches, while effective, still face limitations in implementing real-time interventions due to the complexity of the data processes.
Another relevant study is that of Pedrelli et al., who combined data from wearable trackers and smartphones to monitor physiological changes, such as heart rate and skin temperature, to detect depression. However, the use of devices such as the Empatica E4, which costs around US$1,690, makes these solutions expensive and invasive, resulting in low participant compliance.
Current mobile-based solutions focus primarily on the behavioral aspects of depression but struggle to capture physiological signals rich in emotional detail. Although laboratory studies have shown potential in detecting depression through physiological and emotional signals, effective real-world implementation is still lacking.
FacePsy, a system proposed in the ACM Journal paper by researchers at Stevens Institute of Technology, with a prototype presented at the International Conference on Activity and Behavior Computing in Japan and also at the ACM International Conference on Mobile Human-Computer Interaction (MobileHCI) in Australia, aims to overcome these limitations by passively detecting facial signals during everyday smartphone use. It offers an open-source solution, with local processing on the device, seeking modularity to facilitate collaboration and evolution of the tool.
Data collection occurs unobtrusively when the user unlocks the phone or uses apps, allowing for continuous analysis of emotional and mental state in near real-time.
FacePsy is designed to capture facial behavior primitives in real-time as users interact with their smartphones. The app operates at a 2.5Hz response rate, balancing efficiency and power consumption. It uses the front-facing camera to capture facial images during specific interactions, such as unlocking the device and using apps.
FacePsy’s architecture incorporates advanced technologies, such as facial landmark detection and facial action units (AUs), all of which are processed locally on the device. This protects users’ privacy and reduces the need for sensitive data transmission.
The system accurately calculates pupil diameters, compared to the irises around the eyes, from 10-second “burst” photo streams captured while users are opening their phones or accessing certain social media and other apps.
Example of the image test for capturing facial features and pupil diameter
In an initial test of the system with 25 volunteers over four weeks, the system, embedded in the volunteers’ smartphones, analyzed approximately 16,000 phone interactions after collecting pupil image data. After teaching an AI to differentiate between “normal” and abnormal responses, Bae and Islam, the paper’s authors, processed the photo data and compared it to the volunteers’ reported moods.
The best iteration of PupilSense, known as TSF, which uses only selected, high-quality data points, proved to be 76 percent accurate at flagging moments when people were genuinely feeling depressed. That’s better than the best smartphone-based system currently being developed and tested for depression detection, a platform known as AWARE.
Key features of FacePsy:
Real-time facial data capture: The system is configurable to sample facial data during specific interactions, such as unlocking the phone or using apps.
On-device processing: All facial data processing is performed locally, ensuring that no images leave the device, promoting privacy and power efficiency.
Unobtrusive and efficient data collection: The system records at a rate of 2.5 Hz, with 10-second sessions, which optimizes data collection without overloading the device.
Flexibility for researchers: FacePsy allows researchers to configure data collection parameters, such as triggers and sampling rates, to tailor the system to the needs of each study.
A depressive episode prediction model that achieves 81% AUROC.
Overview of how FacePsy works
Increased smiling, for example, appeared in the pilot study to correlate not with happiness but with potential signs of depressed mood and affect. This could be a coping mechanism, for example, people putting on a “brave face” for themselves and others when they’re feeling down, Bae says. “Or it could be an artifact of the study. More research is needed.”
Other apparent signs of depression revealed in the early data included fewer facial movements during the morning hours and certain very specific patterns of eye and head movement. (Yowling or side-to-side head movements in the morning seemed to be strongly linked to increased depressive symptoms, for example.)
Interestingly, greater detection of wide-open eyes during the morning and evening hours was also associated with potential depression — suggesting that outward expressions of alertness or happiness can sometimes mask depressed feelings underneath.
READ MORE;
FacePsy: An Open-Source Affective Mobile Sensing System - Analyzing Facial Behavior and Head Gestures for Depression Detection in Naturalistic Settings
Rahul Islam and Sang Won Bae
Proceedings of the ACM on Human-Computer Interaction doi.org/10.1145/3676505
Abstract:
Depression, a prevalent and complex mental health issue affecting millions worldwide, presents significant challenges for detection and monitoring. While facial expressions have shown promise in laboratory settings for identifying depression, their potential in real-world applications remains largely unexplored due to the difficulties in developing efficient mobile systems. In this study, we aim to introduce FacePsy, an open-source mobile sensing system designed to capture affective inferences by analyzing sophisticated features and generating real-time data on facial behavior landmarks, eye movements, and head gestures - all within the naturalistic context of smartphone usage with 25 participants. Through rigorous development, testing, and optimization, we identified eye-open states, head gestures, smile expressions, and specific Action Units (2, 6, 7, 12, 15, and 17) as significant indicators of depressive episodes (AUROC=81%). Our regression model predicting PHQ-9 scores achieved moderate accuracy, with a Mean Absolute Error of 3.08. Our findings offer valuable insights and implications for enhancing deployable and usable mobile affective sensing systems, ultimately improving mental health monitoring, prediction, and just-in-time adaptive interventions for researchers and developers in healthcare.
Comentários