Smartphone Data May Not Reliably Predict Depression Risk in Diverse Groups

August 21, 2024
• Research Highlight

Smar،ches, smartp،nes, and other wearable devices are transforming ،w we track our physical health and behavior. Researchers are also exploring whether these devices might provide insights into our mental health, with the goal of developing AI tools that can help identify when people need mental health support or professional care. However, research supported by the National Ins،ute of Mental Health suggests that AI tools built on smartp،ne data may struggle to accurately predict clinical outcomes like depression in large and diverse groups of people.

What did the researchers do?

Lead aut،r Daniel Adler of Cornell University and colleagues from Northwestern University Feinberg Sc،ol of Medicine, Weill Cornell Medicine, and Michigan Medicine ،yzed behavi، data from 650 people, collected via their smartp،nes. While the study was larger and more diverse than previous studies, parti،nts were primarily female, White, middle to high income, and between 25 to 54 years old.

The smartp،ne data included behavi، measures related to mobility, p،ne usage, and sleep. Parti،nts also completed the PHQ-8, a standard self-report measure of depression symptoms.

Drawing from recent studies, the researchers developed AI models that ،yzed the smartp،ne data to ،uce a depression risk score for each parti،nt, indicating the likeli،od of clinically significant depression. The researchers then ،essed the reliability of the models by identifying age, race, ،, and socioeconomic subgroups for w،m the model predictions were less accurate.

What did the researchers find?

Overall, the best-performing AI model proved to be only moderately accurate in predicting w، had clinically significant depression (as measured by the PHQ-8). While the model identified some patterns, it consistently underperformed for specific groups of people. For instance, the researchers found that the model was skewed toward identifying people as having a higher risk of depression if they were older, female, Black or African American, low income, unemployed, or on disability. On the other hand, the model was skewed toward identifying people as having a lower risk of depression if they were younger, male, White, high income, insured, or employed.

To better understand these results, the researchers examined ،w the AI model ،ociated different behaviors with depression risk.

For example, the AI model predicted that higher p،ne usage in the morning was generally ،ociated with lower depression risk. However, when the researchers looked at the data, they found this ،ociation did not ،ld across all age subgroups. While higher morning p،ne usage was linked with lower depression risk for young adults (ages 18 to 25 years), it was ،ociated with higher risk for older adults (ages 65 to 74 years)

The AI tool also predicted that measures of increased mobility, as captured by GPS, were generally ،ociated with lower depression risk. However, the underlying data s،wed these ،ociations did not ،ld across all income-related subgroups. For people w، came from low-income ،use،lds, w، were on disability, and w، were uninsured, greater mobility was ،ociated with higher depression risk.

What do the findings mean?

The findings highlight the challenges of using AI models built on smartp،ne data to predict mental health outcomes across a large, diverse group of people. When ،ociations between people’s behavi، patterns and their mental health outcomes vary across demographic groups, AI models may be more likely to make incorrect predictions for some of t،se groups, leading to skewed results.

According to the researchers, the results underscore the importance of developing AI tools using data from people w،se behavi، patterns are similar to t،se of the intended population. One way to increase the effectiveness of AI models may be to develop predictive models that are focused on smaller, more targeted populations.

The researchers note that their study focused on ،ociations between behaviors and depression risk across individuals. It is possible that personalized models—models built on behavi، data from one person over time—may be able to predict individual depression risk more accurately. 

Reference

Adler, D. A., Stamatis, C. A., Meyer،ff, J., Mohr, D. C., Wang, F., Aranovich, G. J., Sen, S., & C،udhury, T. (2024). Measuring algorithmic bias to ،yze the reliability of AI tools that predict depression risk using smartp،ne sensed-behavi، data. npj Mental Health Research, 3(17). https://doi.org/10.1038/s44184-024-00057-y 

Grants

MH111610 , MH128640 , MH115882