By Meghan McCarthy
Editor’s Note: In light of the 2024 PennAITech Symposium, this article is a part of ongoing coverage on the intersection of technology and dementia.
Since its debut in 2022, the concept of ChatGPT has sparked discussions within the medical community. Perhaps you’ve heard of comparison between the technology’s bedside manner versus doctors or differences between physician discharge instructions versus ChatGPT.
While these studies offer intriguing insights, for many members of the public the primary value of ChatGPT actually lies in its ability to answer short questions. Common inquiries include definitions, general facts, and even philosophical inquiries like “What is the meaning of life?”
In the context of Alzheimer’s disease and related dementias (ADRDs), the question for many users may be simple: “What is dementia?”
This query is particularly prevalent among newly diagnosed patients or their families. Numerous online resources, including the Alzheimer’s Association and the Penn Memory Center, aim to provide answers.
But how much does ChatGPT really know about dementia?
Researchers at the University of British Columbia (UBC) sought to analyze this in a recently published study.
Search of “What is dementia?” on ChatGPT
ChatGPT (Conversational Generative Pretraining Transformer) is an artificial intelligence (AI) platform that can mimic conversations with a friend through written messages. Users can open ChatGPT online and pose a wide range of questions, from open-ended to highly specific.
Jill Dosso, PhD, is a postdoctoral research fellow in the Neuroscience, Engagement, and Smart Tech (NEST) Lab at UBC. The NEST Lab, led by Dr. Julie Robillard, takes a patient-centered approach to understanding how people perceive and interact with new technologies that may support their brain health, including social robots, health wearables, and generative AI like ChatGPT. Their studies run across the lifespan, from children with pediatric anxiety to older adults living with dementia and their care partners.
“My goal is working with potential users of health technologies to make recommendations that align with what they most need and want,” said Dr. Dosso.
After working with people with lived experiences of dementia in the context of social robotics (platforms that support users through interaction and companionship), Dr. Dosso and her colleagues became curious about how ChatGPT might also be relevant to conversations surrounding dementia.
In their recent study, Dr. Dosso and others at the NEST Lab asked ChatGPT common questions about ADRDs and compared its answers to ones from North American nonprofit organizations that support people with ADRDs such as the Alzheimer’s Association.
To determine what questions to ask, the team selected questions from FAQ pages of non-profit dementia organizations in Canada, Mexico, and the USA.
“This was a central question of the paper,” said Dr. Dosso. “Using the FAQs gave us something to compare to, especially because these organizations have a good understanding of the real questions that people are likely to ask.”
The NEST Lab has a background in analyzing the quality of online health information, with previous studies focusing on Twitter and Facebook. Specifically, Dr. Dosso’s mentor and co-author, Dr. Julie Robillard, developed a tool known as the Quality Evaluation Scoring Tool (QUEST) for online health information, which was used in this study. So, how much does ChatGPT know about dementia?
After asking the platform a variety of questions, the team has come to a nuanced conclusion of both pros and cons.
Overall, Dr. Dosso characterizes the answers from ChatGPT as accurate but shallow.
On one hand, ChatGPT did not use stigmatizing language or promote products or services when answering questions. The platform also regularly reminded users to consult a physician.
On the other hand, answers were repetitive and did not link to specific scientific sources or authors. Because nonprofits include links to local resources and contacts, they offer a richer multimedia experience than ChatGPT.
“ChatGPT speaks with an authoritative voice, but there’s no person accountable for the statements,” said Dr. Dosso. “It also lacks a timestamp, so you don’t know if information about drugs, for example, is the most up to date information.”
Beyond the quality of ADRD information, Dr. Dosso and the team also assessed its accessibility. They found that the ChatGPT-generated text was of a higher grade level than the nonprofit text, making it less readable. But both platforms have room for improvement in this area, as even the nonprofit reading level was above the recommended sixth grade reading level.
The NEST Lab team recommends that both ChatGPT and nonprofit organizations make efforts to increase their text accessibility.
Studies and recommendations about ChatGPT are likely to grow and change as the platform evolves.
Moving forward, Dr. Dosso advocates for researchers to directly engage with older adults and ADRD patients understand health technologies in light of their specific needs and concerns.
Infographic curated by the NEST Lab and Dr. Dosso
To read Dr. Dosso’s blog post about this study, please click here.
Jill Dosso, PhD is a Postdoctoral Research Fellow in Neurology in the NEST Lab. She is interested in how people perceive and interact with social robotics and other emerging smart, social health technologies. She works with families to think about how we can build social devices that are both ethical and helpful, with a special focus older adults living with dementia and children living with anxiety. Her publication list can be seen here.