Using LLMs to better understand autism diagnosis

Most autism diagnoses are based on human intuition: medical professionals evaluate patients according to a specific set of criteria and provide their assessment. But what if we could improve this method by understanding what these human assessments have in common?

Our paper was published in the general-domain journal Cell, one of the most influential and prestigious scientific journals in the world (alongside Nature and Science). Our work uses Large Language Models (LLMs) to analyse vast quantities of data from written reports by medical professionals in order to better understand how autism diagnosis is performed, and hopefully improve it.

Although there’s been thousands of research studies on the topic, we have not yet singled out a specific gene or brain region that is unique to autism: the best way to establish a diagnosis is still through clinical observation and decision-making from an experienced medical professional.

As AI researchers, the next best thing that we have to work with is the clinical thought process itself — the elements of clinical intuition used by healthcare professionals when diagnosing an individual who is suspected of having autism.

These reports don’t have any direct hint of what the diagnosis will be yet (as this conclusion is ultimately reached by a consensus of physicians), but LLMs can help find patterns in large amounts of text data such as electronic health records — in this case, the observations of patients’ behaviors and interactions written down by clinicians.

Sifting through thousands of sentences from autism diagnosis reports written by doctors to record their clinical session can be challenging, which is why we designed AI tools that were able to decode this clinical thought process: what is in the clinical experience that ultimately leads up to a confirmed diagnosis of autism?

In collaboration with researchers from Université de Montréal, we started by collecting and curating a rarely available corpus of over 4,000 clinical text documents that we then trained our AI model on.

We used LLMs pre-trained on a lot of text as the backbone of our model and added a separate interpretability module based on attention. This helps us understand why the model makes a certain decision -unboxing the black box- and highlights specific sentences most indicative of autism and of the overall diagnosis.

Indeed, instead of interpreting individual words like typical LLMs, we focused on interpreting whole sentences, making the results easier to understand for humans.

What we found puts a question mark on long-held beliefs on autism diagnosis. 

Existing criteria in the DSM-5 -the worldwide gold standard diagnostic manual for autism diagnosis- puts a lot of weight on social deficits in comparison to other criteria that our model spotlights as relevant.

Indeed, our analysis found stereotyped repetitive behaviors, special interests, and perception-based behavior to be the ones that matter most when diagnosing autism.

These findings were later confirmed by our expert collaborators at the Université de Montréal, based on decades of real-world experience.

This is one of the first studies to take advantage of large language models to leverage such a systematic deconstruction of the clinical thought process, and the results contrast with some 40 years of research in this topic.

With such AI tools, we can finally tap into the treasure trove of text that has been generated for decades throughout our healthcare systems. Moving forward, we’ll hopefully be able to leverage these powerful tools and unlock the information still hidden in these large text databases.

With these language modeling approaches, we can better understand what clinicians are actually looking at when performing a diagnosis and compare that directly to the existing guidelines. We hope to give novel analytical tools in the hands of scientists to hopefully inform the revision of real-life guidelines and improve our understanding of the autism diagnosis process.