ISC 2026 | The development of a large language model to identify thrombolysis contraindications in telestroke

So before I begin, I want to iterate tremendous support of my colleagues, Dr Uchino and co-authors, Dr Fares Antaki, who’s also with the Cleveland Clinic. The issue I have is as a stroke neurologist at 2 a.m., I have to open the chart of a patient I don’t know about and quickly review to see if this patient is eligible for stroke thrombolysis. This is, of course, when I’m on call and there is an acute stroke that is suspected in the ER somewhere. The issue is, as a human who reviews a chart at 2 a.m., I can make mistakes. I sometimes miss an important fact. Let’s say this patient has an aneurysm or has had bleeding in the stomach recently or on certain blood thinners for instance. And even if I’m perfectly accurate it is very possible that I may take two three four even five minutes to review the chart to make sure I absolutely do not miss anything of course that takes time and we know in strokes that every second matters when it comes to saving those brain cells from devastating stroke. So what we imagine is what if we use something like ChatGPT which is a large language model that can scan a large chunk of text within the electronic health records very quickly, far more accurately than what any human can reasonably do? What if that is a tool that can meaningfully assist in our current acute stroke workflow? So we build an app. We took 30 notes from patients, from our cohort, which I’ll explain in a second. Let’s say for a given patient, we take their most recent 30 notes, 30 clinical notes, and we design a custom proprietary prompt for our large language model, which is, in this case, we use ChatGPT, and we just run it through ChatGPT, and the ChatGPT will tell us if the patient has yes or no a contraindication let’s say patient has a GI bleed when five days ago you can find the actual text in this little excerpt so you can verify there’s the date and the author of the note so you can find the source note itself. And we aim to once we designed this kind of app the structure we set out to validate if this is actually accurate and fast, which it was.

So we used 388 patients this is all the patients that came through a random month that we selected through our Cleveland Clinic, either our main campus or one of our 16 sites. So we have a lot of smaller hospitals. But we all share a common electronic health record. So out of these 388 patients, we run through the pipeline I just mentioned. And we took two neurologists to go through each chart. Now, they don’t know anything about ChatGPT. They just go through the chart about 30 minutes for each chart, right? Really thoroughly, much more thoroughly than you would afford in a real case scenario to really make sure that every contraindication is identified. And then we want to have two physicians doing that independently. We really don’t want to miss anything. And then they compare their answers, and maybe one has missed, the other has not. And they come up with this final kind of answer sheet. This is as good as it gets in terms of a gold standard, in terms of the answer sheet. And then we compare what the ChatGPT says about these patients against what these two independent human reviewers have found and what we found is a sensitivity of 95%. So just a bit of background among these 388 patients most of them I would say about 75/80% of them don’t have any contraindications so they you know they’re finding that’s something we expect a minority of patients they do one, two, or sometimes even three contraindications. And if you take the total of all these contraindications among this pool of patients, there are about, if I remember correctly, about roughly 100 contraindications. And ChatGPT missed five. So the sensitivity is 95%. The sensitivity is 95%. And then there is going to be false positive, of course. If ChatGPT says something, and then we’re like, it’s not in our answer sheet. So that’s a false positive, right? So we come across about one in two things that ChatGPT says that is a false positive.

Now, because of the way we design right you can verify with the note has the actual excerpt as a human it takes seconds to to verify and realize oh this is probably a false positive thanks for letting me know of course ChatGPT wants to be overly safe then sorry but I’ll disregard that so that’s where the human element comes in. And in terms of a negative predictive value, so if ChatGPT says the patient has no contraindication, you’re okay to give thrombolysis if everything else checks out, what is the probability that there is something missed? About 1%. So 99% of patients that ChatGPT says you’re fine to give truly do not have any contraindication in the chart. So that is fantastic. And as for cost and how long does it take? So it takes about 14 seconds for ChatGPT to analyze. Now that doesn’t consider the amount of time that you need to first extract the chart and then, you know, all the user interface stuff. It’s about 30 seconds on average, which is very, very fast. And it cost, if I remember correctly, less than $0.10. I believe it was something around $0.07. And we kind of did the math, like how much it would cost us over one whole year. And that’s Cleveland Clinic, which is a huge hospital. We cover 16 different hospitals, less than $500, U.S. dollars, which is very, very cheap. And I think it’s going to be very cost-effective. And our next step is going to try to design a trial that’s going to compare people who use this AI tool and those who do not to see if it impacts on the safety, impact on the speed of treatment delivery and a patient outcome. And we give it a name we haven’t trademarked it so please don’t use it but we are calling it NeuroGlimpse because it’s a collaboration between the ophthalmologist and myself as a neurologist and it’s a glimpse into the chart into the patient’s brain and hopefully that will make stroke treatment a lot safer and faster and better for the provider and referred patients.

This transcript is AI-generated. While we strive for accuracy, please verify this copy with the video.