AAN 2025 | A simulated randomized controlled trial for epilepsy

Last year we published a study where we looked at the concept of putting together a fake clinical trial and the idea of the clinical trial was not that you know we need to fake clinical trials but it is now possible relatively easy to ask an AI please write the clinical note as if a patient had shown up with this in this characteristics. And it comes out in a style that is recognizable as a clinic note. And you can ask for different styles. So we said, okay, use multiple different styles and make a clinic note as if someone had just been in the beginning phase of a clinical trial where we’re just monitoring them and seeing how many seizures they have and what they’re complaining about. And then the second phase after they’ve been on an investigational drug or a placebo. And so two notes per patient, as if they’re either given a placebo or given drug. And the seizures that they report come from our realistic seizure diary simulator, which is a completely different project, but now accurately represents what kind of features you would expect in a seizure diary. So mixing these things together, as well as one more element, which is the symptoms that people will report in an actual clinical trial. So we took a recent clinical trial with cenobamate, which is an anti-seizure medication that’s relatively new. And we said, OK, you know, 6 percent of people complained of this and 8 percent complained of that in the placebo arm and 12 percent of this and 14 percent of that in the drug arm. So we use those percentages to randomly assign the symptoms to our simulated patients. And the idea there is that now the drug and the placebo groups act just like what was previously reported. but the whole thing is simulated notes now. And so we’ve got a collection of before and after for two groups. And so a total of 480 clinic notes for a total of 120 patients per group. And that way we’re kind of similar to this trial. And we said, okay, here’s the question for the AI. Can you please take a look at these 480 notes and figure out what is the deal with this drug? Is it effective? Is it not? What is the deal with the placebo? Is it effective? Is it not? And what symptoms are you seeing? And we asked it not precisely that way, but very similar to what symptoms are you seeing? So we didn’t say like, how many headaches are you seeing and how many this and how many that? Instead, just what are you seeing, which is different from before the investigational drug was started? And the reason to ask it in this open-ended way is that previous systems, previous statistical models and so forth, you can say like, is the drug good or bad? Because you have numbers and you’re taking a table and you’re asking a very precise question. But now when we have these flexible large language models, now you can say like, what did you find? And we’ve learned that if you ask, what did you find to one person, then you get the truth mixed up with lies. And we call these lies hallucinations or confabulations, but fake things that the AI will tell you that’s not true. And despite all the progress in amazing tools with AI, there’s still these hallucinations or conflagulations. So you can’t trust an AI completely when you ask it for what do you see with when you’re talking about one person. But I reasoned that if I’m doing this across hundreds of patients, then the little lies will not accumulate because they’ll be random. But the large amount of truth will accumulate and they won’t be random. And so I would be able to get this thing called inductive reasoning. We’ll be able to, across patients, learn about the drug even if individual mistakes are made at the one-person level. So we asked, does the AI understand from 480 notes the drug and the placebo? And we compared it to having an actual human comb through each of these 480 notes and make a little chart. And we took the chart from the AI and the chart from the human. And we then did standard statistics. And we said, okay, what, You know, how good is this drug? How good is the placebo? And what symptoms are we seeing a little bit of a lot of and how much and so forth? And the amazing thing is that they match extremely well. The efficacy of the drug and the symptoms that you see match up within about 3% of each other from what the human found and what the AI found. What does that actually mean in practice? It means that if you know that a drug causes 10% of people with headaches, then the AI could have figured that out plus or minus a little bit. Maybe they’ll call it 13% or 7%, but it’ll still be in the right place. and when it’s a drug that we don’t know about or when it’s a drug that’s new to market and we don’t yet know what is the practical experience then it means that the AI can inductively learn through the experience of actual patients in actual hospitals what the drug is doing. And so even if the fda didn’t yet know that this drug causes itchiness of the left nostril we can know that really quickly and that capability I think is something that we desperately need we need to learn in practice from our drugs and from our diagnostics and from our prognostics and so on. We need to learn those things and we can’t do that yet, but now we are beginning to obtain tools that can do that. That’s what our study was really about.

This transcript is AI-generated. While we strive for accuracy, please verify this copy with the video.