ESOC 2023 | Machine learning tool improves recognition of stroke during emergency calls
Jonathan Wenstrup, MD, PhD Candidate, Herlev Hospital, Herlev, Denmark, shares the findings of a retrospective study assessing the value of a machine learning framework for stroke recognition during medical emergency calls. Stroke is well recognized as a time critical emergency and thus, better recognition prior to hospital admittance may help to support early management decisions. Dr Wenstrup and his team developed a two-step machine learning framework, that transcribes call audio and predicts stroke risk based on the transcribed text. The tool was trained on over 1.5 million medical helpline calls, of which more than 7000 were stroke calls. The performance of the model was then tested on a further 340,000 calls, in comparison to healthcare telecommunicators. It was found that the machine learning tool was better at recognizing stroke than the call handlers, with a higher positive predictive value and a higher sensitivity. Dr Wenstrup emphasizes how such a tool could help call handlers during difficult calls, suggesting when stroke is more likely and helping them to learn less obvious signs of stroke, rather than as a replacement. This interview took place during the European Stroke Organisation Conference (ESOC) in Munich, Germany.
These works are owned by Magdalen Medical Publishing (MMP) and are protected by copyright laws and treaties around the world. All rights are reserved.
Transcript (edited for clarity)
What we wanted to do with this study was to see if we could if we could develop a supplementary supportive tool for dispatchers or call takers to aid them in detecting stroke on the phone because that has been shown several times to be a challenge. So what we did was, in Denmark there is a mandatory stroke registry that has to be filled out if a patient is admitted to a stroke unit for a stroke...
What we wanted to do with this study was to see if we could if we could develop a supplementary supportive tool for dispatchers or call takers to aid them in detecting stroke on the phone because that has been shown several times to be a challenge. So what we did was, in Denmark there is a mandatory stroke registry that has to be filled out if a patient is admitted to a stroke unit for a stroke. So there we have sort of a gold standard of whether or not patients have a stroke at all. And we took this registry and we merged that with the call registry at the Copenhagen EMS. That yielded us a data set which had all the calls made in the time period 2015 to 2021 and all the calls were then labeled as is this a call concerning stroke or not? And that dataset can be used then for training a machine learning algorithm because we reserved 2021 for just testing, so we didn’t use that for training. And then we gave the rest of the dataset, 2015 to 2020, to the algorithm. It’s more complicated than just giving it to it, but what you do is you ask it basically to find the difference between the calls that are stroke and non-stroke and then you run that a certain number of times and you see whether or not it has learned to detect the stroke calls. So that’s what we did. And then when we had that developed algorithm, we tested it on the 2021 data because if you test it on the same data you train on, you can’t be sure the results will work. It tends to overfit them. So you should test it on something that hasn’t seen before. So that’s what we did.
On the test set we had just over 1.5 million calls for training. Of these, 7370 calls were actually stroke calls. And for the test dataset, we had somewhat over 340,000 calls. Of these, just over 750 calls were stroke calls. So quite a robust dataset. But we looked at how the human dispatchers for the medical helpline were doing. That is the line thats sort of an out of hours line, but on 24/7. They found 52.7% of strokes. So that’s how many they detected correctly and they had a positive predictive value, that is, they were right about thinking something was a stroke 17.1% of the time. So they over triaged by a big margin. We actually want some over triaging stroke calls, of course, because you want to you want to catch as many strokes as possible and then there’s a bit more work for the doctors, but that’s okay. But for the machine learning platform, we had a 63% recognition of stroke calls, so more than 10% higher than the human dispatchers, and the positive predictive value of how right it is in thinking that something is a stroke was 24.9%, compared with the humans 17.1%. So not only do we detect more calls by 10%, more stroke calls, but we also have a better positive predictive value. So it’s not just because it guesses that more stroke and then, you know, then you’re right, just triple the amount of stroke you guess of course you’ll find more strokes. But no, it actually does perform better on both metrics, which was very important for us because that’s the only way these tools can actually be useful in real clinical practice, if they are not creating a massive amount of more work for the clinicians.
When looking at what’s been done to try to fix this lack of stroke recognition for dispatchers, which is a very hard job just on the phone, people are calling in, they don’t know what to say and they have to find all these different symptoms. You’ve tried more training for the dispatchers and stroke. Often you also try specific stroke scales which are supposed to be applied. But all of this requires extra effort from the dispatcher. So they have to realize it’s a stroke and start applying the stroke scale or they have to go to more courses. One of the good things about this is if you implement, it’s simply a warning that only appears to the to the telephone operator when the chance of stroke is high. Simply it can be a pop up or however you decide it, that says high chance of stroke, consider admitting to the hospital immediately, something like that. And doing this will free up the personnel from doing extra things, just aiding them. It’s not that it catches all strokes as it is, but it will aid them in it. So if they find a stroke that the model had not caught, the model doesn’t preclude them from sending them it strokes. It’s merely a supportive tool that doesn’t really require an extra effort on their part. So that’s one thing that’s good about it. The other thing is and that’s one of the reasons why it works is, the chance of stroke in our dataset in any given call is only 2.5 in 1000 calls. So that’s a very low chance of any call being a stroke call. A typical dispatcher probably has high two-digit number of calls a day perhaps but you’re not guaranteed that any stroke call inside you get any stroke calls inside a month even, so there’s limited practice for it. Whereas the algorithm, the framework is trained on all kinds of stroke calls. And it’s unlikely that any call taker has taken 7000 stroke calls. So it has the opportunity to learn from more and perhaps also learn the more tricky cases, the more difficult ones, the ones that aren’t conforming to the to the textbook or the course that we put our telephone operators through. So that’ll be able to aid them further and actually be a tool and they can also learn from it if it’s right, if it pops up they can think oh these symptoms are perhaps something else, then they can apply that the next time and then the framework and the telephone dispatcher can work sort of in tandem. So that’s some of the usefulness.
Read more...
