Pingkun Yan, like many in biomedical research, is a firm believer in the power of artificial intelligence (AI) to improve and potentially revolutionize healthcare technology. On the other hand, he notes, he’s also a realist: For all the hype to become a reality, challenges must be overcome first.
“A lot of people think AI is so capable and that it can solve all of our problems,” says Yan, an assistant professor of biomedical engineering at Rensselaer Polytechnic Institute in Troy, New York. “That may be true in a couple of decades, but right now, it still has weaknesses—we’re really not there yet.”
Yan’s work focuses on what he considers the biggest weakness threatening AI: the technology’s vulnerability to deception and manipulation from “adversarial machine learning.”
Machine learning (ML), he explains, is the process through which an AI-based computer can become “smarter” over time as it incorporates new data into its algorithms. Adversarial ML involves attacks that lead computer systems astray by introducing data they weren’t meant to see.
In healthcare, Yan explains, the top use case for AI is currently in medical image analysis. Radiologists, for example, can use AI-powered applications to help identify and analyze potentially cancerous tumors, and the technology has been touted as a way for the profession to streamline workflows and improve patient care. As of September 2021, the U.S. Food and Drug Administration cleared more than 100 AI algorithms for use in medical imaging devices. The world market for AI in the imaging sector is now approaching $1 billion annually.
An emerging tech threat
Adversarial ML poses a threat to medical imaging and healthcare because it could cause AI-based machines to deliver faulty results, Yan explains. A hacker, for example, might choose to attack a system during its developmental, algorithmic training phase; or they could inject spurious data at the time the system is interpreting a real medical image. Either way, he notes, the output from the affected system could be corrupted and thereby become potentially misleading.
“To the human eye, when you change an image slightly, you usually won’t notice it at all, but that’s not the case with an AI algorithm,” says Yan.” Even a subtle perturbation can lead to mistakes, like missed or entirely wrong diagnoses.”
To the human eye, when you change an image slightly, you usually won’t notice it at all, but that’s not the case with an AI algorithm. Even a subtle perturbation can lead to mistakes, like missed or entirely wrong diagnoses.
—Pingkun Yan, assistant professor of biomedical engineering, Rensselaer Polytechnic Institute
His team’s research aims to develop AI techniques that can guard against adversarial attacks, but which also allow the imaging algorithms they’re protecting to carry out their designated tasks. Funded in part by a $550,000 grant from the National Science Foundation, the project aims to “build more robust AI systems without sacrificing accuracy,” Yan explains. “We want to make sure they operate as intended—that they deliver consistent, stable performance.”
Yan says his research is an acknowledgment of just how far AI and ML have come in healthcare. “When I was in school for my Ph.D.”—in electrical and computer engineering in the early 2000s—“this was still on the edge of the medical imaging analysis community,” he recalls. “But now, we see that AI/ML is everywhere, so it’s important for us to deal with this very dangerous problem.”
Developing solutions to “stay ahead” of attackers
The team’s work, Yan explains, will ultimately build on a growing body of cybersecurity research on methods for mitigating adversarial attacks. Solutions to date, for instance, include exposing systems and ML algorithms to mock attacks early in their development so they learn to recognize adversarial data; and making systems harder to penetrate by swapping out the algorithms they depend on with new ones that attackers have never seen before.
So far, there have been no documented cases of adversarial attacks on the medical imaging industry, but that hasn’t stopped researchers from speculating why such attacks are likely to take place in the future. A 2019 paper in Science, for example, posited that attackers might try to manipulate images to convince insurance companies that bogus claims are valid. And another recent study in Medical Image Analysis said that adversarial attacks “could be used as a tool to manipulate [imaging] systems supporting insurance, clinical, or drug/device approval decisions.”
The way Yan sees it, it’s not whether these attacks will happen, it’s only a matter of when. “It’s been shown through research that these kinds of attacks are possible,” he says. “And we know that when something’s possible, there are people out there who are going to try it. What we want to do is stay one step ahead of them and develop strong defenses before they can do damage.”
His team’s project just started, Yan notes, and he can’t share too many details about the tools they’re developing until they’ve had the chance to publish the work first. He does say, however, that one idea they’ve had is to make ML algorithms more resistant to attackers by exposing them to “data that are different from the training set.”
Today, we have the compute power and the sophistication to use algorithms that weren’t possible a few years ago. But the attackers, they also have access to a similar stack of technology, so they’re adapting, as well.
—Rahul Kashyap, cybersecurity expert, Arista Networks
ML models “need to be trained with a lot of data, and are thus ‘data-driven,'” he explains. When you add new data that vary from the training data—say, data from different racial groups or geographical regions—your model’s performance tends to degrade and become more vulnerable to adversarial attacks. Their idea is somewhat akin to training an athlete to better handle hot weather by holding their workouts in an over-heated room. “We’re working on new methods to more efficiently leverage the existing data by understanding the invariant connections between the data to improve the model robustness,” he says.
Rahul Kashyap, a cybersecurity expert at Arista Networks, agrees with Yan that adversarial ML is a growing problem that must be handled swiftly and decisively. He also sees it as a threat spanning almost every industry—from healthcare to retail to cybersecurity itself—that’s emerged as a result of what he calls the “AI mega-wave.”
“Today, we have the compute power and the sophistication to use algorithms that weren’t possible a few years ago,” he says. “But the attackers, they also have access to a similar stack of technology, so they’re adapting, as well.”
In time, Kashyap predicts, adversarial ML will become much more common, and as it does, manufacturers of AI-powered devices will have to adopt defensive solutions like those developed by Yan and his colleagues.
“I believe we’re not far away from the day when vendors will have to certify that their model has X chance of being affected by adversarial AI,” he says. “People will want to use these technologies, but they’ll also want to know they can trust their algorithms.”