T
he ethical guidelines laid out in the Hippocratic Oath nearly 2,500 years ago are about to collide with 21st century artificial intelligence (AI).
AI promises to be a boon to medical practice, improving diagnoses, personalizing treatment, and spotting future public-health threats. By 2024, experts predict that healthcare AI will be a nearly $20 billion market, with tools that transcribe medical records, assist surgery, and investigate insurance claims for fraud.
Even so, the technology raises some knotty ethical questions. What happens when an AI system makes the wrong decision—and who is responsible if it does? How can clinicians verify, or even understand, what comes out of an AI “black box”? How do they make sure AI systems avoid bias and protect patient privacy?
In June 2018, the American Medical Association (AMA) issued its first guidelines for how to develop, use and regulate AI. (Notably, the association refers to AI as “augmented intelligence,” reflecting its belief that AI will enhance, not replace, the work of physicians.) Among its recommendations, the AMA says, AI tools should be designed to identify and address bias and avoid creating or exacerbating disparities in the treatment of vulnerable populations. Tools, it adds, should be transparent and protect patient privacy.
None of those recommendations will be easy to satisfy. Here is how medical practitioners, researchers, and medical ethicists are approaching some of the most pressing ethical challenges.
Avoiding Bias
In 2017, the data analytics team at University of Chicago Medicine (UCM) used AI to predict how long a patient might stay in the hospital. The goal was to identify patients who could be released early, freeing up hospital resources and providing relief for the patient. A case manager would then be assigned to help sort out insurance, make sure the patient had a ride home, and otherwise smooth the way for early discharge.
In testing the system, the team found that the most accurate predictor of a patient’s length of stay was his or her ZIP code. This immediately raised red flags for the team: ZIP codes, they knew, were strongly correlated with a patient’s race and socioeconomic status. Relying on them would disproportionately affect African-Americans from Chicago’s poorest neighborhoods, who tended to stay in the hospital longer. The team decided that using the algorithm to assign case managers would be biased and unethical.
“If you were to implement the algorithm in practice, you would have the paradoxical result of allocating more [case-management] resources to the whiter, more affluent population,” says Dr. Marshall Chin, an internist and professor of healthcare ethics at UCM.
The data analytics team consulted with UCM’s diversity and equity group and removed ZIP code as one of the factors. The algorithm is still in development, and a new model hasn’t yet been tested.
“How do we, as machine-learning practitioners, handle societal problems while trying to make accurate, actionable models?” asks John Fahrenbach, a UCM data scientist who worked on the algorithm. His answer: Seek outside expertise, such as the UCM diversity group.
“How do we, as machine-learning practitioners, handle societal problems while trying to make accurate, actionable models?”
This case points to a weak spot in AI-based healthcare tools: The data used by the algorithms can often reflect existing racial or gender health disparities. Left unaddressed, this could contribute to perpetuating bias and lock in existing inequalities in healthcare.
Chin and his colleagues, in a recent paper that mentions the UCM case, cited a 1999 study of the use of cardiac catheterization—in which a thin tube is inserted to diagnose heart conditions—for patients with chest pain. The study suggested that African-American women received the technique “at inappropriately low rates" compared to white men, Chin wrote. A machine-learning algorithm trained on that data, he concluded, would recommend the treatment for African-American women less than needed.
Bias can also affect the treatment of rare or novel diseases for which there is limited data. An AI system might infer a proposed treatment is futile without taking into account a patient’s individual circumstance.
Danton Char, an assistant professor of anesthesiology at Stanford University, notes in a recent paper on machine learning that it is common for doctors to halt care for patients with severe brain injury or with extremely premature infants, whose chances for survival are low. A machine-learning algorithm might conclude that all such cases are invariably fatal and recommend withdrawing treatment, even if an individual prognosis were favorable.
The “Black Box” Problem
Another ethical challenge is that it isn’t always possible to know how an AI system reached its results—the so-called black box problem. Advanced machine-learning technologies, which can absorb large quantities of data and identify statistical patterns often without explicit instruction, can be especially hard to verify. Physicians who blindly follow such an inscrutable system might unintentionally harm patients.
“It’s often very difficult to understand what the ‘thought’ process of the algorithm is,” says Eleonore Pauwels, a research fellow in emerging cybertechnologies at the United Nations University’s Centre for Policy Research.
“It’s often very difficult to understand what the ‘thought’ process of the algorithm is.”
A 2015 study highlights this problem. In the study, researchers compared how well different AI models predicted mortality risk for pneumonia patients. Those at greater risk would be admitted to the hospital, while lower-risk patients could be diverted to outpatient care.
One of the models, a “rule-based” system whose decisions were transparent to researchers, predicted that patients who had both pneumonia and asthma had a better chance of surviving than those with just pneumonia—a counterintuitive result, since someone suffering from both ought to have been at greater risk of dying. What researchers knew, but the algorithm didn’t, is that patients with both diseases had been admitted directly to an intensive-care unit, which improved their chance of survival. Relying on the algorithm would have meant the most critical patients wouldn’t have received the care they needed.
Another model that used a neural network, a type of machine learning, produced more accurate results, but its reasoning was opaque. Richard Caruana, a researcher at Microsoft Corp. and the lead author of the study, and his colleagues concluded the neural-net model was too risky to take to clinical trials since there was no way to tell if it was making a similar mistake. “What really scares me is, what other things like this might the neural net have learned that the rule-based system missed?” he says.
Who’s Responsible?
The AMA’s Principles of Medical Ethics advise physicians that responsibility to the patient is “paramount.” But where does responsibility lie when AI enters the equation? The answer is still being worked out by ethicists, researchers, and regulators.
One problem is that AI multiplies the number of people who contribute to a patient’s care and brings in those, like data scientists, who aren’t traditionally bound by medical ethics. Also, as the black box problem shows, it isn’t always possible to know exactly how an AI system made a diagnosis or prescribed a treatment.
This isn’t an entirely novel situation. Char, the Stanford anesthesiologist, compares AI to a prescription drug. While clinicians can’t be expected to understand every biochemical detail of the drugs they prescribe, Char says, they at least need to know that they’re safe and effective, based on their clinical experience and knowledge of the medical literature. As for AI systems, Char says, he wouldn’t use one unless he’s confident after careful research that it’s the best option. “You don’t want to put anyone’s life at risk because you haven’t vetted the tool you’re using,” he says.
Preserving Patient Privacy
The AMA warns that healthcare AI must safeguard the privacy and security of patient information, highlighting what has been a cornerstone of medicine since Hippocrates: a commitment to doctor-patient confidentiality.
But to make accurate predictions, machine-learning systems must have access to enormous amounts of patient data. Without an individual’s medical records, AI can’t provide accurate diagnoses or prescribe useful treatments, especially as the technology enables more personalized medicine. What’s more, if millions of patients withhold their medical data, critical public health trends might be missed, and everyone loses.
One potential solution is technical: AI systems could protect patient privacy by removing individual identifying information from medical records used for research or to train machine-learning systems. However, a recent study led by researchers at the University of California warns that current anonymization techniques don’t guarantee that the data can’t be used to identify individual patients, though more sophisticated ways to collect data could be developed that better protect privacy.
Regardless of technical capabilities, medical experts propose rethinking the whole idea of patient confidentiality. As the healthcare system becomes more complex, they say, more people will have a legitimate need to access sensitive patient information. “The implementation of machine-learning systems will therefore require a reimagining of confidentiality and other core tenets of professional ethics," writes Char in his paper.
In practice this means winning patients’ trust—and their informed consent. They will need to know in detail how the information will be used, and whether the data will benefit them or only future patients.
If patients better understand how AI can improve individual and public health, AI advocates say, they might be willing to let go of traditional notions of patient privacy. “The confidentiality itself is never absolute,” says Nathan Lea, a senior research associate at University College London’s Institute of Health Informatics. “We can’t use confidentiality as an excuse not to do things.”
Credit: Django/GettyImages