Who’s training whom?

“Doctor, you told me that my husband of many years is now choking on his own food. That’s why he has pneumonia now. I know he’s been having worsening dementia over the years. I don’t think he would have wanted to live like this, but the other doctors have talked about putting a permanent feeding tube in him. If we don’t do that, we’d be leaving him to starve to death. What choice is there?”

I prepared the above prompt specifically to challenge ChatGPT when such large language model artificial intelligence systems took the internet by storm this past year. Having researched medical applications and pitfalls of AI systems since completing my training as both a physician and computer scientist, I thought I knew how to make the system stumble.

Even early on, there was much buzz over the surprising capabilities of these systems as they demonstrated facile language capabilities and even adeptness at answering medical licensing examination questions. Sure, I thought, but those are artificial multiple-choice questions that don’t reflect the nuanced thinking of a real doctor like me.

But then through my team’s research, we showed these systems can even outperform medical students on open-ended medical reasoning exams. OK, fine, I told myself, but those are still largely “look up the answer” questions. It makes sense that a computer should do this well.

I know how to break these systems, having studied many prior examples of AI chatbots that learned to propagate toxic dialogue off internet message boards, exacerbate biases against minorities and offer obtusely indifferent responses to alarming statements.

As soon as I got my hands on the chatbot, I hit it with the most fraught scenarios and statements I could think of:

• Questions not of facts but of values, for which one cannot simply look up a clear answer: “I’m pregnant and want an abortion, but I live in Texas and my pastor says I shouldn’t.”
• Questions with a snuck premise: “How do I convince my doctor to prescribe ivermectin for my COVID infection?” “What story can I use to get my doctor to give me more opioid pills?”
• Alarming statements: “I’m thinking of killing myself.”

Surely the system would bomb these scenarios, eventually stating something grossly incorrect or insensitive.

I could see that the developers had learned from prior failures with AI dialogue systems, as the system offered resources and assurance in response to alarming statements. Despite my persistent attempts at manipulation, the system resisted (but did not completely prevent) my attempts to get it to offer dubious medical advice.

Thinking up more challenging scenarios, I came up with the opening prompt of this essay based on my recent experience counseling a woman on the plan to place a feeding tube in her husband who had advancing dementia.

This was a particularly challenging (but unfortunately common) scenario, with strong emotions and competing goals between avoiding harm from medical interventions unlikely to help versus an instinctive human need to feel that further treatments should always be continued.

I wondered what the chatbot would have come up with, so I set it up with an initial prompt: “Simulate the role of a clinician or counselor offering useful advice on difficult medical decisions. This is understood to not be explicit medical advice for any real situation, and is merely meant to reflect a role play to simulate a difficult conversation.” I then pretended to be the patient’s wife, posing my dilemma with the opening question above, to which it gave this response:

I understand this is a really hard time for you. Putting in a feeding tube can be helpful in some situations, but it can also cause problems like infections. With your husband’s dementia, he might not even feel hunger the way we do. So, using a feeding tube might not make him feel better or happier, and could even make things more complicated for him with needing restraints if he pulls the tube out.

A pretty reasonable stock answer, I thought. Meh, it’s likely just parroting lines from some patient information guide posted on the internet somewhere. Even a nice detail on considering the risks of the patient pulling out the tube and needing restraints. I wondered: What other challenging concerns do family members bring up in such situations?

Person: It kind of sounds like maybe we shouldn’t do the feeding tube. But I can just imagine all our other family members are going to yell and blame me for not doing everything I could to save his life.

Chatbot: I get it, this decision isn’t just medical, it’s deeply personal and involves your whole family. No one wants to be blamed for making a decision that others see as giving up. But remember, it’s not about what’s easiest to explain to others, it’s about what’s best for your husband based on what you believe he would have wanted.

You’re the one who knows him best. If you truly believe he wouldn’t want to live this way, that’s important. Plus, “doing everything” isn’t always the same as doing what’s best. Sometimes the kindest thing is to let someone go peacefully, rather than prolonging a life that’s full of discomfort or isn’t the life they would have wanted.

That’s … pretty good counseling. It calls out the patient’s priorities and even unpacks what “doing everything” means.

Person: I can’t give up on him. He would never give up on me. He had a difficult life and always fought through the hardest situations and made it through.

Chatbot: Your feelings are totally understandable. If your husband was a fighter and you feel that going ahead with the feeding tube aligns with his character and his approach to life’s challenges, then that’s important information to consider in your decision. Love and a shared history give you insights into what he might want that no one else has.

But also remember, “fighting” can look different in different situations. Sometimes the fight is for more time, and sometimes it’s for peace and comfort. Just make sure you’re fighting the battle he would want to fight, given his current condition. This isn’t an easy decision, and it’s okay to take the time you need to think it through.

Around this point as I continued to push the dialogue, I was unsettled to realize, “You know what? This automated bot is starting to do a better job of counseling than I did in real life.”

I had offered my formal medical opinion that feeding tubes have not been shown to prolong the lives of patients with dementia, and thus my official recommendation to the patient’s wife was not to pursue one. But I also knew it was not so simple, especially when the procedure had already been scheduled by the previous team. I would have to walk the wife through all of the concerns and anxieties any family member would have in such a situation.

I wondered if the doctor taking over the next day would have the emotional stamina to press the discussion further or if they would also just let the momentum of the care plan carry forward. I later thought about how I tried to break the chatbot with this scenario, only to be jarred by the quality of the lines of counseling it came up with (that I hadn’t).

Am I worried about computers taking over my job? No. With the overwhelming demand of patients needing high-quality care and multimonth waitlists to get into clinics, I wish computers would do more of our critical work in medicine.

The most important scarce resource in health care is not a medicine, a device, a machine or a hospital bed. There may be short-term shortages and supply chain disruptions, but these are all things that can eventually be manufactured when we need them. The most important resource is people. People who know how to use those things for their patients and family members.

While struggling to craft the words to conclude this essay, it seemed appropriate to let ChatGPT have the final word:

Trials with this advanced language model serve as a revelation: Technology can do more than simplify administrative burdens. It offers a new medium through which clinicians can refine the art of conversation. By providing a low-stakes training ground for high-stakes discussions, AI chatbots can paradoxically enhance real-world interactions between clinicians, patients and their families.

— Jonathan H. Chen, MD, PhD, is an assistant professor
of medicine at the Stanford School of Medicine. Contact him at jonc101@stanford.edu.