The AI Clinic · Issue 015AI return on investment
Pick the number before you buy.
Big health systems now want proof that AI pays for itself
Five stories made the cut this week. Most of them circle a question I think small clinics skip: how do you know the AI tool you're paying for is working?
- Bain and KLAS surveyed more than 300 health care executives, and some want a return of 3 to 3.9 times what they spend on AI.
- A PLOS Digital Health study found smaller, more private models can get close to larger ones, but only with the right setup.
- In a survey of 552 US medical students, 87% used generative AI, but only half felt confident judging what it gave them.
The Big Story
Big health systems now want proof that AI pays for itself
Bain & Company and KLAS Research surveyed more than 300 health care executives. Nearly 95% of providers and payers called health IT a top strategic priority, but they want specific use cases with results they can measure. For AI, some are looking for a return of 3 to 3.9 times what they spent.
On the provider side, revenue cycle and patient access tools drew the most attention. Three in four providers were optimistic about AI, especially ambient documentation and chart summarization. For clinical tasks, over half would rather have a basic tool inside their EHR than a better one from a third party. Payers, meanwhile, are spending on utilization management and claims processing. The report's authors say early adoption ran on "intuition and optimism," and that the bill is coming due.
Mark's read: If a hospital with an analytics team wants three times its money back, a five-therapist clinic should at least know whether it's getting its money back at all. Pick one number before you buy, like minutes of charting after the last patient or notes closed the same day, and measure it before and after. And watch the other side of the table. Payers are buying tools built to review your claims.
In the Clinic
Cheaper, more private AI isn't automatically as accurate
A PLOS Digital Health paper tested Google's Gemma and MedGemma models, from 1 billion to 27 billion parameters, on medical question sets. With longer reasoning, reference material, or medical fine-tuning, a smaller model could land within 1 to 4 percentage points of a larger one. Forcing longer reasoning took MedGemma 27B from 58.1% to 80.1% accuracy. Once good reference material was already in front of the model, though, reasoning past roughly 128 to 256 tokens made it worse by 7 to 13 points.
Mark's read: A vendor selling "on-device" or "private" AI is making a privacy claim, and that tells you nothing about accuracy. Ask which model runs under the hood, how it was set up, and what it was tested on. This study used exam questions, not your patients, so use it for the questions it gives you, not as a scorecard.
The Rounds
How do you test AI that acts on its own?
A letter in Annals of Biomedical Engineering points out that we test clinical AI as if it gives one answer for a clinician to check. Agentic AI doesn't work that way. It looks things up, uses other software, and acts on the results before a clinician sees anything.
The authors name four problems, including errors that pass quietly from one step to the next and oversight that checks the conclusion instead of the process. Their fix is a "clinical testing harness" with edge-case scenarios, a record of every step, and tests of when the system hands off to a human.
Mark's read: This is an argument, not data. I like where it lands anyway. Medicine already vets people with simulation, credentialing, and M&M review. An AI agent that books, bills, or messages your patients should clear at least the bar you set for a new hire.
Academic Corner
What med students really do with AI
A survey in The Clinical Teacher reached 552 US medical students from at least 37 schools. Of those who answered, 87% used text-based generative AI and about three in four used AI tools at least weekly. Only half felt confident judging what the AI gave them, and just 12.2% used a tool their school provided. Nearly two-thirds wanted more training or clearer guidance.
Mark's read: It's a convenience sample, and the authors say plainly it isn't a national estimate. I'd still bet a DPT classroom looks similar. If your AI policy is one line in the syllabus that says "don't," your students are already past it. Teach them to check the output, because that's the skill half of them say they don't have.
Academic Corner
Editors are asking authors to explain their own papers
Andy Stapleton's weekly academia roundup covers journal editors questioning the authors of suspected AI-written papers, plus the running argument over where using AI ends and cheating starts.
Mark's read: The fair test is whether the author can defend every sentence and every citation in the paper, however much AI helped write it. I'd hold a student's case write-up to the same standard.
Try This
Put one number on one AI tool
Pick one AI tool your clinic pays for and write down what it costs each month. Then pick the one thing it's supposed to change, like minutes spent on notes after the last patient, and track it for two weeks. If you can't name that number yet, start there before you renew.
The last word
What AI tool in your clinic would be hard to justify if someone asked you for the numbers? Hit reply and tell me.
Get the next issue.
One email a week. The AI that matters for clinical work, in plain language.