What a healthcare AI demo doesn’t answer
A founder showed us a medical-records app that was ChatGPT on a screen share. The demo worked, but it left four questions open, and a team must answer them before it uses the records of real patients.
Over the past year, a few founders have proudly shown us demos they vibe coded over a weekend.
One that sticks with me was built by a healthcare professional. You could upload medical records, and the app would audit them and check whether anything was missing.
The promise was compelling, and he was optimistic about how much time and money it could save once “all doctors start using this app” (his words, not mine).
Demo day arrived, and I was eager to see what all the hype was about. The screen share took a moment to load, and then I saw it.
“ChatGPT…?”
I thought, Maybe he’s sharing the wrong screen.
He wasn’t. That was the demo.
And, annoyingly, it worked surprisingly well.
It could look through records quickly and flag things that seemed to be missing.
I thought, Okay, maybe this is actually pretty cool.
But a useful demo answers one question: Can AI help with this task?
It doesn’t answer the questions a founder or CTO needs answered before putting it into a real healthcare workflow.
I had a few:
- Where does the patient data go?
- What exactly counts as “missing”?
- What happens when the AI is wrong or unsure?
- What happens when a whole care team starts using it every day?
Where does the patient data go?
Before real records go anywhere near the product, the team needs to understand the full data path.
Where does the information go when it’s uploaded? Which services process or store it? What ends up in logs? Who can access it, how long is it kept, and how can it be deleted?
The AI provider is only one part of that picture. The storage, integrations, support tools, and analytics around it matter too, and sometimes more than the provider itself, especially as you scale.
Furthermore, these aren’t questions to answer with “it’s encrypted” or “we host the model ourselves.” Bring in the right security and compliance people early, and make sure you understand what the product actually does with the data.
What exactly counts as “missing”?
“Missing” sounds simple until you have to define it.
Which requirements is the app checking against? What if a document is present but the system can’t read it? What if the information appears in a different place or format? What if the app flags something that isn’t actually required?
A product needs a clear source of truth for what it’s checking and it also needs to distinguish “this information isn’t here” from “the system didn’t find it.”
Otherwise, the person using the tool may have to redo the very review the app was meant to help with.
What happens when the AI is wrong or unsure?
The product shouldn’t just give an answer, but instead it should help the user decide whether to trust that answer.
Can it show what it checked and why it flagged something? Can a person correct the result? What does the system do when it can’t confidently tell whether something is missing?
In a healthcare workflow, “I’m not sure” needs to lead somewhere useful: a review, a correction, or a fallback that lets the person finish the task.
What happens when a whole care team starts using it every day?
A demo usually has one person, a handful of records, and someone watching closely, whereas a real product has a team using it in the middle of their normal work.
Who is allowed to see which records? Where do the results go? Does the tool fit with the systems the team already uses? What happens when an integration is down, more users arrive, or the AI provider changes something?
And does it still save time when the team has to review exceptions, fix mistakes, and manage the new workflow?
That’s what “scale” means here: not just handling more uploads, but working reliably for more people, across the real situations they run into.
Can the right people use it with the right data, check its work, and keep the workflow moving when something goes wrong?
That’s where the real product begins.
