Popcorn in hand, I’ve been watching the specialized clinical vs generalist LLM model feud with some amusement. I recently built an automated e-Consult feature within our own software to produce initial cardiology consult notes that are (nearly) ready for sign-off. Building the RAG content aspect was straightforward - feeding cardiology guidelines, information from medication databases, all that is pretty routine. But optimization of the output to minimize the cognitive burden and review time for the reviewing clinician has been the hard part. For this to scale and meet our gross margin targets, we want the clinician to ideally spend less than 60 seconds reviewing and revising the material while maintaining impeccable quality. When I see the voluminous output from OpenEvidence and many foundation models, I shudder. And when I see the gap between this monstrosity of an output (see the screenshot from Feng et al.’s recent benchmark, where the models were presented a single routine question regarding yearly labs after bariatric surgery) and an actual, actionable, scalable plan, I pity the poor clinician reading this and having to operationalize it. This output is fundamentally incompatible with our efficiency targets and workflows.
Figure 5, reproduced from “Expert Evaluation of Clinical AI Tools on Real Point-of-Care Clinical Queries” by Feng et al., licensed under CC BY 4.0. Cropped from the original.
Taking a step back, I find the topic of how to achieve highly efficient human-AI collaboration, particularly in medicine, where the clinician’s license is on the line and time is at a premium, fascinating. Yet I haven’t seen many others focus on it. It seems that many AI-clinician hybrid solutions have modeled their approach after the resident/fellow/attending model, where, after seeing a patient, the trainee provides a (hopefully) terse summary of the encounter, and the attending physician perhaps asks a few probing questions and then signs off. But my experience is that efficiency in this model predominantly comes from your trust in the person presenting to you, which is usually based on some historic or sample-based impression of their intelligence, how well they seem to be versed in the literature, and your awareness of their level of training. If you don’t trust them, you’ll likely redo everything. But if you are confident they’ve attained some experience and have a baseline level of intelligence and critical reasoning ability, grounded in medical evidence, it’s unlikely they will make a critical-reasoning error. So the summary you’re willing to accept from them is terse and likely incomplete, but you can still sleep well at night knowing you’re signing off on a decision that’s (likely) well-grounded.
In contrast, AI models, which are inherently jagged, prone to unpredictable reasoning failures, and come with a likely intrinsic tendency to hallucinate, cannot give you that same confidence. You cannot safely extrapolate from a correct citation in one area or excellent reasoning in another to being confident that the recommendation provided in another setting is correct. And reading a voluminous passage and double-checking every reference and matching to actual patient details is a non-starter. I suspect that if you’re operating in a low-risk area - short-term prescription refills, low-risk primary and urgent care, algorithmic mass-prescribing of a small number of meds - you perhaps don’t worry that much and largely glance at any AI summary you’ve been given and just sign off (though I’ve written about my own trepidation of the refill area in particular).
But that’s not the space we personally operate in. We’re squarely making complex decisions regarding high-complexity patients and reside in high-stakes situations. Given the LLM failings listed above, our philosophy has been that all recommendations must emerge from a deterministic engine based on computable functions, and the LLM’s role in that system is to provide inputs to the engine based on the interpretation of the conversation or ingested text/reports. As one can imagine, the scope of that engine and the complexity of its reasoning are enormous. But it provides a surface for hazard-based deterministic testing across the full range of cardiology decision-making.
This ultimately comes back to how you achieve efficiency and quality in an AI-human collaborative workflow, and give confidence to the clinician whose license is on the line. There’s no doubt that presenting that material to the clinician, including the ability to walk it back to atomic inputs and underlying evidence, is critical. But, in the same way that efficiency in the attending-trainee model primarily arises from working with trusted residents and fellows, efficiency in this AI-clinician collaborative framework ultimately has to stem from confidence in the properly designed, exhaustively tested deterministic engine.