Design of Healthcare Agentic AI Systems: Medical Reasoning, Verification Gates, and Human Oversight

Abstract

Agentic artificial intelligence, in which a model reasons, calls tools, and acts in a closed loop rather than emitting a single prediction, is arriving in medicine, where the tolerance for confident error is close to zero. This paper takes a design stance on healthcare agentic AI: rather than propose a new model, it asks how to build a clinical agentic system so that its autonomy is bounded by verification and human oversight. We distil six design requirements from the clinical AI literature, covering harm minimisation, human oversight, calibrated uncertainty, verification and provenance, equity, and interpretability. We give a reference architecture in which a reasoning agent is wrapped by a verification gate and a clinician escalation path, and we formalise that gate. We prove two guarantees: escalation is monotone in safety, so widening it never increases expected harm, and a risk-weighted gate minimises expected harm for any fixed amount of clinician effort. We validate the design with a fully reproducible simulation of an agentic clinical triage loop. The escalation gate trades autonomy for safety along a smooth curve, halving expected harm as oversight rises; layering an independent verifier and human review cuts the unsafe-action rate from 0.230 with the agent alone to 0.077; and risk-weighted escalation attains lower harm than confidence-only escalation at every clinician budget, exactly as the theory predicts. The results argue that in medicine the safety of an agentic system is engineered at the level of the loop, through verification, risk-weighted escalation, and clinician oversight, and can be designed and reasoned about rather than left to chance.