MedEvoEval: Evaluating Continual Evolution of Doctor Agents through Simulated Clinical Episodes

Zhang, Hui

MedEvoEval: role-specific patient, examination and manager views support within-episode actions, clinical scoring, event logs and optional memory; cross-episode evaluation measures improvement, held-out transfer, update retention and backward degradation. — Within each episode, valid actions reveal clinical evidence and produce an auditable trace. Across episodes, the protocol evaluates how accumulated experience changes doctor-agent behavior. Vector PDF

Clinical episode demonstration

A recorded clinical episode demonstration (1 min 13 s). Open video

Abstract

Doctor agents are moving beyond single-turn answer generation toward evolving clinical decision systems that acquire evidence within episodes and reuse experience across episodes. Fixed-input medical QA and many interactive benchmarks provide limited support for evaluating how episode-level decisions interact with cross-episode experience. We introduce MedEvoEval, an executable longitudinal evaluation framework based on action-gated simulated outpatient episodes. Each case is converted into role-specific patient, examination, and manager views; evidence is revealed through valid actions, and structured traces link observations, actions, final outputs, scores, and optional experience write-back. Experiments show that these traces expose process costs hidden by final-answer scoring, characterize how MDT-style consultation reallocates resources, and support analyses of memory maturation, held-out transfer, update-stage response, and backward retention. On 200 held-out cases, the mature-memory condition with sequential write-back reduces interaction turns by 0.915 and improves composite utility by 0.0483 after Holm correction. MedEvoEval provides a concrete basis for evaluating experience-driven improvement, transfer, and retention. The accompanying artifact provides 700 processed episode records with provenance, role-specific schemas, and execution, scoring, and analysis code.

Using MedEvoEval

Run clinical episodes. The code includes the episode engine, role prompts, model configurations, scoring, and memory read/write logic.
Reproduce the reported analyses. Released result inputs and scripts generate the paper’s tables and vector figures without model calls.
Inspect the dataset. The release provides 700 processed case records, a 500-record stream and a 200-record held-out set, role-specific schemas, provenance, and process trajectories.

Citation

@misc{zhang2026medevoeval,
  title = {MedEvoEval: Evaluating Continual Evolution of Doctor
           Agents through Simulated Clinical Episodes},
  author = {Hui Zhang},
  year = {2026},
  eprint = {2606.28900},
  archivePrefix = {arXiv},
  primaryClass = {cs.AI},
  url = {https://arxiv.org/abs/2606.28900}
}