MedClaw: Heuristic Agent Harness
for Long-Horizon Surgical Video Reasoning

Improving by evolving context rather than optimizing weights.

Yingying Fan1,2,*, Penghui Du3,*, Leyan Zhu3, Runze He3, Zimeng Wu3, Yuxuan Zhang5, Liang Chen3, Jiahao Xie3, Jiangtang Wang4, Shuai Shao4, Anchao Yang6, Yutong Bai6, Yan Wang1

1Beijing Jiaotong University   2University of Maryland, College Park   3UniPat.ai
4Suzhou Institute for Advanced Research, USTC   5University of British Columbia
6Department of Neurosurgery, Beijing Tiantan Hospital, Capital Medical University

* Equal contribution  ·  Corresponding authors: Anchao Yang, Yutong Bai, Yan Wang

Abstract

Understanding tens-of-minutes surgical videos requires long-horizon temporal reasoning: answering what happens before, after, or across stages of a procedure by grounding the question in visual evidence spread across time. Existing approaches handle this poorly. A one-shot vision–language model compresses the whole procedure to fit its context window and loses the detail a “before” or “after” question depends on, while video agents that train the model where to look are data-hungry and transfer poorly to out-of-domain surgery.

We build an agent harness that separates reasoning from perception and improves by evolving context rather than optimizing weights. A text-only orchestrator plans which evidence to gather and issues an auditable sequence of tool calls, while frozen vision–language sub-agents execute each call over the pixels — viewing, cropping, inspecting frames, and retrieving external knowledge. We further propose a gradient-free, reward-gated Heuristic Skill Distillation loop that mines the agent's own low-scoring traces and keeps a candidate skill only when it raises a validation reward, yielding reusable retrieval skills, notably directed re-look. Growing an external skill library rather than tuning weights, the loop adapts from only about one hundred labeled examples.

To evaluate this agent we introduce MedClawBench, a de-leaked, doctor-grounded benchmark of 1,123 questions over self-built long neurosurgery recordings and a held-out public lecture-video test split. Across both datasets and all four evaluation dimensions, our agent consistently outperforms one-shot VLMs and general video-agent frameworks, with the largest gains on the long, out-of-domain neurosurgery videos.

Three approaches to a long surgical video question. A one-shot VLM samples sparse keyframes and misses the evidence; SFT-based agents need massive training data and generalize poorly; MedClaw keeps every model frozen and distills reusable skills from about one hundred examples.
Figure 1. Long-horizon temporal reasoning on a long surgery video, where an “after” question is answered by evidence lying outside the window that depicts the named event. (a) A one-shot VLM reads the whole procedure in a single pass, losing the detail the question needs. (b) SFT-based agents learn where to look through supervised or reinforcement fine-tuning, which needs massive data and transfers poorly out of domain. (c) MedClaw keeps every model frozen and instead distills reusable skills through Heuristic Skill Distillation, collecting failure traces, diagnosing the gap, and retaining a skill only when it passes a reward gate.

Method

No weights are updated anywhere in the system. A text-only orchestrator (Opus-4.8) decides where to look but never sees a pixel; frozen vision sub-agents execute each call and return text. Competence accrues in an external skill library.

Overview of MedClaw. Left: a real trace on a temporal before question, where the orchestrator loads the relook skill and issues tool calls that frozen vision sub-agents execute. Top right: the external knowledge base and tunable skill library. Bottom right: the Heuristic Skill Distillation loop.
Figure 2. Overview of MedClaw. Left: a real trace on a temporal “before” question, where the orchestrator loads the discovered directed re-look skill and issues an auditable sequence of tool calls that frozen vision sub-agents execute on the pixels, viewing the full video to localize a window, cropping it, and inspecting a frame, before grounding its answer. Top right: the orchestrator is backed by an external knowledge base queried via RAG and by a library of tunable skills derived from past failures. Bottom right: these skills come from our Heuristic Skill Distillation loop, which collects low-scoring traces on a mining set, diagnoses each gap as operational, unreachable, or already correct, and retains a candidate only if it raises the four-dimensional reward on a validation set.

Tool suite

view_full_video

Localizes where. A low-resolution pass over the whole procedure emits a dense timeline and a candidate window; a second pass re-watches that window at higher resolution and returns the tight sub-window.

crop_video

Inspects what happens in a chosen range, re-read at crop resolution. This is the call that directed re-look aims at the adjacent window.

select_video_frame

One frame at one timestamp, described by an independent frame model — a cross-model check on a specific instrument or action.

crop_video_frame

A normalized spatial region of a single frame, for disambiguating small or overlapping structures.

search_surgical_kb

Retrieval over 1,039 PubMed-derived passages indexed in FAISS under cosine similarity. General surgical background only, with no benchmark answers.

Directed re-lookDiscovered skill

Not a tool and not a rule we wrote. This is the entry the Heuristic Skill Distillation loop mined from the harness's own failed trajectories and then kept, because installing it raised the validation reward past the gate. It lives in an external skill library the orchestrator consults at inference, and it can be inspected, edited, or removed without touching a single weight.

The rule it learned: for a question about what precedes or follows a located event, the answer lies in the window adjacent to that event rather than inside it — so crop and re-read the following window for an “after” question, the preceding one for a “before” or preparation question, and ground the answer there.

Removing it costs more than removing any hand-designed component (−0.38 on the neurosurgery split), and it is exactly the failure mode the VTR partition is built to stress.

Heuristic Skill Distillation

  1. Answer blind, then see the gold.

    On a 60-question mining set the agent answers using the same tools it has at test time, then is shown the reference answer.

  2. Diagnose the gap.

    Operational (a different way of looking would have caught it), unreachable (no amount of looking recovers it), or already correct. Only the first carries an actionable lesson.

  3. Write the rule, operationally.

    The agent writes the skill itself, and it must change how it looks, never what the answer should assert. Learning wording from an unreachable gold only teaches confident hallucination.

  4. Gate it on reward.

    Install one candidate, re-answer a 40-question validation set, re-judge. Keep it only if the four-dimensional mean rises by more than δ = 0.05 and no single question drops by two points or more. A rejected skill leaves no trace.

Results

Four dimensions, each scored 1–5 by a GPT-5.5 judge under the SurgVidLM rubric: correctness (CI), detail (DO), context (CU), temporal (TU). Reported separately for Visual Temporal Reasoning (VTR) and Visual Perception Reasoning (VPR).

Method Avg Visual Temporal Reasoning Visual Perception Reasoning
AvgCIDOCUTU AvgCIDOCUTU

Frontier API models on MedClawBench

How today's strongest commercial models do on the same questions, ranked by the four-dimension mean. Every row is scored by the identical GPT-5.5 rubric, so the numbers are directly comparable.

#ModelAvg VTRVPR CIDOCUTU

Ablations

All on the neurosurgery split (259 questions), reporting the four-dimensional average.

Removing one component at a time

Four-dimensional average, and the change relative to the full harness. † marks the component discovered by the distillation loop rather than designed by hand.

AvgΔ

Wrapping other backbones in the harness

The same open-source model answering one-shot, then as the visual sub-agent inside our harness. The tick on each bar marks its one-shot score.

AvgGain

Judge reliability

Quadratic weighted Cohen's κ against expert ratings: a neurosurgeon on 30 neurosurgery items, surgical experts on 70 SVU items.

DimensionNeurosurgerySVU
CI — correctness0.710.87
DO — detail0.660.86
CU — context0.720.85
TU — temporal0.740.86
Overall0.720.86

Judge means track the experts almost exactly: 2.95 vs 2.94 on neurosurgery, 3.64 vs 3.65 on SVU.

MedClawBench

Every question must survive a cascade certifying it can be answered only by watching the moment it targets: not from priors alone, not from a whole-procedure summary, not by listening to the narration, and — for neurosurgery — only if it matches the neurosurgeon's own interval annotation.

MedClawBench statistics. Panel a: neurosurgery question types, inner ring VTR 109 versus VPR 150, outer ring the ten fine-grained types. Panel b: video-duration distribution against SVU-31K. Panel c: surgical type distribution by questions and videos.
Figure 3. MedClawBench statistics. (a) Neurosurgery question types (259): the inner ring is the Visual Temporal (VTR, 109) versus Visual Perception (VPR, 150) split and the outer ring the ten fine-grained types. (b) Source-video duration for MedClawBench against SVU-31K, as the percentage of videos per bin; our filter drops all sub-ten-minute videos and shifts mass toward the long (≥20 min) tail. (c) Surgical-type distribution over both splits, with the neurosurgery half split into MVD, tumour resection and epileptogenic-lesion resection (HPB* is Hepatobiliary & Pancreas).

Comparison with surgical video QA datasets

DatasetVideo len.≥20 min TemporalDe-leakedNeuro.
EndoVis18-VQAper-frame—✗✓✗
PitVQAper-frame—✗✓✓
SurgViVQA8 frames—short✓✗
SVU-31K (test)13.9 min28%partial✗✗
MedClawBench (ours)18.4 min41%✓✓✓
neurosurgery split28.5 min71%✓✓✓

Video len.: median source-video duration. De-leaked: not answerable without watching, whether by explicit filtering or by the nature of the source.

Data and code

SVU lecture split

The held-out test split of SVU-31K — 864 questions over 141 lecture videos, publicly available. Each clip is stored under its SVU-31K id. Only test_data.zip (157 GB, unpacking to about 148 GB) is needed; the training archives are not. 

# only the test split is needed — the full repo is ~982 GB
huggingface-cli download gkw0010/SVU-31K test_data.zip \
    --repo-type dataset --local-dir SVU-31K
unzip SVU-31K/test_data.zip -d SVU-31K   # -> test_videos_part1/, test_videos_part2/

Neurosurgery split

259 questions over 14 recordings from Beijing Tiantan Hospital, with golds grounded in a neurosurgeon's interval annotations. Because this is identifiable clinical footage, the videos are not yet public; they will be released through an application process. The questions and reference answers ship with the benchmark file.

Code

The harness, the benchmark file, and the per-question scored answers behind every table live at github.com/fyyCS/medclaw-agent. We are finishing the final cleanup and will open the repository shortly.

Reproducing the tables

# run the agent (lite backbone on SVU, flash on neurosurgery)
export SURG_VIDEO_MODEL=gemini-3.1-flash-lite
export SURG_SKILL_TOOLSET=1
python scripts/open_baseline_hermes.py --src <questions>.jsonl --out <answers>.jsonl

# score: four dimensions, 1-5, broken down by VTR / VPR
python surg_env/score_final.py --split neuro
python surg_env/score_final.py --split svu

BibTeX

@misc{fan2026medclawheuristicagentharness,
      title={MedClaw: Heuristic Agent Harness for Long-Horizon Surgical Video Reasoning},
      author={Yingying Fan and Penghui Du and Leyan Zhu and Runze He and Zimeng Wu and Yuxuan Zhang and Liang Chen and Jiahao Xie and Jiangtang Wang and Shuai Shao and Anchao Yang and Yutong Bai and Yan Wang},
      year={2026},
      eprint={2608.14015},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2608.14015},
}