Disagreement is the useful signal

When an agent works on a long task, checking its output becomes as hard as producing it. Caiqi Zhang and six co-authors ask how far verification can be pushed with a fixed base model, without reference answers or grading rubrics at test time. Their starting observation inverts the usual instinct: across repeated rollouts, disagreement often exposes correct alternatives, while consensus can conceal shared errors.

VeriHarness turns the same model the generator uses into an agentic verifier by giving it a workspace, evidence tools and reusable verification skills. Two components do the work. A disagreement resolver checks competing claims against evidence from the environment. A consensus challenger tests the claims every rollout agrees on and looks for requirements nobody addressed. What they find then guides which artefact is selected and how it is revised.

Across five long-horizon workspace benchmarks and two frontier models, VeriHarness reached the highest selection scores among the baselines the authors evaluated. Allowing evidence-backed revision on top of selection raised the gain over a single rollout to 6.2 points with Gemini 3.5 Flash and 6.4 points with Claude Opus 4.8.

The authors also report that the verification skills improve themselves from failure feedback, which matters for the economics of the approach: the harness is not a fixed checklist that ages as tasks change. Code has been released alongside the paper, which carries a CC BY 4.0 licence.

The practical reading is narrower than the headline. Nothing here needs a stronger judge model or a human rubric. What it needs is the willingness to sample several times and then spend compute on reconciling the differences rather than counting votes, which is the opposite of how self-consistency is usually applied.