SVGBench · harness matrix

84 model-generated drawings, seven upstream questions, six harness × model pairs, one judge — September 7, 2026. Scores from bench/harness-matrix/results/svgbench/scores.json.

requirements met ÷ requirements total

SVGBench is a question bank of 105 drawing prompts, each with a checklist of concrete requirements (pinned commit dd42e2c, July 2025). Seven harness-matrix prompts matched a question, so 84 SVGs were rendered in headless Chrome and graded strictly, one requirement at a time: half-met counts as failed, and anything the renderer doesn't show doesn't count. 413 of 794 requirement checks met. Every verdict on this board comes from a single judge, claude-opus-5. The board it replaces used three judges, two of them grading their own drawings — what that was worth is measured below. Different subset, route and judge make these results unsuitable for comparison with the upstream leaderboard.

Leaderboard

Mean over each pair's scored artifacts. Every figure carries its n: the per-question cells are that pair's mean on that question over that many drawings. Underlines mark the best value in a column. Two pairs drew a single question each and are listed under the board, per-prompt only — they are not ranked.

PairDrawingsQ0Q4Q5Q6Q7Q12Q13MeanWorth knowing

Question coverage still differs per pair, so the means are not like-for-like across rows; Q7, drawn by five of the six pairs, is the widest shared column. The ranking on it matches the ranking on the means for the two pairs with real coverage: claude-opus-5 leads qwen3.8-flash-next 0.857 to 0.786 on Q7 (n=5 and n=10) and 0.693 to 0.591 overall (n=21 and n=25).

Single-prompt pairsper-prompt figures, not ranked

PairDrawingsQuestionScore on that questionWorth knowing

What one judge changed

The previous board used three judges, and most verdicts were self-judged. opencode/gpt-5.6-terra wrote 28 of the 43 verdicts, all on its own pair's drawings. claude-fable-5-1 wrote 12, none of them on its own drawings. qwen3.8-flash-next wrote 3, all on its own. Every artifact was re-judged by claude-opus-5. 43 artifacts appear on both boards, so their old and new verdicts can be compared directly.

PairOld boardThis boardChangeSame drawings?

Two pairs gained coverage between boards, so their means moved for two reasons at once. opencode × gpt-5.6-terra is the one clean comparison: the same 28 drawings on both boards, judged by the model that drew them at 0.421 and at 0.336 by claude-opus-5, which drew none of these 28 — −0.085, with 15 of the 28 scoring lower than they did under self-judging.

The 43 artifacts that carry both verdictssplit by who wrote the old one

Old verdict written byArtifactsOld meanNew meanChangeFell / rose / unchanged

Self-judging moved 31 artifacts down 0.079 on average; the 12 verdicts claude-fable-5-1 wrote, none of them on its own drawings, moved +0.010. The smaller self-judged block is qwen3.8-flash-next's three fruit-stall takes, 0.273 under its own judging and 0.242 here. This is the whole case for re-judging: the judge, not the drawing, was moving the score.

The second take helped; animation never did

OpenCode × GPT-5.6 Terra generated v1 without prior artifacts, then used fresh agents for v2, v3, and an animated take. Under the single judge, v2 is the best take on three of the seven prompts and v3 on one; two prompts never move across all four takes, and the rubber ducky ends three requirements below its own v2. The animated take scores exactly what v3 scores on all seven prompts — SMIL motion neither adds nor loses points with a judge that reads a PNG.

28 scored SVGs 0.336 mean score (n=28) 7 SVGBench prompts 4 takes each

Each line is normalized by that question's requirement count. Values under the points are requirements met, not percentages. Every take's SVG is in the logs tree; requirement-level verdicts are linked from the run record.

Q7 head-to-head

The dolphin through a hula hoop — five of the six pairs drew it, 21 artifacts in all. Each card carries the pair's Q7 mean with its n, and shows that pair's highest-scoring take with the requirements it missed. All 21 drawings put the trainer's arm in the frame; the requirement that fails most often is the jump itself — 13 of 21 leave part of the dolphin in or on the water. Frames follow the sibling report: teal for the local rig models, raspberry for the Anthropic-hosted controls.

By question

One matrix per question, requirement text verbatim from the pinned questions file. A cell is how many of that pair's drawings met that requirement, out of how many it drew — hover a cell that lost points for the judge's reason. Column heads carry the pair's mean on the question and its n.

Unscored artifacts

SVGs in the logs tree that scores.json records but does not grade. Five of them, all the same prompt: "dawn on the beach" is not an SVGBench question, so there is no requirement checklist to grade it against. They stay in the artifact gallery, not on this scoreboard. Every other artifact in the tree is scored.

Open raw file