Tracer
← polyphony · eval · frankenstein · ci

polyphony · claude-opus-5-5

✗ failincomplete: ensemble skipped
Pass rate
71%
Cases
8 (2 failed)
Mode
eval
Suite
frankenstein
Env
ci
Started
2026-10-24 05:42 UTC
Duration
—
Cost
—
Tokens in/out
— / —
Commit
Branch
claude/determined-bell-wo0vsx
Source

Metrics

attribution.accuracy0.533
attribution.chance0.333
attribution.score0.533
continuity.detection_recall1
continuity.false_positive_rate0
continuity.score1
extraction.f10.421
extraction.precision0.333
extraction.recall0.571
extraction.score0.421
ingestion.score0.467
outline.score0.880
prose.score0.801
prose.words3,591
retrieval.mrr0.833
retrieval.precision_at_k0.815
retrieval.score0.815
revise.score0.762

Labels

cadenceon-demand
experimentw15-pre-regrade
harness_shab46d695794a77670e2a34b6d9a4520e1ccb00850
judge_providergroq+anthropic
judge_self_gradedTrue
repeat1
skippedensemble
triggerworkflow_dispatch

Compared with the previous run

Baseline 2026-10-23 08:18 UTC · 86% → this run 71% · code diff ↗

✗ newly failing · 1
  • eval:ingestion
± score moved · 4
  • eval:revise: 0.94 → 0.76
  • eval:attribution: 0.60 → 0.53
  • eval:outline: 0.50 → 0.88
  • eval:continuity: 0.50 → 1.00

Cases (8, 2 failed, failures first)

gated 5/7 passed

eval:extractionscore 0.42✗ fail
f1=0.421score=0.421recall=0.571windows=2precision=0.333f1_selected=0.462recall_selected=0.429precision_selected=0.500principal_fields_rate=1principal_mentions_min=4
Output
F1=0.4211 P=0.3333 R=0.5714 predicted=['Emeric Vandel', 'Fenwick', 'The creature (narrator)', 'Alina', 'The old man (cottager)', 'Isolde Sarrow', 'Hollis Crane', 'my cousin', 'Robert Walton (not used)', 'Uncle Thomas', 'The lieutenant', 'The master']
eval:ingestionscore 0.47✗ fail
score=0.467grounded=7avg_chunks=10.2n_characters=15
Output
grounded=7/15 avg_chunks=10.2
eval:continuityscore 1.00✓ pass
score=1precision=0.667n_findings=3n_injected=2quote_verbatim=1detection_recall=1n_control_findings=0false_positive_rate=0
Output
detection_recall=1.0 fpr=0.0
eval:outlinescore 0.88✓ pass
judge=0.920score=0.880n_nodes=6chapters_target=6unknown_rate_canon=0
Output
nodes=6 structural_ok=True
⚖︎ Judge rationale

R1: All principal characters appear with correct names and roles, and no one acts against their canon description. | R6: The outline tracks the confirmed premise and every act turn faithfully (eyes opening, unmasking, destroyed companion, Emeric's death with the creature's farewell), with six chapters matching the request exactly and escalating conflict, but a few beats are near-static (e.g., 'The Tale's Last Pages', | R3: Every planned beat appears in the planned order and each beat's change lands, with only small unplanned additions (the cottage burning, the crew's demand to turn south, Emeric's plea to Rowan, the marriage promise) that do not pre-empt later beats.

eval:prosescore 0.80✓ pass
score=0.801words=3,591queued=4deferred=0n_scenes=4r2_lines=0cast_named=1prose_chapter=1r2_applicable=0cast_from_beat=1r1_capped_scenes=0estimate_over_actual=0.787beat_seam_max_overlap=0scene_seam_max_overlap=0
Output
words=3591 path=planned rubrics={'R1': 0.935, 'R2': None, 'R3': 0.8975, 'R4': 0.8, 'R5': 0.8725} seams=0/0
⚖︎ Judge rationale

scene 1 R1: Rowan Merrow and Emeric Vandel appear as planned and act per the brief; names are exact, and the only other figures are minor functional crew and a surgeon. Small deductions: Emeric calls himself 'Vandel' with no issue, but the beat that Rowan 'pours out' his hunger is handled fine; the minor flaw i | scene 1 R3: All five beats are realised in order with their changes landing (isolation, floe sighting, destination demand, cabin thaw and name, warning and keeper role), with only minor additions like the surgeon, and the final 'keeper' shift is implied rather than explicit. | scene 1 R4: Close-third past tense on Rowan holds throughout, and the eerie, desolate tone is concrete and well paced, with no beat labels or stitching. Small slips keep it under 1.0: the repeated 'three hundred miles' and 'ruined hand' phrasing, a few stock similes ('quick as a hunted thing's', 'like black wat | scene 1 R5: The scene follows the brief and canon (names, setting, the question about the ship's heading, the cabin thaw, the warning), with only small slips: one blackened hand becomes 'ruined hands' plural, and 'black square of night' sits oddly with the thinning fog on the same day. | scene 1 R2: not applicable: under two cast speakers with voice samples have attributed lines | scene 2 R1: All four beat-cast principals appear with exact names and behave per the brief, and invented figures (parents, professor, Rowan's sister and uncle) are minor and functional, with only trivial embellishments. | scene 2 R3: All five beats are realised in order with their changes landing (Rowan's pang, discomfort, foreboding, final unsettling realisation), and nothing major is added; minor quibbles include the beat 2 'pang' being folded into a later envy line and the Rowan/Hollis moment being slightly diffuse. | scene 2 R4: The close third-person past-tense POV on Rowan holds throughout, the tone is tender with building foreboding, and the prose is concrete and varied with no labels or repetition; minor demerits are a few generic phrases ('a ruin of a man', 'something in him drew tight') and some close adherence to the | scene 2 R5: The scene contradicts nothing in the canon or prior text: names, the cabin, the lamp, Vandel's frailty, the promise to tell his story, and the beat order (Alina, Hollis, alchemists, university, closing reflection) all hold, and the new details about Rowan's sister and uncle's library are compatible | scene 2 R2: not applicable: under two cast speakers with voice samples have attributed lines | scene 3 R1: All four beat-cast principals (Rowan Merrow, Emeric Vandel, Alina, Hollis Crane) appear exactly as planned: Alina and Hollis are recalled by name in Emeric's confession, and Rowan acts as POV; the only other people are unnamed porters and crew, and no one acts against canon, with the minor quibble t | scene 3 R3: All five planned beats are realised in order with their stated changes (the airless cabin, the loving hands, the faltering on names, the sour turn into self-loathing, the warning landing on Rowan), with only trivial embellishments such as the gallery and stair, the falling leaves and the porters; th | scene 3 R4: The scene keeps Rowan's close-third past-tense POV throughout, mirrors the sample's measured, concrete, gothic register, hits every beat without labels or summary, and has only minor slips (e.g., 'cold talking' phrasing, some generic images like 'light striking ice' and 'breath from a bellows', and | scene 3 R5: The scene stays consistent with the canon names, the prior text (closed eyes reopening, stool, blanket, lamp, hull creak, Alina and Hollis) and all five planned beats, with only trivial unestablished embellishments such as 'frayed cuffs' and the crew murmuring 'for a week now'. | scene 3 R2: not applicable: under two cast speakers with voice samples have attributed lines | scene 4 R1: Only Rowan Merrow and Emeric Vandel appear, as the beat cast requires, and both act in line with the plan; ther

eval:retrievalscore 0.81✓ pass
mrr=0.833score=0.815n_queries=9precision_at_k=0.815
Output
precision@k=0.8148 mrr=0.8333
eval:revisescore 0.76✓ pass
score=0.762tighten_n=3n_revisions=5
Output

reported 1 reported

eval:attribution0.53 · reported
n=30empty=0score=0.533chance=0.333correct=16accuracy=0.533mean_gap=0.056generated=30
Output
accuracy=0.5333 chance=0.3333 n=30