Tracer
← polyphony · eval · frankenstein · ci

polyphony · claude-opus-5-5

✗ failincomplete: ensemble skipped
Pass rate
71%
Cases
8 (2 failed)
Mode
eval
Suite
frankenstein
Env
ci
Started
2026-10-23 22:47 UTC
Duration
—
Cost
—
Tokens in/out
— / —
Commit
Branch
main
Source

Metrics

attribution.accuracy0.567
attribution.chance0.333
attribution.score0.567
continuity.detection_recall0.500
continuity.false_positive_rate0
continuity.score0.500
extraction.f10.571
extraction.precision0.429
extraction.recall0.857
extraction.score0.571
ingestion.score0.429
outline.score0.200
prose.score0.781
prose.words3,564
retrieval.mrr0.833
retrieval.precision_at_k0.815
retrieval.score0.815
revise.score0.875

Labels

cadenceon-demand
experimentw15-pre-claude
harness_shaa52817cd8a3d5ac5af676829a3c31445fc86d830
judge_providergroq
judge_self_gradedFalse
repeat1
skippedensemble
triggerworkflow_dispatch

Compared with the previous run

Baseline 2026-10-23 08:18 UTC · 86% → this run 71% · code diff ↗

✗ newly failing · 2
  • eval:ingestion
  • eval:outline
✓ newly passing · 1
  • eval:extraction
± score moved · 1
  • eval:revise: 0.94 → 0.88

Cases (8, 2 failed, failures first)

gated 5/7 passed

eval:ingestionscore 0.43✗ fail
score=0.429grounded=6avg_chunks=6.43n_characters=14
Output
grounded=6/14 avg_chunks=6.43
eval:outlinescore 0.20✗ fail
judge=0.467score=0.200n_nodes=6chapters_target=6unknown_rate_canon=0
Output
nodes=6 structural_ok=True
⚖︎ Judge rationale

R1: Several listed characters (Alina, Cosima Lorenzi, Fenwick, Soraya) do not appear at all, violating cast fidelity. | R6: The outline faithfully follows the confirmed premise, each act’s turn is realized, all chapters show change, and the chapter count matches the request. | R3: No artifact text was provided, so it fails to meet any rubric criteria.

eval:continuityscore 0.50✓ pass
score=0.500precision=0.500n_findings=2n_injected=2quote_verbatim=1detection_recall=0.500n_control_findings=0false_positive_rate=0
Output
detection_recall=0.5 fpr=0.0
eval:extractionscore 0.57✓ pass
f1=0.571score=0.571recall=0.857windows=2precision=0.429f1_selected=0.615recall_selected=0.571precision_selected=0.667principal_fields_rate=0.667principal_mentions_min=4
Output
F1=0.5714 P=0.4286 R=0.8571 predicted=['The narrator (the creature)', 'Emeric Vandel', 'Fenwick', 'Alina', "Fenwick's father", 'Isolde Sarrow', 'Hollis Crane', 'Cosima', 'Rowan Merrow', 'Uncle Thomas', "Crane's father", 'The lieutenant', 'The master', "Vandel's father"]
eval:prosescore 0.78✓ pass
score=0.781words=3,564queued=4deferred=0n_scenes=4cast_named=1prose_chapter=1cast_from_beat=1r1_capped_scenes=0estimate_over_actual=0.709beat_seam_max_overlap=0scene_seam_max_overlap=0
Output
words=3564 path=planned rubrics={'R1': 1.0, 'R2': 0.175, 'R3': 1.0, 'R4': 0.95, 'R5': 1.0} seams=0/0
⚖︎ Judge rationale

scene 1 R1: All required canon characters (Rowan Merrow, Hollis Crane) appear correctly and act as described, with no extraneous or missing principals. | scene 1 R2: No dialogue provided to assess voice distinctness, so cannot attribute lines to characters. | scene 1 R3: The scene follows all five planned beats in order, realizing each change without adding unplanned events. | scene 1 R4: The passage maintains consistent Rowan POV and present tense, matches tone, and is polished narrative prose without structural errors. | scene 1 R5: The prose aligns with canon and planned beats, contains no contradictions, and would identify all planted errors (none present) without false positives. | scene 2 R1: All principal characters required by the beat (Rowan Merrow and Emeric Vandel) appear with correct names and behavior, and no unauthorized major characters are introduced. | scene 2 R2: No dialogue was provided to evaluate voice distinctness, so attribution cannot be assessed. | scene 2 R3: All five planned beats appear in order, each with the required changes and no extraneous material. | scene 2 R4: The scene maintains consistent POV and tense, matches the intimate foreboding tone, and is polished narrative prose without headings or labels. | scene 2 R5: The passage aligns with the established canon and prior scene, introducing no contradictions and faithfully continuing the storyline. | scene 3 R1: All principal characters (Emeric Vandel and Rowan Merrow) appear correctly and act in line with canon, with only minor supporting mention of Hollis Crane. | scene 3 R2: The lines mostly reflect Emeric's gothic, obsessive voice, allowing about 70‑80% correct attribution, but some generic phrasing prevents reaching the 90% threshold. | scene 3 R3: All five planned beats appear in order, each achieving its intended change with no unplanned additions. | scene 3 R4: The scene maintains consistent POV and tense, matches the dark, obsessive tone, and is well‑crafted prose with only minor slips, meeting the rubric near the top tier. | scene 3 R5: The scene aligns with the established canon and prior text, containing no contradictions and fulfilling the continuity criteria fully. | scene 4 R1: All principal characters required (Emeric Vandel and Rowan Merrow) appear correctly with accurate names and behavior, and no extraneous or mischaracterized cast members are introduced. | scene 4 R2: All lines match Emeric’s voice; none can be attributed to Rowan, so voice distinctness is essentially nonexistent. | scene 4 R3: All five planned beats appear sequentially, each achieving its described change, with no extra unplanned material. | scene 4 R4: The passage maintains consistent POV and tense, captures the intended horror‑guilt tone, and is well‑crafted prose with only minor occasional phrasing issues. | scene 4 R5: The passage aligns fully with the established canon and prior scene, containing no contradictions.

eval:retrievalscore 0.81✓ pass
mrr=0.833score=0.815n_queries=9precision_at_k=0.815
Output
precision@k=0.8148 mrr=0.8333
eval:revisescore 0.88✓ pass
score=0.875tighten_n=3n_revisions=4
Output

reported 1 reported

eval:attribution0.57 · reported
n=30empty=0score=0.567chance=0.333correct=17accuracy=0.567mean_gap=0.048generated=30
Output
accuracy=0.5667 chance=0.3333 n=30