{"benchmark":"rcm-benchmark-1","evaluatedAt":"2026-10-13","cases":97,"trialsPerCase":3,"scoring":"Average the trial outcomes per case, average cases within each workflow, then equally average the six workflow scores. Failed attempts remain in the denominator.","limitations":"The 16 September batch runs share revision 2a90be400a5a88c88766e2f986cd3f0f82d52c94, 97 cases and six equal-weight tracks. Includes ticket/refund grading fixes #2278/#2280 and evaluated provider adapters. Provider and reasoning configurations differ; direct Sonnet 5.5 and Opus 5.5 use one concurrent trial, others eight. Scores describe agent configurations, not standalone model quality. DeepSeek V4.1 Flash and Opus 4.6 each have one provider error included in their denominators. GPT-6.1 Sol adds the first September 29 run at b27df812545618fbfe306505b725f39433479c75 with unchanged cases/graders and a routing adapter update. Gemini 3.8 Flash adds the October 5 run at afc7667901bb1243a2de3250f92f391ab5048a65: 291 completed trials, no recorded engine/provider errors, and medium reasoning. Claude Haiku 5.5 adds the October 7 direct Anthropic run at 1d94adad588c5151607fdbc88b146e79f0273f2b with provider-default reasoning: 291 completed trials and no recorded engine/provider errors. Its later harness revision is not a controlled same-revision comparison. Gemini 4 Argon adds the October 13 Vertex global run at 539012ace88a97da56eb50b0bcb815d538428874: 291 completed trials, no agent errors, medium reasoning, and an OAuth refresh fix. Five benchmark case files and the tool grader changed since the October 5 Gemini run; comparisons are not controlled for revision. No uncertainty intervals or comparable latency ranking; Gemini agent execution timing is disclosed only in its run note.","costMethodology":"Estimated agent API cost, not invoices. Uses captured usage and harness rates; GPT-6 Sol/Luna are reconstructed at assumed Standard short-context OpenAI rates. Opus 5 is reconstructed at standard rates with an assumed 10% US Bedrock premium and five-minute cache writes. Opus 5.5 is reconstructed at standard direct Anthropic rates with five-minute cache writes. All four reconstructions cover 291 usage records but billing tiers are not invoice-verified. DeepSeek and Opus 4.6 cover 290/291 priced attempts; run cost is a partial subtotal. Excludes judge, tool-internal model calls and infrastructure. GPT-6.1 Sol is reconstructed from all 291 usage records at Standard short-context rates verified September 29 ($2 input, $0.10 cached input, $2.50 cache writes and $10 output per million); per-request billing tiers are unavailable. Gemini 3.8 Flash has a $5.8129509 recorded-usage estimate at Google Standard rates effective through December 31, 2026. All 291 saved usage records are priced; complete usage coverage refers to those records. No additional thinking-token estimate is included. Claude Haiku 5.5 uses all 291 saved usage records at October 7 Standard direct Anthropic rates for prompts up to 100k tokens ($0.10 input, $0.01 cache reads, $0.125 five-minute cache writes, $0.50 output per million): $0.88280101 total. Per-request context lengths were not retained; the estimate assumes the lower tier, while requests above 100k cost five times these rates. Gemini 4 Argon uses all 291 saved usage records at Vertex global Standard effective rates through November 30 ($2 input, $0.10 cached input, $10 output per million): $25.1070706 after 50% promotional credits back, or $50.2141412 at full rates. Credits and billing completeness are not invoice-verified; separate thinking counts were not retained. Cost does not affect score.","flows":[{"id":"claim_submission","label":"Claim submission","cases":20,"description":"Investigate, correct, and submit claims; handle approval and escalation."},{"id":"encounter_billing","label":"Encounter billing","cases":11,"description":"Adjust balances, apply write-offs, and manage patient responsibility."},{"id":"patient_billing","label":"Patient billing","cases":8,"description":"Explain balances, apply credits, and handle refunds and permissions."},{"id":"insurance_update","label":"Insurance updates","cases":5,"description":"Update coverage, map payers, and relink insurance to claims."},{"id":"eligibility_check","label":"Eligibility checks","cases":5,"description":"Check coverage, recheck updates, and handle inconclusive results."},{"id":"knowledge","label":"RCM knowledge","cases":48,"description":"Answer revenue cycle management knowledge questions."}],"runs":[{"id":"38436156750","name":"Gemini 4 Argon","provider":"Google Vertex AI (global)","kind":"Single model; medium reasoning","reasoning":"Medium","color":"#1967d2","date":"2026-10-13","commit":"539012ace88a97da56eb50b0bcb815d538428874","passed":[49,26,24,14,15,137],"agentErrors":0,"note":"All 291 trial artifacts identify gemini-4-argon on Gemini via Vertex AI global, with completed engine status and no recorded agent/provider errors. Original grading: 265/291 attempts passed across 97 cases, three trials each. Medium reasoning, default temperature, buffered responses. Recorded-usage cost is $25.1070706 after promotional credits, or $50.2141412 at full rates; no additional thinking-token estimate is included.","cost":{"mean_usd_per_attempt":0.08627859312714777,"priced_attempts_subtotal_usd":25.1070706,"estimated_usd":25.1070706,"known_subtotal_usd":25.1070706,"complete":true,"priced_attempts":291,"total_attempts":291,"provenance":"Calculated October 13, 2026 from all 291 saved usage_by_model records using https://cloud.google.com/vertex-ai/generative-ai/pricing. Vertex global Standard effective rates through November 30, 2026: $2/M uncached input, $0.10/M cached input, $10/M output including thinking. Promotion is delivered as 50% credits back on net spend; receipt of credits is not invoice-verified. Full rates from December 1: $4/$0.20/$20. Recorded totals: 11,531,686 uncached input, 13,297,486 cached input, 71,395 completion tokens. Input is already net of cache hits. All saved records are priced, not all billed usage verified: separate thinking counts were not retained, and no additional thinking estimate is included. Excludes unrecorded thinking, grounding, judge, tool-internal model calls and infrastructure. Original unpriced artifacts remain unchanged; not an invoice.","estimate_note":"After promotional credits; recorded usage only","rates_per_million":{"input":2,"cachedInput":0.1,"output":10}},"weightedScore":91.4877946127946,"completedTrials":291,"passedTrials":265,"runnerErrors":0},{"id":"37666178262","name":"Claude Haiku 5.5","provider":"Anthropic (direct)","kind":"Single model; provider-default reasoning","reasoning":"Provider default","color":"#238579","date":"2026-10-07","commit":"1d94adad588c5151607fdbc88b146e79f0273f2b","passed":[52,28,23,12,14,129],"agentErrors":0,"note":"All 291 trial artifacts identify claude-haiku-5-5 on direct Anthropic with completed engine status and no recorded engine/provider errors. Original grading: 258/291 attempts passed; assistant 169/195 and email 89/96. Provider-default reasoning with no explicit effort or thinking budget. All 291 saved usage records are priced at published Standard rates for prompts up to 100k tokens, assuming five-minute cache writes. Request-level context lengths were not retained, so long-context pricing is not verified.","cost":{"mean_usd_per_attempt":0.003033680446735395,"priced_attempts_subtotal_usd":0.88280101,"estimated_usd":0.88280101,"known_subtotal_usd":0.88280101,"complete":true,"priced_attempts":291,"total_attempts":291,"provenance":"Estimated October 7, 2026 from all 291 saved usage records using https://platform.claude.com/docs/en/about-claude/pricing. Standard direct Anthropic rates for prompts up to 100k tokens: $0.10/M uncached input, $0.01/M cache reads, $0.125/M five-minute cache writes, $0.50/M output. Recorded totals: 2,352 uncached input, 40,656,206 cache-read, 2,306,658 cache-write, 375,343 output tokens. Request-level context lengths were not retained; requests over 100k tokens cost five times these rates. Complete coverage means all saved usage records priced under this assumption, not invoice completeness. Excludes judge, tool-internal model calls and infrastructure. Original unknown_model_price artifacts remain unchanged.","rates_per_million":{"input":0.1,"cachedInput":0.01,"output":0.5}},"weightedScore":88.37752525252526,"completedTrials":291,"passedTrials":258,"runnerErrors":0},{"id":"37378208147","name":"Gemini 3.8 Flash","provider":"Google Gemini (OpenAI-compatible endpoint)","kind":"Single model; medium reasoning","reasoning":"Medium","color":"#327cad","date":"2026-10-05","commit":"afc7667901bb1243a2de3250f92f391ab5048a65","passed":[49,27,24,12,15,141],"agentErrors":0,"note":"All 291 trial artifacts identify gemini-3.8-flash on gemini with completed engine status and no recorded engine/provider errors. Original grading: 268/291 attempts passed across 97 cases, three trials per case. Medium reasoning and default temperature. Mean agent execution time was 21.0 seconds per trial; this is not token-level latency, which the buffered adapter does not expose. Recorded-usage estimate is $5.8129509 at Google Standard paid rates effective through December 31, 2026. All 291 saved usage records are priced; no additional thinking-token estimate is included.","cost":{"mean_usd_per_attempt":0.01997577628865979,"priced_attempts_subtotal_usd":5.8129509,"estimated_usd":5.8129509,"known_subtotal_usd":5.8129509,"complete":true,"priced_attempts":291,"total_attempts":291,"provenance":"Reconstructed October 5, 2026 from all 291 saved usage records using https://ai.google.dev/gemini-api/docs/pricing#gemini-3.8-flash. Standard paid rates through December 31, 2026: $0.75/M input, $0.075/M cached input, $3.75/M output including thinking. Recorded totals: 4,927,743 uncached input, 25,026,082 cached input, 64,050 completion tokens. Input is already net of cached tokens. Coverage denotes all 291 saved usage records priced at the stated rates, not invoice completeness. No additional thinking-token estimate is included; separate thinking counts and request-level billing tiers were not retained. Excludes unrecorded thinking, cache storage, grounding, judge/tool-internal model calls and infrastructure; not an invoice. Original artifacts remain unchanged.","estimate_note":"Recorded usage; no additional thinking estimate","rates_per_million":{"input":0.75,"cachedInput":0.075,"output":3.75}},"weightedScore":90.23358585858587,"completedTrials":291,"passedTrials":268,"runnerErrors":0},{"id":"36604788518","name":"GPT-6.1 Sol","provider":"OpenAI","kind":"Single model; medium reasoning","reasoning":"Medium","color":"#7059a1","date":"2026-09-29","commit":"b27df812545618fbfe306505b725f39433479c75","passed":[45,30,23,12,13,138],"agentErrors":0,"note":"All 291 artifacts identify gpt-6.1-sol on OpenAI via Responses API with medium reasoning; 290 completed and one tool-limit stop, retained as a failure. Cost reconstructed from all 291 usage records at Standard short-context rates verified September 29; request-level billing tiers are unavailable.","cost":{"mean_usd_per_attempt":0.021357653608247422,"priced_attempts_subtotal_usd":6.2150772,"estimated_usd":6.2150772,"known_subtotal_usd":6.2150772,"complete":true,"priced_attempts":291,"total_attempts":291,"provenance":"Reconstructed September 29, 2026 from all 291 saved usage records using https://developers.openai.com/api/docs/pricing, matching the supplied pricing screenshot. Standard short-context rates assumed; request-level context lengths and billing tiers are unavailable. Long-context, Fast/Ultrafast and regional premiums are excluded. Original unpriced artifacts are unchanged; not an invoice.","estimate_note":"Standard short-context estimate","rates_per_million":{"input":2,"cachedInput":0.1,"output":10}},"weightedScore":87.37373737373738,"completedTrials":291,"passedTrials":261,"runnerErrors":0},{"id":"36502531672","name":"Claude Opus 5","provider":"AWS Bedrock (no fallback)","kind":"Single model; provider-default reasoning","reasoning":"Provider default","color":"#536f78","date":"2026-09-29","commit":"2a90be400a5a88c88766e2f986cd3f0f82d52c94","passed":[55,29,24,12,15,139],"agentErrors":0,"note":"All 291 trial artifacts verify claude-opus-5 on bedrock. Eight concurrent trials. 0 engine/provider errors remain in the score. Standard Opus 5 pricing with an assumed 10% US Bedrock premium and five-minute cache writes; routing premium is not invoice-verified.","cost":{"mean_usd_per_attempt":0.15208976683848802,"priced_attempts_subtotal_usd":44.25812215000001,"estimated_usd":44.25812215000001,"known_subtotal_usd":44.25812215000001,"complete":true,"priced_attempts":291,"total_attempts":291,"provenance":"Reconstructed from all 291 usage records using published pricing verified September 29, 2026. Standard Opus 5 pricing with an assumed 10% US Bedrock premium and five-minute cache writes; routing premium is not invoice-verified.","estimate_note":"Reconstructed estimate","rates_per_million":{"input":5.5,"cachedInput":0.55,"output":27.5}},"weightedScore":92.67887205387206,"completedTrials":291,"passedTrials":274,"runnerErrors":0},{"id":"36502537207","name":"Claude Sonnet 5","provider":"AWS Bedrock (no fallback)","kind":"Single model; provider-default reasoning","reasoning":"Provider default","color":"#817d41","date":"2026-09-29","commit":"2a90be400a5a88c88766e2f986cd3f0f82d52c94","passed":[56,29,19,14,15,135],"agentErrors":0,"note":"All 291 trial artifacts verify claude-sonnet-5 on bedrock. Eight concurrent trials. 0 engine/provider errors remain in the score.","cost":{"mean_usd_per_attempt":0.05061683161512029,"priced_attempts_subtotal_usd":14.729498000000005,"estimated_usd":14.729498000000005,"known_subtotal_usd":14.729498000000005,"complete":true,"priced_attempts":291,"total_attempts":291,"provenance":"Captured trial usage and harness pricing snapshots; not a provider invoice.","rates_per_million":{"input":2,"cachedInput":0.2,"output":10}},"weightedScore":91.24368686868686,"completedTrials":291,"passedTrials":268,"runnerErrors":0},{"id":"36502547281","name":"GPT-5.6 Sol","provider":"OpenAI","kind":"Single model; medium reasoning","reasoning":"Medium","color":"#aa8451","date":"2026-09-29","commit":"2a90be400a5a88c88766e2f986cd3f0f82d52c94","passed":[49,31,23,12,15,138],"agentErrors":0,"note":"All 291 trial artifacts verify gpt-5.6-sol on openai. Eight concurrent trials. 0 engine/provider errors remain in the score.","cost":{"mean_usd_per_attempt":0.06969780756013745,"priced_attempts_subtotal_usd":20.282062,"estimated_usd":20.282062,"known_subtotal_usd":20.282062,"complete":true,"priced_attempts":291,"total_attempts":291,"provenance":"Captured trial usage and harness pricing snapshots; not a provider invoice.","rates_per_million":{"input":5,"cachedInput":0.5,"output":30}},"weightedScore":91.21212121212122,"completedTrials":291,"passedTrials":268,"runnerErrors":0},{"id":"36502568619","name":"Claude Sonnet 5.5","provider":"Anthropic","kind":"Single model; provider-default reasoning","reasoning":"Provider default","color":"#526b89","date":"2026-09-29","commit":"2a90be400a5a88c88766e2f986cd3f0f82d52c94","passed":[52,31,24,11,15,133],"agentErrors":0,"note":"All 291 trial artifacts verify claude-sonnet-5-5 on anthropic. One concurrent trial. 0 engine/provider errors remain in the score.","cost":{"mean_usd_per_attempt":0.05176158075601374,"priced_attempts_subtotal_usd":15.062619999999999,"estimated_usd":15.062619999999999,"known_subtotal_usd":15.062619999999999,"complete":true,"priced_attempts":291,"total_attempts":291,"provenance":"Captured trial usage and harness pricing snapshots; not a provider invoice.","rates_per_million":{"input":2,"cachedInput":0.2,"output":10}},"weightedScore":91.05008417508417,"completedTrials":291,"passedTrials":266,"runnerErrors":0},{"id":"36518313926","name":"Claude Opus 5.5","provider":"Anthropic (direct)","kind":"Single model; provider-default reasoning","reasoning":"Provider default","color":"#a65e48","date":"2026-09-29","commit":"2a90be400a5a88c88766e2f986cd3f0f82d52c94","passed":[51,33,24,11,13,138],"agentErrors":0,"note":"All 291 trial artifacts verify claude-opus-5-5 on anthropic, with 97 cases and no engine/provider errors. One concurrent trial. Cost estimated from all 291 usage records.","cost":{"mean_usd_per_attempt":0.08632571615120276,"priced_attempts_subtotal_usd":25.1207834,"estimated_usd":25.1207834,"known_subtotal_usd":25.1207834,"complete":true,"priced_attempts":291,"total_attempts":291,"provenance":"Reconstructed from all 291 usage records at published standard direct Anthropic rates: $4 input, $0.20 cache read, $5 five-minute cache write and $20 output per million tokens. Usage: 2,348 input, 36,704,732 cache read, 2,669,697 cache write and 221,098 output tokens. Not invoice-verified.","estimate_note":"Reconstructed estimate","rates_per_million":{"input":4,"cachedInput":0.2,"output":20}},"weightedScore":90.13888888888889,"completedTrials":291,"passedTrials":270,"runnerErrors":0},{"id":"36502539666","name":"Claude Sonnet 4.6","provider":"AWS Bedrock (no fallback)","kind":"Single model; provider-default reasoning","reasoning":"Provider default","color":"#8c7860","date":"2026-09-29","commit":"2a90be400a5a88c88766e2f986cd3f0f82d52c94","passed":[54,31,21,12,15,126],"agentErrors":0,"note":"All 291 trial artifacts verify claude-sonnet-4-6 on bedrock. Eight concurrent trials. 0 engine/provider errors remain in the score.","cost":{"mean_usd_per_attempt":0.049040728522336735,"priced_attempts_subtotal_usd":14.27085199999999,"estimated_usd":14.27085199999999,"known_subtotal_usd":14.27085199999999,"complete":true,"priced_attempts":291,"total_attempts":291,"provenance":"Captured trial usage and harness pricing snapshots; not a provider invoice.","rates_per_million":{"input":3,"cachedInput":0.30000000000000004,"output":15}},"weightedScore":89.82323232323233,"completedTrials":291,"passedTrials":259,"runnerErrors":0},{"id":"36502560191","name":"DeepSeek V4.1 Flash","provider":"Baseten","kind":"Single model; provider-default reasoning","reasoning":"Provider default","color":"#c26343","date":"2026-09-29","commit":"2a90be400a5a88c88766e2f986cd3f0f82d52c94","passed":[49,32,23,11,15,124],"agentErrors":1,"note":"All 291 trial artifacts verify deepseek-ai/DeepSeek-V4.1-Flash on baseten. Eight concurrent trials. One engine/provider error is included in the score.","cost":{"mean_usd_per_attempt":null,"priced_attempts_subtotal_usd":3.250703999999999,"estimated_usd":null,"known_subtotal_usd":3.250703999999999,"complete":false,"priced_attempts":290,"total_attempts":291,"provenance":"Captured trial usage and harness pricing snapshots; not a provider invoice.","rates_per_million":{"input":0.3,"cachedInput":0.03,"output":1.2}},"weightedScore":88.98569023569024,"completedTrials":291,"passedTrials":254,"runnerErrors":0},{"id":"36502562802","name":"GLM-5.3","provider":"Baseten","kind":"Single model; provider-default reasoning","reasoning":"Provider default","color":"#3b7f73","date":"2026-09-29","commit":"2a90be400a5a88c88766e2f986cd3f0f82d52c94","passed":[53,29,21,12,15,124],"agentErrors":0,"note":"All 291 trial artifacts verify zai-org/GLM-5.3 on baseten. Eight concurrent trials. 0 engine/provider errors remain in the score.","cost":{"mean_usd_per_attempt":0.022301656357388303,"priced_attempts_subtotal_usd":6.489781999999996,"estimated_usd":6.489781999999996,"known_subtotal_usd":6.489781999999996,"complete":true,"priced_attempts":291,"total_attempts":291,"provenance":"Captured trial usage and harness pricing snapshots; not a provider invoice.","rates_per_million":{"input":1.4,"cachedInput":0.14,"output":4.4}},"weightedScore":88.30387205387206,"completedTrials":291,"passedTrials":254,"runnerErrors":0},{"id":"36502565865","name":"Grok 4.7","provider":"xAI","kind":"Single model; provider-default reasoning","reasoning":"Provider default","color":"#805b81","date":"2026-09-29","commit":"2a90be400a5a88c88766e2f986cd3f0f82d52c94","passed":[51,31,21,11,14,138],"agentErrors":0,"note":"All 291 trial artifacts verify grok-4.7 on xai. Eight concurrent trials. 0 engine/provider errors remain in the score.","cost":{"mean_usd_per_attempt":0.0713880549828179,"priced_attempts_subtotal_usd":20.773924000000008,"estimated_usd":20.773924000000008,"known_subtotal_usd":20.773924000000008,"complete":true,"priced_attempts":291,"total_attempts":291,"provenance":"Captured trial usage and harness pricing snapshots; not a provider invoice.","rates_per_million":{"input":2,"cachedInput":0.5,"output":6}},"weightedScore":88.15656565656566,"completedTrials":291,"passedTrials":266,"runnerErrors":0},{"id":"36502534368","name":"Claude Opus 4.6","provider":"AWS Bedrock (no fallback)","kind":"Single model; provider-default reasoning","reasoning":"Provider default","color":"#557882","date":"2026-09-29","commit":"2a90be400a5a88c88766e2f986cd3f0f82d52c94","passed":[47,33,21,12,15,117],"agentErrors":1,"note":"All 291 trial artifacts verify claude-opus-4-6 on bedrock. Eight concurrent trials. One engine/provider error is included in the score.","cost":{"mean_usd_per_attempt":null,"priced_attempts_subtotal_usd":24.032078000000013,"estimated_usd":null,"known_subtotal_usd":24.032078000000013,"complete":false,"priced_attempts":290,"total_attempts":291,"provenance":"Captured trial usage and harness pricing snapshots; not a provider invoice.","rates_per_million":{"input":5,"cachedInput":0.5,"output":25}},"weightedScore":87.84722222222221,"completedTrials":291,"passedTrials":245,"runnerErrors":0},{"id":"36502555491","name":"GPT-5.6 Terra","provider":"OpenAI","kind":"Single model; medium reasoning","reasoning":"Medium","color":"#39745b","date":"2026-09-29","commit":"2a90be400a5a88c88766e2f986cd3f0f82d52c94","passed":[48,29,21,12,14,134],"agentErrors":0,"note":"All 291 trial artifacts verify gpt-5.6-terra on openai. Eight concurrent trials. 0 engine/provider errors remain in the score.","cost":{"mean_usd_per_attempt":0.03666498281786941,"priced_attempts_subtotal_usd":10.669509999999999,"estimated_usd":10.669509999999999,"known_subtotal_usd":10.669509999999999,"complete":true,"priced_attempts":291,"total_attempts":291,"provenance":"Captured trial usage and harness pricing snapshots; not a provider invoice.","rates_per_million":{"input":2.5,"cachedInput":0.25,"output":15}},"weightedScore":86.96127946127946,"completedTrials":291,"passedTrials":258,"runnerErrors":0},{"id":"36502549945","name":"GPT-6 Luna","provider":"OpenAI","kind":"Single model; medium reasoning","reasoning":"Medium","color":"#718b9b","date":"2026-09-29","commit":"2a90be400a5a88c88766e2f986cd3f0f82d52c94","passed":[47,28,24,11,13,132],"agentErrors":0,"note":"All 291 trial artifacts verify gpt-6-luna on openai. Eight concurrent trials. 0 engine/provider errors remain in the score. Standard short-context OpenAI pricing; per-request tiers unavailable; excludes long-context, Fast and regional premiums.","cost":{"mean_usd_per_attempt":0.001910246701030928,"priced_attempts_subtotal_usd":0.55588179,"estimated_usd":0.55588179,"known_subtotal_usd":0.55588179,"complete":true,"priced_attempts":291,"total_attempts":291,"provenance":"Reconstructed from all 291 usage records using published pricing verified September 29, 2026. Standard short-context OpenAI pricing; per-request tiers unavailable; excludes long-context, Fast and regional premiums.","estimate_note":"Reconstructed estimate","rates_per_million":{"input":0.1,"cachedInput":0.01,"output":0.5}},"weightedScore":85.80808080808082,"completedTrials":291,"passedTrials":255,"runnerErrors":0},{"id":"36502557823","name":"GPT-5 mini","provider":"OpenAI","kind":"Single model; provider-default reasoning","reasoning":"Provider default","color":"#686d76","date":"2026-09-29","commit":"2a90be400a5a88c88766e2f986cd3f0f82d52c94","passed":[44,29,21,12,13,130],"agentErrors":0,"note":"All 291 trial artifacts verify gpt-5-mini on openai. Eight concurrent trials. 0 engine/provider errors remain in the score.","cost":{"mean_usd_per_attempt":0.008452151202749147,"priced_attempts_subtotal_usd":2.459576000000002,"estimated_usd":2.459576000000002,"known_subtotal_usd":2.459576000000002,"complete":true,"priced_attempts":291,"total_attempts":291,"provenance":"Captured trial usage and harness pricing snapshots; not a provider invoice.","rates_per_million":{"input":0.25,"cachedInput":0.025,"output":2}},"weightedScore":84.27609427609428,"completedTrials":291,"passedTrials":249,"runnerErrors":0},{"id":"36502544877","name":"GPT-6 Sol","provider":"OpenAI","kind":"Single model; medium reasoning","reasoning":"Medium","color":"#a77460","date":"2026-09-29","commit":"2a90be400a5a88c88766e2f986cd3f0f82d52c94","passed":[42,26,24,7,15,135],"agentErrors":0,"note":"All 291 trial artifacts verify gpt-6-sol on openai. Eight concurrent trials. 0 engine/provider errors remain in the score. Standard short-context OpenAI pricing; per-request tiers unavailable; excludes long-context, Fast and regional premiums.","cost":{"mean_usd_per_attempt":0.03053312989690721,"priced_attempts_subtotal_usd":8.885140799999999,"estimated_usd":8.885140799999999,"known_subtotal_usd":8.885140799999999,"complete":true,"priced_attempts":291,"total_attempts":291,"provenance":"Reconstructed from all 291 usage records using published pricing verified September 29, 2026. Standard short-context OpenAI pricing; per-request tiers unavailable; excludes long-context, Fast and regional premiums.","estimate_note":"Reconstructed estimate","rates_per_million":{"input":2,"cachedInput":0.2,"output":10}},"weightedScore":81.53409090909092,"completedTrials":291,"passedTrials":249,"runnerErrors":0},{"id":"36502552471","name":"GPT-5.6 Luna","provider":"OpenAI","kind":"Single model; reasoning disabled for tool compatibility","reasoning":"None (tool compatibility)","color":"#4c8390","date":"2026-09-29","commit":"2a90be400a5a88c88766e2f986cd3f0f82d52c94","passed":[39,22,19,9,11,122],"agentErrors":0,"note":"All 291 trial artifacts verify gpt-5.6-luna on openai. Eight concurrent trials. 0 engine/provider errors remain in the score.","cost":{"mean_usd_per_attempt":0.003304048109965635,"priced_attempts_subtotal_usd":0.9614779999999998,"estimated_usd":0.9614779999999998,"known_subtotal_usd":0.9614779999999998,"complete":true,"priced_attempts":291,"total_attempts":291,"provenance":"Captured trial usage and harness pricing snapshots; not a provider invoice.","rates_per_million":{"input":0.2,"cachedInput":0.020000000000000004,"output":1.2}},"weightedScore":71.48148148148148,"completedTrials":291,"passedTrials":222,"runnerErrors":0},{"id":"36502542258","name":"Claude Haiku 4.5","provider":"AWS Bedrock (no fallback)","kind":"Single model; provider-default reasoning","reasoning":"Provider default","color":"#995f7d","date":"2026-09-29","commit":"2a90be400a5a88c88766e2f986cd3f0f82d52c94","passed":[44,27,12,9,15,81],"agentErrors":0,"note":"All 291 trial artifacts verify claude-haiku-4-5 on bedrock. Eight concurrent trials. 0 engine/provider errors remain in the score.","cost":{"mean_usd_per_attempt":0.014311859106529216,"priced_attempts_subtotal_usd":4.164751000000002,"estimated_usd":4.164751000000002,"known_subtotal_usd":4.164751000000002,"complete":true,"priced_attempts":291,"total_attempts":291,"provenance":"Captured trial usage and harness pricing snapshots; not a provider invoice.","rates_per_million":{"input":1,"cachedInput":0.1,"output":5}},"weightedScore":70.23358585858587,"completedTrials":291,"passedTrials":188,"runnerErrors":0}]}