Does the failure
show up again?
After the original tied scores, we publicly registered new money cases and the analysis plan before collecting any follow-up answers. This page shows every new answer, including the controls and both repetitions. Nothing here replaces the original 84-response study or the public Kaggle leaderboard.
| Model | Repetition 1 | Repetition 2 | Explicit-convention targets: R1; R2 |
|---|---|---|---|
| Gemini 3.8 Flash | 12/12 | 12/12 | 8/8; 8/8 |
| Claude Sonnet 4.6 | 7/12 | 6/12 | 3/8; 2/8 |
| Gemma 4 26B A4B | 11/12 | 12/12 | 7/8; 8/8 |
A point still requires the correct status, exact canonical value and bare JSON. Secondary labels below explain a strict failure; they never repair an answer or change its score.
Unedited model responses
What this follow-up can and cannot establish
The cases were designed after seeing the original findings. They are a targeted follow-up, not a blind holdout or a random sample of administrative work. Values and wording changed together, so a different answer does not identify its cause. Two repetitions are not evidence of general model reliability or statistical significance. SDK run caching was disabled; provider-side caching and effective seed support were not controlled.
All 72 prompts and answers were regraded and reconciled with six native parent runs and 72 distinct native conversations. This is a consistency check, not a signed third-party attestation. Codex provided a separate AI review, not a human or blind audit.