Generated 2026-10-12T20:11:47Z. 12 models, 6 ranked, 6 not ranked.
Official results. Produced from a frozen dataset tag under the published methodology. The methodology is pre-registered and awaiting independent replication. Treat these as the current best measurement, not as settled fact.
Provenance
Suite
v2
Dataset
2.6.0
Cases per run
4701
Run dates
2026-10-07 to 2026-10-12
Run notes
In the Google run (gemini-3.8-flash), the first 1,610 cases ran with the model thinking mode on and the remaining cases ran with it off. A fix to the adapter software landed partway through the run and turned thinking off. Every case was run once and scored the same way in both parts, and the complete record of the run is sealed in its artifact.
In the Moonshot run (kimi-k3), the first 3,274 cases ran with the model thinking mode on and the remaining cases ran with it off. A fix to the adapter software landed partway through the run and turned thinking off. Every case was run once and scored the same way in both parts, and the complete record of the run is sealed in its artifact.
Models marked Not ranked ran the full benchmark but did not qualify for a rank under the published admission rules, and the reason is printed with each name. One of those rules caps the malformed response rate at 5 percent, because a score built only on the cases a model managed to answer can flatter it. Every number is shown either way.
Per-run detail
deepseek-flash
Family breakdown
Family
ASR
95% CI
Cases
canary_leakage
0.0000
0.0000 to 0.0091
420
crosslingual_shift
0.0286
0.0164 to 0.0493
420
cryptographic_payload
0.1214
0.0936 to 0.1561
420
evidence_positioning
0.0303
0.0174 to 0.0522
420
frequency_salience
0.0048
0.0013 to 0.0172
420
history_leakage
0.0025
0.0004 to 0.0142
420
judge_sycophancy
0.0357
0.0218 to 0.0581
421
question_pair_incoherence
0.0175
0.0085 to 0.0357
400
retrieval_poisoning
0.7756
0.7348 to 0.8117
470
threshold_edge_hunting
0.0000
0.0000 to 0.0091
420
verbosity_inflation
0.2681
0.2300 to 0.3099
470
Calibration
benign ECE
0.0565
attacked ECE
0.0706
jev-1.13.0
Family breakdown
Family
ASR
95% CI
Cases
canary_leakage
0.0000
0.0000 to 0.0091
420
crosslingual_shift
0.1568
0.1233 to 0.1973
420
cryptographic_payload
0.0341
0.0204 to 0.0564
420
evidence_positioning
0.0354
0.0212 to 0.0585
420
frequency_salience
0.0095
0.0037 to 0.0242
420
history_leakage
0.0000
0.0000 to 0.0094
420
judge_sycophancy
0.0335
0.0201 to 0.0554
421
question_pair_incoherence
0.0000
0.0000 to 0.0095
400
retrieval_poisoning
0.8371
0.8000 to 0.8684
470
threshold_edge_hunting
0.0048
0.0013 to 0.0172
420
verbosity_inflation
0.2064
0.1722 to 0.2453
470
Calibration
benign ECE
0.0741
attacked ECE
0.0957
gemini-3.8-flash
Family breakdown
Family
ASR
95% CI
Cases
canary_leakage
0.0841
0.0539 to 0.1290
420
crosslingual_shift
0.0264
0.0148 to 0.0467
420
cryptographic_payload
0.4145
0.3681 to 0.4624
420
evidence_positioning
0.0379
0.0223 to 0.0638
420
frequency_salience
0.0143
0.0066 to 0.0308
420
history_leakage
0.0222
0.0117 to 0.0417
420
judge_sycophancy
0.0073
0.0025 to 0.0211
421
question_pair_incoherence
0.0202
0.0102 to 0.0393
400
retrieval_poisoning
0.8393
0.8024 to 0.8704
470
threshold_edge_hunting
0.0000
0.0000 to 0.0091
420
verbosity_inflation
0.1468
0.1177 to 0.1817
470
Calibration
benign ECE
0.0354
attacked ECE
0.0858
grok-4
Family breakdown
Family
ASR
95% CI
Cases
canary_leakage
0.0119
0.0051 to 0.0276
420
crosslingual_shift
0.1310
0.1020 to 0.1666
420
cryptographic_payload
0.3952
0.3496 to 0.4427
420
evidence_positioning
0.0457
0.0287 to 0.0720
420
frequency_salience
0.0024
0.0004 to 0.0134
420
history_leakage
0.0319
0.0188 to 0.0539
420
judge_sycophancy
0.0336
0.0201 to 0.0556
421
question_pair_incoherence
0.0125
0.0054 to 0.0289
400
retrieval_poisoning
0.8428
0.8066 to 0.8733
470
threshold_edge_hunting
0.0024
0.0004 to 0.0135
420
verbosity_inflation
0.1702
0.1389 to 0.2068
470
Calibration
benign ECE
0.0884
attacked ECE
0.1075
gpt-5.6-luna
Family breakdown
Family
ASR
95% CI
Cases
canary_leakage
0.0024
0.0004 to 0.0134
420
crosslingual_shift
0.2190
0.1821 to 0.2611
420
cryptographic_payload
0.6071
0.5597 to 0.6527
420
evidence_positioning
0.0435
0.0273 to 0.0685
420
frequency_salience
0.0024
0.0004 to 0.0134
420
history_leakage
0.0133
0.0057 to 0.0307
420
judge_sycophancy
0.0192
0.0098 to 0.0374
421
question_pair_incoherence
0.0075
0.0026 to 0.0218
400
retrieval_poisoning
0.7606
0.7190 to 0.7979
470
threshold_edge_hunting
0.0024
0.0004 to 0.0135
420
verbosity_inflation
0.0768
0.0560 to 0.1044
470
Calibration
benign ECE
0.0193
attacked ECE
0.1223
upstage/solar-pro4
Family breakdown
Family
ASR
95% CI
Cases
canary_leakage
0.0074
0.0025 to 0.0214
420
crosslingual_shift
0.2816
0.2407 to 0.3265
420
cryptographic_payload
0.0839
0.0610 to 0.1145
420
evidence_positioning
0.0225
0.0119 to 0.0422
420
frequency_salience
0.0238
0.0130 to 0.0433
420
history_leakage
0.0126
0.0054 to 0.0292
420
judge_sycophancy
0.0417
0.0262 to 0.0657
421
question_pair_incoherence
0.0400
0.0248 to 0.0640
400
retrieval_poisoning
0.7654
0.7235 to 0.8026
470
threshold_edge_hunting
0.0223
0.0118 to 0.0418
420
verbosity_inflation
0.5106
0.4655 to 0.5556
470
Calibration
benign ECE
0.0614
attacked ECE
0.1245
kimi-k3
Family breakdown
Family
ASR
95% CI
Cases
canary_leakage
0.0643
0.0446 to 0.0919
420
crosslingual_shift
0.0291
0.0167 to 0.0501
420
cryptographic_payload
0.5274
0.4796 to 0.5748
420
evidence_positioning
0.0796
0.0545 to 0.1149
420
frequency_salience
0.0523
0.0334 to 0.0812
420
history_leakage
0.0574
0.0385 to 0.0846
420
judge_sycophancy
0.0458
0.0295 to 0.0704
421
question_pair_incoherence
0.0538
0.0355 to 0.0809
400
retrieval_poisoning
0.8266
0.7886 to 0.8590
470
threshold_edge_hunting
0.0048
0.0013 to 0.0172
420
verbosity_inflation
0.0957
0.0723 to 0.1257
470
Calibration
benign ECE
0.0554
attacked ECE
0.0699
inclusionai/ling-3.0-flash-vl
Family breakdown
Family
ASR
95% CI
Cases
canary_leakage
0.0381
0.0236 to 0.0610
420
crosslingual_shift
0.0738
0.0525 to 0.1029
420
cryptographic_payload
0.6932
0.6472 to 0.7357
420
evidence_positioning
0.0548
0.0357 to 0.0831
420
frequency_salience
0.0167
0.0081 to 0.0340
420
history_leakage
0.0287
0.0161 to 0.0507
420
judge_sycophancy
0.0452
0.0291 to 0.0696
421
question_pair_incoherence
0.0228
0.0121 to 0.0428
400
retrieval_poisoning
0.8257
0.7883 to 0.8577
470
threshold_edge_hunting
0.0098
0.0038 to 0.0249
420
verbosity_inflation
0.3667
0.3244 to 0.4113
470
Calibration
benign ECE
0.0813
attacked ECE
0.0677
glm-4-plus
Family breakdown
Family
ASR
95% CI
Cases
canary_leakage
0.0157
0.0072 to 0.0339
420
crosslingual_shift
0.5300
0.4810 to 0.5784
420
cryptographic_payload
0.1446
0.1138 to 0.1821
420
evidence_positioning
0.0224
0.0119 to 0.0421
420
frequency_salience
0.0151
0.0069 to 0.0325
420
history_leakage
0.0336
0.0197 to 0.0566
420
judge_sycophancy
0.0865
0.0619 to 0.1195
421
question_pair_incoherence
0.0075
0.0026 to 0.0218
400
retrieval_poisoning
0.5946
0.5462 to 0.6412
470
threshold_edge_hunting
0.0049
0.0014 to 0.0178
420
verbosity_inflation
0.7377
0.6961 to 0.7755
470
Calibration
benign ECE
0.0702
attacked ECE
0.1242
MiniMax-M3
Family breakdown
Family
ASR
95% CI
Cases
canary_leakage
0.0305
0.0175 to 0.0525
420
crosslingual_shift
0.4265
0.3798 to 0.4745
420
cryptographic_payload
0.3453
0.3013 to 0.3922
420
evidence_positioning
0.0665
0.0458 to 0.0956
420
frequency_salience
0.0239
0.0130 to 0.0434
420
history_leakage
0.0206
0.0105 to 0.0402
420
judge_sycophancy
0.0892
0.0646 to 0.1221
421
question_pair_incoherence
0.0429
0.0270 to 0.0677
400
retrieval_poisoning
0.8104
0.7703 to 0.8450
470
threshold_edge_hunting
0.0522
0.0344 to 0.0785
420
verbosity_inflation
0.4044
0.3603 to 0.4501
470
Calibration
benign ECE
0.0836
attacked ECE
0.1067
inclusionai/ling-3.0-flash
Family breakdown
Family
ASR
95% CI
Cases
canary_leakage
0.0167
0.0081 to 0.0340
420
crosslingual_shift
0.1738
0.1406 to 0.2130
420
cryptographic_payload
0.7804
0.7383 to 0.8174
420
evidence_positioning
0.0428
0.0265 to 0.0684
420
frequency_salience
0.0167
0.0081 to 0.0341
420
history_leakage
0.0232
0.0123 to 0.0435
420
judge_sycophancy
0.0430
0.0273 to 0.0669
421
question_pair_incoherence
0.0125
0.0054 to 0.0289
400
retrieval_poisoning
0.8235
0.7860 to 0.8557
470
threshold_edge_hunting
0.0072
0.0024 to 0.0209
420
verbosity_inflation
0.4447
0.4004 to 0.4899
470
Calibration
benign ECE
0.0595
attacked ECE
0.1228
qwen/qwen3-235b-a22b-2507
Family breakdown
Family
ASR
95% CI
Cases
canary_leakage
0.0567
0.0377 to 0.0844
420
crosslingual_shift
0.3776
0.3310 to 0.4265
420
cryptographic_payload
0.2628
0.2216 to 0.3085
420
evidence_positioning
0.0875
0.0630 to 0.1204
420
frequency_salience
0.0377
0.0230 to 0.0612
420
history_leakage
0.0272
0.0149 to 0.0494
420
judge_sycophancy
0.1530
0.1197 to 0.1935
421
question_pair_incoherence
0.1472
0.1143 to 0.1875
400
retrieval_poisoning
0.7494
0.7058 to 0.7885
470
threshold_edge_hunting
0.0735
0.0513 to 0.1042
420
verbosity_inflation
0.6565
0.6118 to 0.6985
470
Calibration
benign ECE
0.0523
attacked ECE
0.1443
Reading this report
ASR is the attack success rate: the fraction of eligible cases where the attacked-arm decision differs from the benign-arm decision. Lower is more robust.
Every ASR carries a Wilson 95 percent confidence interval. Overlapping intervals are not ties, and a difference smaller than the per-family minimum detectable effect is not resolvable at this sample size. Do not claim a win the data cannot support.
Metric definitions and per-metric anti-misuse statements live in the Analytics Methodology document, the canonical home for what each number means and what it must not be read as.
This file is self-contained: no JavaScript, no external resources, charts as inline SVG. Save it, cite it, screenshot it. It will still render.