Peira leaderboard

PEIRA
Decision models under attack · suite v2 · dataset 2.6.0
12
MODELS TESTED · 8 RANKED · 4 FLAGGED
4,701
PAIRED CASES EACH · ELIGIBLE 4,297–4,638
11.0–12.6%
LEAD BAND · 4 MODELS TIED AT THE TOP
59.5–84.3%
RETRIEVAL POISONING · LOW END IS FLAGGED GLM-4-PLUS AT 59.5% · NO RANK
≥$0.20 to $21.02
COST SPREAD PER RUN · JEV-1.13.0 (LOWER BOUND) TO KIMI-K3

Attack success rates are shares of eligible attacked cases · headline bases 4,297 to 4,638 per model · family bases 358 to 470

Model ranking

Attack success rate with 95% confidence whiskers · lower is stronger · hatched bars carry no rank · a rank requires a malformed response rate at or under 5% · ranks count the 8 ranked models only · the top 4 intervals overlap, a tie · ranks 5 to 7 overlap, a tie · ranks follow point estimates where intervals overlap

Model ranking by attack success rateBars in attack success order. The top 4 intervals overlap, a tie. Ranks 5 to 7 overlap, a tie. kimi-k3 11.0% 95% CI 10.1% to 11.9% rank 1, tied band 1 to 4, malformed 0.0%. gemini-3.8-flash 12.2% 95% CI 11.3% to 13.2% rank 2, tied band 1 to 4, malformed 0.3%. deepseek-flash 12.4% 95% CI 11.5% to 13.4% rank 3, tied band 1 to 4, malformed 0.0%. jev-1.13.0 12.6% 95% CI 11.7% to 13.6% rank 4, tied band 1 to 4, malformed 1.3%. grok-4 16.1% 95% CI 15.0% to 17.2% rank 5, tied band 5 to 7, malformed 0.0%. gpt-5.6-luna 16.5% 95% CI 15.5% to 17.6% rank 6, tied band 5 to 7, malformed 2.2%. solar-pro4 17.4% 95% CI 16.3% to 18.5% rank 7, tied band 5 to 7, malformed 0.6%. ling-3.0-flash-vl 20.9% 95% CI 19.7% to 22.1% flagged no rank, malformed 10.7%. glm-4-plus 21.0% 95% CI 19.8% to 22.2% flagged no rank, malformed 6.2%. MiniMax-M3 21.8% 95% CI 20.6% to 23.0% rank 8, malformed 4.2%. ling-3.0-flash 22.8% 95% CI 21.7% to 24.1% flagged no rank, malformed 7.3%. qwen3-235b-a22b-2507 25.2% 95% CI 23.9% to 26.5% flagged no rank, malformed 11.9%.lower is stronger · hatched bars carry no rank0%5%10%15%20%25%30%1kimi-k311.0%malformed 0.0% · tied 1–42gemini-3.8-flash12.2%malformed 0.3% · tied 1–43deepseek-flash12.4%malformed 0.0% · tied 1–44jev-1.13.012.6%malformed 1.3% · tied 1–45grok-416.1%malformed 0.0% · tied 5–76gpt-5.6-luna16.5%malformed 2.2% · tied 5–77solar-pro417.4%malformed 0.6% · tied 5–7•ling-3.0-flash-vl20.9%no rank · malformed 10.7%•glm-4-plus21.0%no rank · malformed 6.2%8MiniMax-M321.8%malformed 4.2%•ling-3.0-flash22.8%no rank · malformed 7.3%•qwen3-235b-a22b-250725.2%no rank · malformed 11.9%
Bars run 0% to 30% · lower is stronger
kimi-k3#1 · TIED 1–411.0%
95% CI 10.1% to 11.9% · malformed 0.0%
gemini-3.8-flash#2 · TIED 1–412.2%
95% CI 11.3% to 13.2% · malformed 0.3%
deepseek-flash#3 · TIED 1–412.4%
95% CI 11.5% to 13.4% · malformed 0.0%
jev-1.13.0#4 · TIED 1–412.6%
95% CI 11.7% to 13.6% · malformed 1.3%
grok-4#5 · TIED 5–716.1%
95% CI 15.0% to 17.2% · malformed 0.0%
gpt-5.6-luna#6 · TIED 5–716.5%
95% CI 15.5% to 17.6% · malformed 2.2%
solar-pro4#7 · TIED 5–717.4%
95% CI 16.3% to 18.5% · malformed 0.6%
ling-3.0-flash-vlFLAGGED20.9%
95% CI 19.7% to 22.1% · no rank · malformed 10.7%
glm-4-plusFLAGGED21.0%
95% CI 19.8% to 22.2% · no rank · malformed 6.2%
MiniMax-M3#821.8%
95% CI 20.6% to 23.0% · malformed 4.2%
ling-3.0-flashFLAGGED22.8%
95% CI 21.7% to 24.1% · no rank · malformed 7.3%
qwen3-235b-a22b-2507FLAGGED25.2%
95% CI 23.9% to 26.5% · no rank · malformed 11.9%

Attack family heatmap

Each cell rests on 358 to 470 eligible cases.At this width the last families sit to the right · scroll sideways for them

the remaining families sit to the right · scroll sideways

rows in overall ASR order · cells are family attack success rate in percent, rounded · the top 4 intervals overlap, a tie · ranks 5 to 7 overlap, a tie · ranks follow point estimates where intervals overlap · flagged rows carry no rank · families run by average attack success over the models carrying each family, highest first

Family attack success rates by model in percent. Columns in order, canary leakage, crosslingual, crypto payload, evidence positioning, frequency salience, history leakage, judge sycophancy, question pair incoherence, retrieval poisoning, threshold edge hunting, verbosity inflation.
canary leakagecrosslingualcrypto payloadevidence positioningfrequency saliencehistory leakagejudge sycophancyquestion pair incoherenceretrieval poisoningthreshold edge huntingverbosity inflation
kimi-k3canary leakage 0.0% · n 420crosslingual 0.7% · n 420crypto payload 10.5% · n 420evidence positioning 2.3% · n 400frequency salience 2.4% · n 420history leakage 3.4% · n 411judge sycophancy 1.2% · n 414question pair incoherence 2.3% · n 399retrieval poisoning 82.7% · n 444threshold edge hunting 0.5% · n 420verbosity inflation 9.6% · n 470
gemini-3.8-flashcanary leakage 0.0% · n 420crosslingual 1.7% · n 420crypto payload 18.8% · n 420evidence positioning 0.6% · n 358frequency salience 1.4% · n 420history leakage 2.2% · n 405judge sycophancy 0.7% · n 413question pair incoherence 2.0% · n 397retrieval poisoning 83.9% · n 448threshold edge hunting 0.0% · n 420verbosity inflation 14.7% · n 470
deepseek-flashcanary leakage 0.0% · n 420crosslingual 2.9% · n 420crypto payload 12.1% · n 420evidence positioning 3.0% · n 396frequency salience 0.5% · n 420history leakage 0.3% · n 395judge sycophancy 3.6% · n 420question pair incoherence 1.8% · n 400retrieval poisoning 77.6% · n 450threshold edge hunting 0.0% · n 420verbosity inflation 26.8% · n 470
jev-1.13.0canary leakage 0.0% · n 420crosslingual 15.7% · n 370crypto payload 3.4% · n 411evidence positioning 3.5% · n 396frequency salience 1.0% · n 420history leakage 0.0% · n 404judge sycophancy 3.3% · n 418question pair incoherence 0.0% · n 400retrieval poisoning 83.7% · n 448threshold edge hunting 0.5% · n 419verbosity inflation 20.6% · n 470
grok-4canary leakage 1.2% · n 420crosslingual 13.1% · n 420crypto payload 39.5% · n 420evidence positioning 4.6% · n 372frequency salience 0.2% · n 420history leakage 3.2% · n 407judge sycophancy 3.4% · n 417question pair incoherence 1.3% · n 400retrieval poisoning 84.3% · n 458threshold edge hunting 0.2% · n 417verbosity inflation 17.0% · n 470
gpt-5.6-lunacanary leakage 0.2% · n 418crosslingual 21.9% · n 420crypto payload 60.7% · n 420evidence positioning 4.3% · n 391frequency salience 0.2% · n 420history leakage 1.3% · n 376judge sycophancy 1.9% · n 417question pair incoherence 0.8% · n 400retrieval poisoning 76.1% · n 447threshold edge hunting 0.2% · n 417verbosity inflation 7.7% · n 469
solar-pro4canary leakage 0.7% · n 408crosslingual 28.2% · n 419crypto payload 8.4% · n 417evidence positioning 2.3% · n 400frequency salience 2.4% · n 420history leakage 1.3% · n 396judge sycophancy 4.2% · n 408question pair incoherence 4.0% · n 400retrieval poisoning 76.5% · n 439threshold edge hunting 2.2% · n 404verbosity inflation 51.1% · n 470
ling-3.0-flash-vl · no rankcanary leakage 3.8% · n 420crosslingual 7.4% · n 420crypto payload 69.3% · n 414evidence positioning 5.5% · n 365frequency salience 1.7% · n 420history leakage 2.9% · n 383judge sycophancy 4.5% · n 420question pair incoherence 2.3% · n 394retrieval poisoning 82.6% · n 459threshold edge hunting 1.0% · n 408verbosity inflation 36.7% · n 469
glm-4-plus · no rankcanary leakage 1.6% · n 381crosslingual 53.0% · n 400crypto payload 14.5% · n 408evidence positioning 2.2% · n 401frequency salience 1.5% · n 398history leakage 3.4% · n 387judge sycophancy 8.6% · n 370question pair incoherence 0.8% · n 400retrieval poisoning 59.5% · n 407threshold edge hunting 0.5% · n 406verbosity inflation 73.8% · n 469
MiniMax-M3canary leakage 3.0% · n 394crosslingual 42.7% · n 415crypto payload 34.5% · n 417evidence positioning 6.6% · n 391frequency salience 2.4% · n 419history leakage 2.1% · n 388judge sycophancy 8.9% · n 381question pair incoherence 4.3% · n 396retrieval poisoning 81.0% · n 422threshold edge hunting 5.2% · n 402verbosity inflation 40.4% · n 455
ling-3.0-flash · no rankcanary leakage 1.7% · n 420crosslingual 17.4% · n 420crypto payload 78.0% · n 419evidence positioning 4.3% · n 374frequency salience 1.7% · n 419history leakage 2.3% · n 388judge sycophancy 4.3% · n 419question pair incoherence 1.3% · n 400retrieval poisoning 82.4% · n 459threshold edge hunting 0.7% · n 417verbosity inflation 44.5% · n 470
qwen3-235b-a22b-2507 · no rankcanary leakage 5.7% · n 388crosslingual 37.8% · n 392crypto payload 26.3% · n 392evidence positioning 8.8% · n 377frequency salience 3.8% · n 398history leakage 2.7% · n 367judge sycophancy 15.3% · n 366question pair incoherence 14.7% · n 360retrieval poisoning 74.9% · n 419threshold edge hunting 7.3% · n 381verbosity inflation 65.6% · n 457
Family attack success rates by model in percent. Columns ordered by average attack success rate, highest first. Averages run over the models carrying each family. retrieval poisoning, verbosity inflation, crypto payload, crosslingual, judge sycophancy, evidence positioning, question pair incoherence, history leakage, frequency salience, threshold edge hunting, canary leakage.
retrieval poisoningverbosity inflationcrypto payloadcrosslingualjudge sycophancyevidence positioningquestion pair incoherencehistory leakagefrequency saliencethreshold edge huntingcanary leakage
kimi-k3retrieval poisoning 82.7% · n 444verbosity inflation 9.6% · n 470crypto payload 10.5% · n 420crosslingual 0.7% · n 420judge sycophancy 1.2% · n 414evidence positioning 2.3% · n 400question pair incoherence 2.3% · n 399history leakage 3.4% · n 411frequency salience 2.4% · n 420threshold edge hunting 0.5% · n 420canary leakage 0.0% · n 420
gemini-3.8-flashretrieval poisoning 83.9% · n 448verbosity inflation 14.7% · n 470crypto payload 18.8% · n 420crosslingual 1.7% · n 420judge sycophancy 0.7% · n 413evidence positioning 0.6% · n 358question pair incoherence 2.0% · n 397history leakage 2.2% · n 405frequency salience 1.4% · n 420threshold edge hunting 0.0% · n 420canary leakage 0.0% · n 420
deepseek-flashretrieval poisoning 77.6% · n 450verbosity inflation 26.8% · n 470crypto payload 12.1% · n 420crosslingual 2.9% · n 420judge sycophancy 3.6% · n 420evidence positioning 3.0% · n 396question pair incoherence 1.8% · n 400history leakage 0.3% · n 395frequency salience 0.5% · n 420threshold edge hunting 0.0% · n 420canary leakage 0.0% · n 420
jev-1.13.0retrieval poisoning 83.7% · n 448verbosity inflation 20.6% · n 470crypto payload 3.4% · n 411crosslingual 15.7% · n 370judge sycophancy 3.3% · n 418evidence positioning 3.5% · n 396question pair incoherence 0.0% · n 400history leakage 0.0% · n 404frequency salience 1.0% · n 420threshold edge hunting 0.5% · n 419canary leakage 0.0% · n 420
grok-4retrieval poisoning 84.3% · n 458verbosity inflation 17.0% · n 470crypto payload 39.5% · n 420crosslingual 13.1% · n 420judge sycophancy 3.4% · n 417evidence positioning 4.6% · n 372question pair incoherence 1.3% · n 400history leakage 3.2% · n 407frequency salience 0.2% · n 420threshold edge hunting 0.2% · n 417canary leakage 1.2% · n 420
gpt-5.6-lunaretrieval poisoning 76.1% · n 447verbosity inflation 7.7% · n 469crypto payload 60.7% · n 420crosslingual 21.9% · n 420judge sycophancy 1.9% · n 417evidence positioning 4.3% · n 391question pair incoherence 0.8% · n 400history leakage 1.3% · n 376frequency salience 0.2% · n 420threshold edge hunting 0.2% · n 417canary leakage 0.2% · n 418
solar-pro4retrieval poisoning 76.5% · n 439verbosity inflation 51.1% · n 470crypto payload 8.4% · n 417crosslingual 28.2% · n 419judge sycophancy 4.2% · n 408evidence positioning 2.3% · n 400question pair incoherence 4.0% · n 400history leakage 1.3% · n 396frequency salience 2.4% · n 420threshold edge hunting 2.2% · n 404canary leakage 0.7% · n 408
ling-3.0-flash-vl · no rankretrieval poisoning 82.6% · n 459verbosity inflation 36.7% · n 469crypto payload 69.3% · n 414crosslingual 7.4% · n 420judge sycophancy 4.5% · n 420evidence positioning 5.5% · n 365question pair incoherence 2.3% · n 394history leakage 2.9% · n 383frequency salience 1.7% · n 420threshold edge hunting 1.0% · n 408canary leakage 3.8% · n 420
glm-4-plus · no rankretrieval poisoning 59.5% · n 407verbosity inflation 73.8% · n 469crypto payload 14.5% · n 408crosslingual 53.0% · n 400judge sycophancy 8.6% · n 370evidence positioning 2.2% · n 401question pair incoherence 0.8% · n 400history leakage 3.4% · n 387frequency salience 1.5% · n 398threshold edge hunting 0.5% · n 406canary leakage 1.6% · n 381
MiniMax-M3retrieval poisoning 81.0% · n 422verbosity inflation 40.4% · n 455crypto payload 34.5% · n 417crosslingual 42.7% · n 415judge sycophancy 8.9% · n 381evidence positioning 6.6% · n 391question pair incoherence 4.3% · n 396history leakage 2.1% · n 388frequency salience 2.4% · n 419threshold edge hunting 5.2% · n 402canary leakage 3.0% · n 394
ling-3.0-flash · no rankretrieval poisoning 82.4% · n 459verbosity inflation 44.5% · n 470crypto payload 78.0% · n 419crosslingual 17.4% · n 420judge sycophancy 4.3% · n 419evidence positioning 4.3% · n 374question pair incoherence 1.3% · n 400history leakage 2.3% · n 388frequency salience 1.7% · n 419threshold edge hunting 0.7% · n 417canary leakage 1.7% · n 420
qwen3-235b-a22b-2507 · no rankretrieval poisoning 74.9% · n 419verbosity inflation 65.6% · n 457crypto payload 26.3% · n 392crosslingual 37.8% · n 392judge sycophancy 15.3% · n 366evidence positioning 8.8% · n 377question pair incoherence 14.7% · n 360history leakage 2.7% · n 367frequency salience 3.8% · n 398threshold edge hunting 7.3% · n 381canary leakage 5.7% · n 388

Cost vs attack success rate

Log cost axis, each step right multiplies cost by ten · one point per full 4,701 case run · filled points are ranked, hollow points are flagged · a pale dot in the centre marks a lower bound cost · cheap and strong sits bottom left · the top 4 intervals overlap, a tie · ranks 5 to 7 overlap, a tie · ranks follow point estimates where intervals overlap

Cost per run against attack success rateCost per run against attack success rate. The cost axis is log scale. Filled points are ranked models and hollow points are flagged models with no rank. kimi-k3 $21.02 at 11.0%, rank 1, tied band 1 to 4. gemini-3.8-flash $3.17 (lower bound) at 12.2%, rank 2, tied band 1 to 4. deepseek-flash $1.48 at 12.4%, rank 3, tied band 1 to 4. jev-1.13.0 $0.20 (lower bound) at 12.6%, rank 4, tied band 1 to 4. grok-4 $11.09 at 16.1%, rank 5, tied band 5 to 7. gpt-5.6-luna $1.51 (lower bound) at 16.5%, rank 6, tied band 5 to 7. solar-pro4 $0.45 (lower bound) at 17.4%, rank 7, tied band 5 to 7. ling-3.0-flash-vl $0.26 (lower bound) at 20.9%, flagged no rank. glm-4-plus $4.27 (lower bound) at 21.0%, flagged no rank. MiniMax-M3 $3.62 (lower bound) at 21.8%, rank 8. ling-3.0-flash $0.24 (lower bound) at 22.8%, flagged no rank. qwen3-235b-a22b-2507 $0.46 (lower bound) at 25.2%, flagged no rank.$0.1$1$10$100cost per run in USD, log scaleattack success rate0%10%20%30%tied bands 1–4 · 5–7dots mark the lower bound costsrankedflagged · no ranklower bound costkimi-k3gemini-3.8-flashdeepseek-flashjev-1.13.0grok-4gpt-5.6-lunasolar-pro4MiniMax-M3ling-3.0-flash-vlglm-4-plusling-3.0-flashqwen3-235b-a22b-2507Cost per run against attack success rateCost per run against attack success rate. The cost axis is log scale. Filled points are ranked models and hollow points are flagged models with no rank. kimi-k3 $21.02 at 11.0%, rank 1, tied band 1 to 4. gemini-3.8-flash $3.17 (lower bound) at 12.2%, rank 2, tied band 1 to 4. deepseek-flash $1.48 at 12.4%, rank 3, tied band 1 to 4. jev-1.13.0 $0.20 (lower bound) at 12.6%, rank 4, tied band 1 to 4. grok-4 $11.09 at 16.1%, rank 5, tied band 5 to 7. gpt-5.6-luna $1.51 (lower bound) at 16.5%, rank 6, tied band 5 to 7. solar-pro4 $0.45 (lower bound) at 17.4%, rank 7, tied band 5 to 7. ling-3.0-flash-vl $0.26 (lower bound) at 20.9%, flagged no rank. glm-4-plus $4.27 (lower bound) at 21.0%, flagged no rank. MiniMax-M3 $3.62 (lower bound) at 21.8%, rank 8. ling-3.0-flash $0.24 (lower bound) at 22.8%, flagged no rank. qwen3-235b-a22b-2507 $0.46 (lower bound) at 25.2%, flagged no rank.$0.1$1$10$100cost per run in USD, log scaleattack success rate0%10%20%30%tied bands 1–4 · 5–7lower bound dotsrankedflagged · no ranklower bound costkimi-k3gemini-3.8-flashdeepseek-flashjev-1.13.0grok-4gpt-5.6-lunasolar-pro4MiniMax-M3ling-3.0-flash-vlglm-4-plusling-3.0-flashqwen3-235b-a22b-2507

Malformed response rate

Fraction of the 4,701 cases with any malformed call record, benign or attacked · the admission rule for earning a rank · amber names are flagged models with no rank

Malformed response rate by modelMalformed response rate by model, worst first. qwen3-235b-a22b-2507 11.9%, flagged no rank. ling-3.0-flash-vl 10.7%, flagged no rank. ling-3.0-flash 7.3%, flagged no rank. glm-4-plus 6.2%, flagged no rank. MiniMax-M3 4.2%, rank 8. gpt-5.6-luna 2.2%, rank 6, tied band 5 to 7. jev-1.13.0 1.3%, rank 4, tied band 1 to 4. solar-pro4 0.6%, rank 7, tied band 5 to 7. gemini-3.8-flash 0.3%, rank 2, tied band 1 to 4. kimi-k3 0.0%, rank 1, tied band 1 to 4. deepseek-flash 0.0%, rank 3, tied band 1 to 4. grok-4 0.0%, rank 5, tied band 5 to 7. The dashed line marks the 5% admission line. Bars past the line are amber and bars under it are green.worst first• qwen3-235b-a22b-2507• ling-3.0-flash-vl• ling-3.0-flash• glm-4-plusMiniMax-M3gpt-5.6-lunajev-1.13.0solar-pro4gemini-3.8-flashkimi-k3deepseek-flashgrok-411.9%10.7%7.3%6.2%4.2%2.2%1.3%0.6%0.3%0.0%0.0%0.0%5% admission line0%10%13%bars past the line are over the 5% admission limit
Bars run 0% to 13% · the dashed tick is the 5% admission line · bars past it are over the 5% admission limit
qwen3-235b-a22b-250711.9%, flagged, no rank
ling-3.0-flash-vl10.7%, flagged, no rank
ling-3.0-flash7.3%, flagged, no rank
glm-4-plus6.2%, flagged, no rank
MiniMax-M34.2%, rank 8 of 8 ranked
gpt-5.6-luna2.2%, rank 6 of 8 ranked, tied band 5 to 7
jev-1.13.01.3%, rank 4 of 8 ranked, tied band 1 to 4
solar-pro40.6%, rank 7 of 8 ranked, tied band 5 to 7
gemini-3.8-flash0.3%, rank 2 of 8 ranked, tied band 1 to 4
kimi-k30.0%, rank 1 of 8 ranked, tied band 1 to 4
deepseek-flash0.0%, rank 3 of 8 ranked, tied band 1 to 4
grok-40.0%, rank 5 of 8 ranked, tied band 5 to 7

Model cards

Spark bars show the 11 family ASRs in fixed order (canary leakage, crosslingual, crypto payload, evidence positioning, frequency salience, history leakage, judge sycophancy, question pair incoherence, retrieval poisoning, threshold edge hunting, verbosity inflation) on a shared 0 to 100% scale · bar fill is green on ranked cards and grey on flagged cards, apart from the orange retrieval spike · a dotted outline marks a family a model does not carry · a nonzero rate always draws at least a 1px sliver, so a missing bar is a true zero and small bars are read by their printed values · the orange spike is retrieval poisoning, bar 9 of 11 · latency and calibration are measured on the attacked runs, where lower ECE and Brier scores are better · calibration rests on the cases that reported a confidence · C·H·M is critical, high, medium · robust tax adds the benign accuracy and attacked ECE shortfalls from the best ranked runs, in points · severity flip rate confidence intervals reach about 2.7 points to one side at the widest

kimi-k3

RANK 1 OF 8 RANKED · TIED 1–4
Malformed response rate at or under 5%
First 3,274 cases re-run under the fixed adapter · see Notes
11.0%ASR · 95% CI 10.1% to 11.9%
Orange is retrieval poisoning · no other family reaches 20%
ELIGIBLE CASES4,638
BENIGN ACC · DECIDED VARIANTS98.7%
COST$21.02
LATENCY P50 · ATTACKED3.85s
LATENCY P95 · ATTACKED6.00s
CALIB N · ATTACKED · SHARE OF ELIGIBLE4,638 · 100.0%
ECE · ATTACKED0.064
BRIER · ATTACKED0.086
ROBUST TAX · IT SETS BOTH RANKED BENCHMARKS0.0 pts
MALFORMED0.0%
SEV FLIP RATE C·H·M7.7%·9.8%·13.4%
SEV BASES C·H·M1,035·1,550·2,053

gemini-3.8-flash

RANK 2 OF 8 RANKED · TIED 1–4
Malformed response rate at or under 5%
First 1,610 cases re-run under the fixed adapter · see Notes
12.2%ASR · 95% CI 11.3% to 13.2%
Orange is retrieval poisoning · no other family reaches 20%
ELIGIBLE CASES4,591
BENIGN ACC · DECIDED VARIANTS97.7%
COST · LOWER BOUND$3.17
LATENCY P50 · ATTACKED2.37s
LATENCY P95 · ATTACKED6.85s
CALIB N · ATTACKED · SHARE OF ELIGIBLE4,527 · 98.6%
ECE · ATTACKED0.080
BRIER · ATTACKED0.093
ROBUST TAX · VS BEST RANKED RUNS2.6 pts
MALFORMED0.3%
SEV FLIP RATE C·H·M9.7%·10.4%·14.8%
SEV BASES C·H·M1,029·1,544·2,018

deepseek-flash

RANK 3 OF 8 RANKED · TIED 1–4
Malformed response rate at or under 5%
12.4%ASR · 95% CI 11.5% to 13.4%
Orange is retrieval poisoning · top other spike is verbosity inflation 26.8%
ELIGIBLE CASES4,631
BENIGN ACC · DECIDED VARIANTS98.5%
COST$1.48
LATENCY P50 · ATTACKED1.31s
LATENCY P95 · ATTACKED1.85s
CALIB N · ATTACKED · SHARE OF ELIGIBLE4,631 · 100.0%
ECE · ATTACKED0.071
BRIER · ATTACKED0.094
ROBUST TAX · VS BEST RANKED RUNS0.8 pts
MALFORMED0.0%
SEV FLIP RATE C·H·M7.8%·11.0%·15.8%
SEV BASES C·H·M1,037·1,549·2,045

jev-1.13.0

RANK 4 OF 8 RANKED · TIED 1–4
Malformed response rate at or under 5%
12.6%ASR · 95% CI 11.7% to 13.6%
Orange is retrieval poisoning · top other spike is verbosity inflation 20.6%
ELIGIBLE CASES4,576
BENIGN ACC · DECIDED VARIANTS98.6%
COST · LOWER BOUND$0.20
LATENCY P50 · ATTACKED594ms
LATENCY P95 · ATTACKED1.14s
CALIB N · ATTACKED · SHARE OF ELIGIBLE4,576 · 100.0%
ECE · ATTACKED0.096
BRIER · ATTACKED0.118
ROBUST TAX · VS BEST RANKED RUNS3.2 pts
MALFORMED1.3%
SEV FLIP RATE C·H·M8.8%·10.4%·16.3%
SEV BASES C·H·M1,035·1,539·2,002

grok-4

RANK 5 OF 8 RANKED · TIED 5–7
Malformed response rate at or under 5%
16.1%ASR · 95% CI 15.0% to 17.2%
Orange is retrieval poisoning · top other spike is crypto payload 39.5%
ELIGIBLE CASES4,621
BENIGN ACC · DECIDED VARIANTS98.3%
COST$11.09
LATENCY P50 · ATTACKED4.45s
LATENCY P95 · ATTACKED8.85s
CALIB N · ATTACKED · SHARE OF ELIGIBLE4,621 · 100.0%
ECE · ATTACKED0.108
BRIER · ATTACKED0.126
ROBUST TAX · VS BEST RANKED RUNS4.7 pts
MALFORMED0.0%
SEV FLIP RATE C·H·M14.4%·14.5%·18.1%
SEV BASES C·H·M1,036·1,554·2,031

gpt-5.6-luna

RANK 6 OF 8 RANKED · TIED 5–7
Malformed response rate at or under 5%
16.5%ASR · 95% CI 15.5% to 17.6%
Orange is retrieval poisoning · top other spike is crypto payload 60.7%
ELIGIBLE CASES4,595
BENIGN ACC · DECIDED VARIANTS97.8%
COST · LOWER BOUND$1.51
LATENCY P50 · ATTACKED2.13s
LATENCY P95 · ATTACKED4.68s
CALIB N · ATTACKED · SHARE OF ELIGIBLE4,495 · 97.8%
ECE · ATTACKED0.122
BRIER · ATTACKED0.130
ROBUST TAX · VS BEST RANKED RUNS6.6 pts
MALFORMED2.2%
SEV FLIP RATE C·H·M17.9%·16.3%·16.0%
SEV BASES C·H·M1,036·1,544·2,015

solar-pro4

RANK 7 OF 8 RANKED · TIED 5–7
Malformed response rate at or under 5%
Model ID · upstage/solar-pro4
17.4%ASR · 95% CI 16.3% to 18.5%
Orange is retrieval poisoning · top other spike is verbosity inflation 51.1%
ELIGIBLE CASES4,581
BENIGN ACC · DECIDED VARIANTS97.7%
COST · LOWER BOUND$0.45
LATENCY P50 · ATTACKED2.33s
LATENCY P95 · ATTACKED5.69s
CALIB N · ATTACKED · SHARE OF ELIGIBLE4,564 · 99.6%
ECE · ATTACKED0.125
BRIER · ATTACKED0.145
ROBUST TAX · VS BEST RANKED RUNS7.0 pts
MALFORMED0.6%
SEV FLIP RATE C·H·M14.5%·14.3%·21.3%
SEV BASES C·H·M1,029·1,541·2,011

ling-3.0-flash-vl

FLAGGED · NO RANK
Malformed response rate above 5%
Model ID · inclusionai/ling-3.0-flash-vl
20.9%ASR · 95% CI 19.7% to 22.1%
Orange is retrieval poisoning · top other spike is crypto payload 69.3%
ELIGIBLE CASES4,572
BENIGN ACC · DECIDED VARIANTS98.3%
COST · LOWER BOUND$0.26
LATENCY P50 · ATTACKED2.07s
LATENCY P95 · ATTACKED5.30s
CALIB N · ATTACKED · SHARE OF ELIGIBLE4,131 · 90.4%
ECE · ATTACKED0.068
BRIER · ATTACKED0.090
ROBUST TAX · VS BEST RANKED RUNSn/a
MALFORMED10.7%
SEV FLIP RATE C·H·M21.5%·18.6%·22.3%
SEV BASES C·H·M1,022·1,528·2,022

glm-4-plus

FLAGGED · NO RANK
Malformed response rate above 5%
21.0%ASR · 95% CI 19.8% to 22.2%
Orange is retrieval poisoning · top spike is verbosity inflation 73.8%
ELIGIBLE CASES4,427
BENIGN ACC · DECIDED VARIANTS98.9%
COST · LOWER BOUND$4.27
LATENCY P50 · ATTACKED2.03s
LATENCY P95 · ATTACKED4.05s
CALIB N · ATTACKED · SHARE OF ELIGIBLE4,361 · 98.5%
ECE · ATTACKED0.124
BRIER · ATTACKED0.148
ROBUST TAX · VS BEST RANKED RUNSn/a
MALFORMED6.2%
SEV FLIP RATE C·H·M19.6%·17.0%·24.9%
SEV BASES C·H·M1,019·1,511·1,897

MiniMax-M3

RANK 8 OF 8 RANKED
Malformed response rate at or under 5%
21.8%ASR · 95% CI 20.6% to 23.0%
Orange is retrieval poisoning · top other spike is crosslingual 42.7%
ELIGIBLE CASES4,480
BENIGN ACC · DECIDED VARIANTS97.8%
COST · LOWER BOUND$3.62
LATENCY P50 · ATTACKED2.84s
LATENCY P95 · ATTACKED6.58s
CALIB N · ATTACKED · SHARE OF ELIGIBLE4,408 · 98.4%
ECE · ATTACKED0.107
BRIER · ATTACKED0.152
ROBUST TAX · VS BEST RANKED RUNS5.1 pts
MALFORMED4.2%
SEV FLIP RATE C·H·M19.9%·17.8%·25.8%
SEV BASES C·H·M1,033·1,498·1,949

ling-3.0-flash

FLAGGED · NO RANK
Malformed response rate above 5%
Model ID · inclusionai/ling-3.0-flash
22.8%ASR · 95% CI 21.7% to 24.1%
Orange is retrieval poisoning · top other spike is crypto payload 78.0%
ELIGIBLE CASES4,605
BENIGN ACC · DECIDED VARIANTS98.3%
COST · LOWER BOUND$0.24
LATENCY P50 · ATTACKED1.96s
LATENCY P95 · ATTACKED6.23s
CALIB N · ATTACKED · SHARE OF ELIGIBLE4,281 · 93.0%
ECE · ATTACKED0.123
BRIER · ATTACKED0.138
ROBUST TAX · VS BEST RANKED RUNSn/a
MALFORMED7.3%
SEV FLIP RATE C·H·M23.8%·21.2%·23.7%
SEV BASES C·H·M1,034·1,550·2,021

qwen3-235b-a22b-2507

FLAGGED · NO RANK
Malformed response rate above 5%
Model ID · qwen/qwen3-235b-a22b-2507
25.2%ASR · 95% CI 23.9% to 26.5%
Orange is retrieval poisoning · top other spike is verbosity inflation 65.6%
ELIGIBLE CASES4,297
BENIGN ACC · DECIDED VARIANTS97.8%
COST · LOWER BOUND$0.46
LATENCY P50 · ATTACKED4.88s
LATENCY P95 · ATTACKED20.01s
CALIB N · ATTACKED · SHARE OF ELIGIBLE4,044 · 94.1%
ECE · ATTACKED0.144
BRIER · ATTACKED0.166
ROBUST TAX · VS BEST RANKED RUNSn/a
MALFORMED11.9%
SEV FLIP RATE C·H·M21.0%·21.4%·30.2%
SEV BASES C·H·M965·1,445·1,887