leakgauge leaderboard

Final result. 3 models evaluated: gpt-4o, gpt-4o-mini, llama-3.3-70b-instruct. Kendall τ = 1.000.

3 model(s) · suite: all · n=37 cases · roster compute $2.51

Does verified leakage change the ranking?

Ranking by verified leakage does NOT reorder these models Kendall τ = 1.00 (perfect agreement) no model changes rank worse ← rank by hijack-ASR rank by leakage-verified ASR → worse openai:gpt-4o · 8% 8% · openai:gpt-4o openrouter:meta-llama/llama-3.3-70b-instruct · 2% 2% · openrouter:meta-llama/llama-3.3-70b-instruct openai:gpt-4o-mini · 0% 0% · openai:gpt-4o-mini Each line is one model. A crossing = its rank changed when success was scored as verified data leakage instead of task hijacking.

Final result: no. Ranking these 3 models by verified data leakage gives the same order as ranking by task hijacking (Kendall τ = 1.000 — the lines above are parallel, no crossing). The leakage-vs-hijack reorder does not occur for this roster.

Why, and what it means. On current frontier models these attacks score near the floor — the models barely leak, and hijack-ASR ≈ leakage-ASR. With so little spread there is almost nothing for leakage-vs-hijack scoring to reorder. Demonstrating the reorder needs harder cases that pull the two rates apart (future work). The contribution here is the programmatic leakage-verification methodology and a verified leaderboard — not a positive reorder.

Limitations. qwen-2.5-72b stalled provider-side and is excluded. Anthropic (Sonnet/Haiku) was deferred on cost — agentic runs take many tool-call steps (~$15–60/model) and, being similarly robust, would not flip this null.

Per-model rates (95% bootstrap CI)

modelhijack-ASRleakage-verified ASRutility-under-attackn
openai:gpt-4o0.08 [0.03, 0.15]0.08 [0.03, 0.14]0.66 [0.52, 0.79]37
openrouter:meta-llama/llama-3.3-70b-instruct0.02 [0.01, 0.04]0.02 [0.00, 0.03]0.43 [0.30, 0.57]37
openai:gpt-4o-mini0.00 [0.00, 0.01]0.00 [0.00, 0.01]0.78 [0.66, 0.90]37

Rates are means over cases with a percentile bootstrap 95% CI. The suite is small (n = 37), so the intervals are wide — shown openly rather than hidden. The highlighted column, leakage-verified ASR, is the metric the project is about.