README.md

metadata

title: CAPTCHAv2 Leaderboard
emoji: 🏆
colorFrom: indigo
colorTo: purple
sdk: gradio
sdk_version: 5.49.1
app_file: app.py
pinned: false
license: mit

CAPTCHAv2 Leaderboard

A comprehensive leaderboard for comparing model performance across different CAPTCHA puzzle types. This interactive dashboard allows you to:

• 📊 View performance rankings across different CAPTCHA categories
• 📈 Compare models using interactive visualizations
• 💰 Analyze cost-effectiveness of different models
• 🔄 Upload and update results easily

Features

• Interactive Leaderboard Table: Sortable rankings with color-coded performance indicators
• Performance Comparison Charts: Visual bar charts showing pass rates across models
• Performance by Type: Detailed breakdown of performance across different CAPTCHA puzzle types
• Cost-Effectiveness Analysis: Scatter plot comparing performance vs. cost
• Easy Upload: Support for CSV and JSON result files

How to Use

1. View the Leaderboard: Browse the current rankings and filter by category
2. Sort Results: Sort by Pass Rate, Duration, or Cost
3. Upload Results: Use the upload section to add new evaluation results
4. Compare Models: Use the visualizations to compare different models

Uploading Results

The leaderboard supports multiple file formats:

• CSV files: Aggregated results with columns for Model, Provider, Agent Framework, Type, metrics, and per-type pass rates
• JSON files: Single object or array of aggregated results
• benchmark_results.json: Per-puzzle results in JSONL format (auto-converted)

See the upload section in the app for detailed instructions and file format requirements.

Categories

The leaderboard tracks performance across various CAPTCHA types including:

• Dice Count
• Color Cipher
• Color Counting
• Dynamic Jigsaw
• Mirror
• Set Game
• Shadow Plausible
• Spooky variants (Circle, Jigsaw, Shape Grid, Size, Text)
• Trajectory Recovery
• Transform Pipeline
• And more...

Model Types

Models are automatically categorized as:

• Proprietary: Commercial models (OpenAI, Anthropic, Google, etc.)
• Open source: Open source models (Llama, Mistral, Qwen, etc.)