olmo-eval: An evaluation workbench for the model development loop
• 8
AI & ML interests
Building breatkthrough AI to solve the world's biggest problems.
Recent Activity
View all activity
Papers
EMO: Pretraining Mixture of Experts for Emergent Modularity
MolmoAct2: Action Reasoning Models for Real-world Deployment
Articles
Collection of the evaluation rollouts for MolmoAct2 conducted by Cortex AI
-
MolmoAct2: Action Reasoning Models for Real-world Deployment
Paper • 2605.02881 • Published • 348 -
allenai/eval_molmoact_candy_sorting_in-distribution
Viewer • Updated • 59.6k • 457 -
allenai/eval_molmoact_cup_stacking_in-distribution
Viewer • Updated • 32k • 428 -
allenai/eval_molmoact_cup_storing_in-distribution
Viewer • Updated • 45.4k • 465
Collection of the MolmoAct2-BimanualYAM Dataset
-
MolmoAct2: Action Reasoning Models for Real-world Deployment
Paper • 2605.02881 • Published • 348 -
allenai/MolmoAct2-BimanualYAM-Dataset
Viewer • Updated • 76M • 7.69k • 1 -
Lerobot Visualizer V3
🏢4Visualize LeRobot datasets with interactive charts
-
Dataset Stats
📊1Fetch and view stats for MolmoAct2 datasets
Collection of the fine-tuned models for MolmoAct2
Artifacts for Branch-Adapt-Route
This is the collection of all datasets in MolmoWebMix.
Data used in the MolmoPoint models
Models collection for MolmoBot release
-
allenai/Dolci-Think-SFT-Olmo-Hybrid
Viewer • Updated • 2.93M • 975 • 13 -
allenai/Dolci-Think-SFT-Olmo-Hybrid-Tool-Use-SA
Viewer • Updated • 1.6k • 92 • 13 -
allenai/Olmo-Hybrid-Think-SFT-7B
Text Generation • 7B • Updated • 2.07k • 19 -
allenai/Olmo-Hybrid-Instruct-SFT-7B
Text Generation • 7B • Updated • 2.8k • 17
Artifacts for the Molmo2 release
Artifacts for the Molmo2 data release
Smart Any-Horizon Agent for Long Video Reasoning
-
allenai/SAGE-MM-Qwen3-VL-8B-SFT_RL
Video-Text-to-Text • 9B • Updated • 348 • 5 -
allenai/SAGE-MM-Molmo2-8B-SFT_RL
Video-Text-to-Text • 9B • Updated • 7 • 5 -
allenai/SAGE-MM-Qwen3-VL-4B-SFT_RL
Video-Text-to-Text • 5B • Updated • 33 • 6 -
allenai/SAGE-MM-Qwen2.5-VL-7B-SFT_RL
Video-Text-to-Text • 8B • Updated • 35 • 2
All artifacts for post-training Olmo 3. Datasets follow the model that resulted from training on them.
All models for the MolmoAct (Multimodal Open Language Model for Action) release.
Datasets for IFBench benchmark and paper!
Artifacts for the OLMo 2 release.
A suite of models, data, and evals over 25 corpora, 14 sizes, and 3 seeds to measure how accurately small experiments predict rankings at large scale.
A set of vision-language datasets built by Ai2 and used to train the Molmo family of models. Read more at https://molmo.allenai.org/blog
All datasets released with Tulu 3 -- state of the art open post-training recipes.
-
allenai/tulu-3-sft-mixture
Viewer • Updated • 939k • 20.7k • 251 -
allenai/llama-3.1-tulu-3-8b-preference-mixture
Viewer • Updated • 273k • 5.28k • 26 -
allenai/llama-3.1-tulu-3-70b-preference-mixture
Viewer • Updated • 337k • 100 • 19 -
allenai/llama-3.1-tulu-3-405b-preference-mixture
Viewer • Updated • 361k • 88 • 6
Artifacts for open mixture-of-experts language models.
A suite of models trained using DPO and PPO across a wide variety (up to 14) of preference datasets. See https://arxiv.org/abs/2406.09279 for more!
Dataset and baseline models for Paloma, a benchmark of language model fit to 546 textual domains
-
AI2 WildBench Leaderboard (V2)
🦁232Display LLM performance leaderboards with customizable views
-
allenai/WildBench
Viewer • Updated • 2.3k • 2.71k • 39 -
allenai/WildBench-V2-Model-Outputs
Viewer • Updated • 62.5k • 1.35k • 2 -
WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wild
Paper • 2406.04770 • Published • 28
Safety data, moderation tools and safe LLMs.
These model's tokenizer did not use HF's fast tokenizer, resulting in variations in how pre-tokenization was applied. Resolved in latest versions.
-
allenai/OLMo-2-1124-13B-Instruct-preview
Text Generation • 14B • Updated • 10 • 58 -
allenai/OLMo-2-1124-7B-Instruct-preview
Text Generation • 7B • Updated • 15 • 47 -
allenai/OLMo-2-1124-7B-SFT-Preview
Text Generation • Updated • 16 • 3 -
allenai/OLMo-2-1124-7B-DPO-Preview
Text Generation • Updated • 11 • 2
Collection of the embodied reasoning datasets for MolmoAct2
Collection of robotics datasets for MolmoAct2
-
MolmoAct2: Action Reasoning Models for Real-world Deployment
Paper • 2605.02881 • Published • 348 -
allenai/MolmoAct2-BimanualYAM-Dataset
Viewer • Updated • 76M • 7.69k • 1 -
allenai/MolmoAct2-SO100_101-Dataset
Viewer • Updated • 8.42k • 880 • 6 -
allenai/MolmoAct2-DROID-Dataset
Viewer • Updated • 17.8M • 5.74k • 4
Collection of the base models for MolmoAct2
Collection of models from the paper "Cracks in the Foundation: Seemingly Minor Architectural Choices Impact Long Context Extension".
This is the collection of WildDet3D artifacts, including demos, model checkpoints and data. https://github.com/allenai/WildDet3D
This is the collection of MolmoWeb artifacts, including model checkpoints and data.
MolmoPoint models
Training and assets data for MolmoBot release
Ai2 Open Coding Agents - Django, Sphinx, Sympy Data
The latest members of the Olmo 3 family: another 3 weeks of RL for 32B Think, the 32B Instruct model, large post-training research datasets...
Artifacts for the Bolmo release: https://allenai.org/papers/bolmo.
Artifacts for the Olmo 3 release.
All artifacts related to Olmo 3 pre-training
OlmoEarth pre-trained and fine-tuned foundation models for remote sensing
All datasets for the MolmoAct (Multimodal Open Language Model for Action) release.
Datasets, spaces, and models for Reward Bench 2 benchmark and paper!
olmOCR is a document recognition pipeline for efficiently converting documents into plain text.
olmocr.allenai.org
Improved OLMoE for iOS app. Read more: https://allenai.org/blog/olmoe-app
All models released with Tulu 3 -- state of the art open post-training recipes.
Artifacts for open multimodal language models.
Artifacts for the first set of OLMo models.
Datasets, spaces, and models for the reward model benchmark!
The set of models associated with the paper "Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2"
Data and models to enhance instruction-following for scientific literature understanding.
ZebraLogic Bench: Testing the Limits of LLMs in Logical Reasoning
Collection of the evaluation rollouts for MolmoAct2 conducted by Cortex AI
-
MolmoAct2: Action Reasoning Models for Real-world Deployment
Paper • 2605.02881 • Published • 348 -
allenai/eval_molmoact_candy_sorting_in-distribution
Viewer • Updated • 59.6k • 457 -
allenai/eval_molmoact_cup_stacking_in-distribution
Viewer • Updated • 32k • 428 -
allenai/eval_molmoact_cup_storing_in-distribution
Viewer • Updated • 45.4k • 465
Collection of the embodied reasoning datasets for MolmoAct2
Collection of the MolmoAct2-BimanualYAM Dataset
-
MolmoAct2: Action Reasoning Models for Real-world Deployment
Paper • 2605.02881 • Published • 348 -
allenai/MolmoAct2-BimanualYAM-Dataset
Viewer • Updated • 76M • 7.69k • 1 -
Lerobot Visualizer V3
🏢4Visualize LeRobot datasets with interactive charts
-
Dataset Stats
📊1Fetch and view stats for MolmoAct2 datasets
Collection of robotics datasets for MolmoAct2
-
MolmoAct2: Action Reasoning Models for Real-world Deployment
Paper • 2605.02881 • Published • 348 -
allenai/MolmoAct2-BimanualYAM-Dataset
Viewer • Updated • 76M • 7.69k • 1 -
allenai/MolmoAct2-SO100_101-Dataset
Viewer • Updated • 8.42k • 880 • 6 -
allenai/MolmoAct2-DROID-Dataset
Viewer • Updated • 17.8M • 5.74k • 4
Collection of the fine-tuned models for MolmoAct2
Collection of the base models for MolmoAct2
Collection of models from the paper "Cracks in the Foundation: Seemingly Minor Architectural Choices Impact Long Context Extension".
Artifacts for Branch-Adapt-Route
This is the collection of WildDet3D artifacts, including demos, model checkpoints and data. https://github.com/allenai/WildDet3D
This is the collection of all datasets in MolmoWebMix.
This is the collection of MolmoWeb artifacts, including model checkpoints and data.
Data used in the MolmoPoint models
MolmoPoint models
Models collection for MolmoBot release
Training and assets data for MolmoBot release
-
allenai/Dolci-Think-SFT-Olmo-Hybrid
Viewer • Updated • 2.93M • 975 • 13 -
allenai/Dolci-Think-SFT-Olmo-Hybrid-Tool-Use-SA
Viewer • Updated • 1.6k • 92 • 13 -
allenai/Olmo-Hybrid-Think-SFT-7B
Text Generation • 7B • Updated • 2.07k • 19 -
allenai/Olmo-Hybrid-Instruct-SFT-7B
Text Generation • 7B • Updated • 2.8k • 17
Ai2 Open Coding Agents - Django, Sphinx, Sympy Data
The latest members of the Olmo 3 family: another 3 weeks of RL for 32B Think, the 32B Instruct model, large post-training research datasets...
Artifacts for the Molmo2 release
Artifacts for the Bolmo release: https://allenai.org/papers/bolmo.
Artifacts for the Molmo2 data release
Artifacts for the Olmo 3 release.
Smart Any-Horizon Agent for Long Video Reasoning
-
allenai/SAGE-MM-Qwen3-VL-8B-SFT_RL
Video-Text-to-Text • 9B • Updated • 348 • 5 -
allenai/SAGE-MM-Molmo2-8B-SFT_RL
Video-Text-to-Text • 9B • Updated • 7 • 5 -
allenai/SAGE-MM-Qwen3-VL-4B-SFT_RL
Video-Text-to-Text • 5B • Updated • 33 • 6 -
allenai/SAGE-MM-Qwen2.5-VL-7B-SFT_RL
Video-Text-to-Text • 8B • Updated • 35 • 2
All artifacts related to Olmo 3 pre-training
All artifacts for post-training Olmo 3. Datasets follow the model that resulted from training on them.
OlmoEarth pre-trained and fine-tuned foundation models for remote sensing
All models for the MolmoAct (Multimodal Open Language Model for Action) release.
All datasets for the MolmoAct (Multimodal Open Language Model for Action) release.
Datasets for IFBench benchmark and paper!
Datasets, spaces, and models for Reward Bench 2 benchmark and paper!
Artifacts for the OLMo 2 release.
olmOCR is a document recognition pipeline for efficiently converting documents into plain text.
olmocr.allenai.org
A suite of models, data, and evals over 25 corpora, 14 sizes, and 3 seeds to measure how accurately small experiments predict rankings at large scale.
Improved OLMoE for iOS app. Read more: https://allenai.org/blog/olmoe-app
A set of vision-language datasets built by Ai2 and used to train the Molmo family of models. Read more at https://molmo.allenai.org/blog
All models released with Tulu 3 -- state of the art open post-training recipes.
All datasets released with Tulu 3 -- state of the art open post-training recipes.
-
allenai/tulu-3-sft-mixture
Viewer • Updated • 939k • 20.7k • 251 -
allenai/llama-3.1-tulu-3-8b-preference-mixture
Viewer • Updated • 273k • 5.28k • 26 -
allenai/llama-3.1-tulu-3-70b-preference-mixture
Viewer • Updated • 337k • 100 • 19 -
allenai/llama-3.1-tulu-3-405b-preference-mixture
Viewer • Updated • 361k • 88 • 6
Artifacts for open multimodal language models.
Artifacts for open mixture-of-experts language models.
Artifacts for the first set of OLMo models.
A suite of models trained using DPO and PPO across a wide variety (up to 14) of preference datasets. See https://arxiv.org/abs/2406.09279 for more!
Datasets, spaces, and models for the reward model benchmark!
Dataset and baseline models for Paloma, a benchmark of language model fit to 546 textual domains
The set of models associated with the paper "Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2"
-
AI2 WildBench Leaderboard (V2)
🦁232Display LLM performance leaderboards with customizable views
-
allenai/WildBench
Viewer • Updated • 2.3k • 2.71k • 39 -
allenai/WildBench-V2-Model-Outputs
Viewer • Updated • 62.5k • 1.35k • 2 -
WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wild
Paper • 2406.04770 • Published • 28
Data and models to enhance instruction-following for scientific literature understanding.
Safety data, moderation tools and safe LLMs.
ZebraLogic Bench: Testing the Limits of LLMs in Logical Reasoning
These model's tokenizer did not use HF's fast tokenizer, resulting in variations in how pre-tokenization was applied. Resolved in latest versions.
-
allenai/OLMo-2-1124-13B-Instruct-preview
Text Generation • 14B • Updated • 10 • 58 -
allenai/OLMo-2-1124-7B-Instruct-preview
Text Generation • 7B • Updated • 15 • 47 -
allenai/OLMo-2-1124-7B-SFT-Preview
Text Generation • Updated • 16 • 3 -
allenai/OLMo-2-1124-7B-DPO-Preview
Text Generation • Updated • 11 • 2