At the moment it's a an exact-match, later will support partial match as well.
Stas Bekman
stas
AI & ML interests
Toolmaker. Software creator, optimizer and harmonizer.
Makes things work and fly at Snowflake AI Research
Training LLM/RAG/Generative AI/Machine Learning/Scalability
Recent Activity
repliedto their post about 17 hours ago
After many months of intense work the
Snowflake AI Research team is happy to present to you the new open source project: Arctic RL
https://snowflake.com/en/blog/engineering/arctic-rl-open-source-backend/
- Arctic RL integrates with VeRL and SkyRL today; enable ZoRRo with one config flag, no code changes required
- ZoRRo delivers up to 6x actor-update acceleration and a 3.5x end-to-end training speedup, reducing Arctic-Text2SQL-R2 training from ~5 days to ~36 hours on 32 H200 GPUs
- Arctic-Text2SQL-R2 achieved higher accuracy scores (48.7) than Gemini 3.1 Pro (47.9) and Claude 4.7 (47.3) on Snowflake's evaluated enterprise SQL benchmark under the tested conditions
- Two open source recipes ship with this release: a text-to-SQL recipe that improved BIRD dev accuracy from 59.92% to 70.35%, and a multi-hop QA recipe that improved average accuracy from 69.6% to 72.3% updated a model about 17 hours ago
stas/the-art-of-debugging-book published a model about 17 hours ago
stas/the-art-of-debugging-bookOrganizations
replied to their post about 17 hours ago
updated a
model about 17 hours ago
published a
model about 17 hours ago
replied to their post about 17 hours ago
Thank you for the kind words, Dipankar
The lion share of speed up comes from prompt deduplication during generation and training.
updated a
model about 18 hours ago
posted an update 1 day ago
Post
103
In parallel we announce a new open source repo:
https://github.com/Snowflake-AI-Research/Arctic-Platform
This is the framework for very fast RL (and future other optimizations rolled into it)
It currently has all the code you need to use or integrate Arctic RL into RL frameworks, with SkyRL and Verl available and more framework integrations coming.
Please kindly spread the word! Thank you!
https://github.com/Snowflake-AI-Research/Arctic-Platform
This is the framework for very fast RL (and future other optimizations rolled into it)
It currently has all the code you need to use or integrate Arctic RL into RL frameworks, with SkyRL and Verl available and more framework integrations coming.
Please kindly spread the word! Thank you!
posted an update 1 day ago
Post
1629
After many months of intense work the
Snowflake AI Research team is happy to present to you the new open source project: Arctic RL
https://snowflake.com/en/blog/engineering/arctic-rl-open-source-backend/
- Arctic RL integrates with VeRL and SkyRL today; enable ZoRRo with one config flag, no code changes required
- ZoRRo delivers up to 6x actor-update acceleration and a 3.5x end-to-end training speedup, reducing Arctic-Text2SQL-R2 training from ~5 days to ~36 hours on 32 H200 GPUs
- Arctic-Text2SQL-R2 achieved higher accuracy scores (48.7) than Gemini 3.1 Pro (47.9) and Claude 4.7 (47.3) on Snowflake's evaluated enterprise SQL benchmark under the tested conditions
- Two open source recipes ship with this release: a text-to-SQL recipe that improved BIRD dev accuracy from 59.92% to 70.35%, and a multi-hop QA recipe that improved average accuracy from 69.6% to 72.3%
Snowflake AI Research team is happy to present to you the new open source project: Arctic RL
https://snowflake.com/en/blog/engineering/arctic-rl-open-source-backend/
- Arctic RL integrates with VeRL and SkyRL today; enable ZoRRo with one config flag, no code changes required
- ZoRRo delivers up to 6x actor-update acceleration and a 3.5x end-to-end training speedup, reducing Arctic-Text2SQL-R2 training from ~5 days to ~36 hours on 32 H200 GPUs
- Arctic-Text2SQL-R2 achieved higher accuracy scores (48.7) than Gemini 3.1 Pro (47.9) and Claude 4.7 (47.3) on Snowflake's evaluated enterprise SQL benchmark under the tested conditions
- Two open source recipes ship with this release: a text-to-SQL recipe that improved BIRD dev accuracy from 59.92% to 70.35%, and a multi-hop QA recipe that improved average accuracy from 69.6% to 72.3%
- 4 replies
posted an update 15 days ago
Post
136
PSA for DeepSpeed users - a long outstanding precision-related critical bug has been identified and fixed in https://github.com/deepspeedai/DeepSpeed/pull/8066 and a new release has been made.
The issue was about mixed precision mode downcasting buffers that had to be in fp32 - massively impacting correctness due to large static buffers - e.g. RoPE in Qwen3 models when using long sequence lengths 32K+.
Hopefully this fix brings Deepspeed to a close parity with FSDP2 which has been an issue since a long time.
You can still have the old behavior but you'd now need to manually configure it - by default the model's buffers will now remain in the original precision.
Please install deepspeed==0.19.2 which will do the right thing.
Thanks to Tunji Ruwase and Claude Opus 4.8 via Cursor for identifying and fixing the problem.
The issue was about mixed precision mode downcasting buffers that had to be in fp32 - massively impacting correctness due to large static buffers - e.g. RoPE in Qwen3 models when using long sequence lengths 32K+.
Hopefully this fix brings Deepspeed to a close parity with FSDP2 which has been an issue since a long time.
You can still have the old behavior but you'd now need to manually configure it - by default the model's buffers will now remain in the original precision.
Please install deepspeed==0.19.2 which will do the right thing.
Thanks to Tunji Ruwase and Claude Opus 4.8 via Cursor for identifying and fixing the problem.
posted an update 4 months ago
Post
255
Good news! Ulysses Sequence Parallelism from the Snowflake AI Research and the Deepspeed teams has been integrated into
HuggingFace Trainer, Accelerate and TRL
For extensive details please see this writeup:
https://huggingface.co/blog/ulysses-sp
Thanks a lot to Kashif Rasul for helping make it happen. Also the others in the HF team who helped with integration.
HuggingFace Trainer, Accelerate and TRL
For extensive details please see this writeup:
https://huggingface.co/blog/ulysses-sp
Thanks a lot to Kashif Rasul for helping make it happen. Also the others in the HF team who helped with integration.
published an article 4 months ago
Article
Ulysses Sequence Parallelism: Training with Million-Token Contexts
kashif, stas
• • 30 authored a
paper 5 months ago
updated a
dataset 8 months ago
published a
dataset 8 months ago
liked a
Space 9 months ago
Update generation_config.json
1
#4 opened 9 months ago
by stas
posted an update over 1 year ago
Post
2390
Do you want ArcticTraining at @SnowflakeDB to add an ability to post-train DeepSeek V3/R1 models with DPO using just a few GPU nodes?
Please vote here and tell others about it: https://github.com/snowflakedb/ArcticTraining/discussions/58
ArcticTraining is an open-source, easy to use post-training framework for NVIDIA GPUs built on top of DeepSpeed.
Please vote here and tell others about it: https://github.com/snowflakedb/ArcticTraining/discussions/58
ArcticTraining is an open-source, easy to use post-training framework for NVIDIA GPUs built on top of DeepSpeed.
posted an update over 1 year ago
Post
1269
If you remember my work on MAMF - to find the realistic TFLOPS achievable ceiling - the Intel AI team has shared their measurements and they scored ...
an incredible 99.4% TFLOPS efficiency for Gaudi 2!
That's quite amazing! Your ROI on these accelerators will be very high.
The full table is here: https://github.com/stas00/ml-engineering/tree/master/compute/accelerator#maximum-achievable-matmul-flops-comparison-table
As we have seen the competitors get their achievable efficiency worse with each new generation, I'm looking forward to see if Gaudi 3 will keep the high bar!
Thanks to Avi Rubin, Lakshman Chari, Imtiaz Sajwani, Ramy J and Zhiqi Tao for helping to get these numbers to the community.
an incredible 99.4% TFLOPS efficiency for Gaudi 2!
That's quite amazing! Your ROI on these accelerators will be very high.
The full table is here: https://github.com/stas00/ml-engineering/tree/master/compute/accelerator#maximum-achievable-matmul-flops-comparison-table
As we have seen the competitors get their achievable efficiency worse with each new generation, I'm looking forward to see if Gaudi 3 will keep the high bar!
Thanks to Avi Rubin, Lakshman Chari, Imtiaz Sajwani, Ramy J and Zhiqi Tao for helping to get these numbers to the community.
upvoted an article almost 2 years ago
Article
Mixture of Experts Explained
- +4
osanseviero, lewtun, philschmid, smangrul, ybelkada, pcuenq
• • 1.15k Fix FileNotFoundError
❤️👍 2
3
#2 opened almost 2 years ago
by 
lhoestq
updated a
model almost 2 years ago
