BenchMAXXXED
BenchMAXXXED for investors.
I dont think so. I did some of my own benchmarking on this. I don+t know what would happen if test was harder on Agentic and tools, the difference might be bigger. It for sure has some downsides but maybe its a good thing all models are not mediokre on all areas. Running Laguna together with another model might be great. For my particular hardware its crazy fast. 180+t/s in concurrent 1 on TG and 6000t/s in PP.
Scores per Benchmark
Bold = best in row.
| Benchmark | Qwen3.5-122B | Laguna | Solar-250B | Qwen3.6-27B | MiniMax-M2.7 |
|---|---|---|---|---|---|
| Internal Tool Use | 98.6 | 100.0 | 100.0 | 96.9 | 96.6 |
| OpenClaw Agentic | 87.7 | 93.0 | 81.6 | 85.1 | 85.1 |
| MATH Hard | 83.0 | 57.0 | 80.0 | 78.0 | 78.0 |
| MuSR | 91.0 | 76.0 | 72.0 | 88.0 | 88.0 |
| GPQA Diamond | 57.0 | 59.0 | 62.0 | 64.0 | 58.0 |
| MMLU-Pro | 82.0 | 68.7 | 79.0 | 82.3 | 60.3 |
| HumanEval | 94.0 | 93.0 | 91.0 | 81.0 | 81.0 |
| BFCL | 93.0 | 90.0 | 94.0 | 84.0 | 92.0 |
| HellaSwag | 96.7 | 90.0 | 94.0 | 96.7 | 82.7 |
| GSM8K | 96.0 | 88.0 | 91.0 | 92.0 | 93.0 |
@Danne980 great benchmark, but no any SWE (coding skills)? Qwen3.6-27B is very strong in coding and math but with tool calling issues, it seems the Laguna is slightly better in coding than Qwen3.6-27B.
I dont think so. I did some of my own benchmarking on this. I don+t know what would happen if test was harder on Agentic and tools, the difference might be bigger. It for sure has some downsides but maybe its a good thing all models are not mediokre on all areas. Running Laguna together with another model might be great. For my particular hardware its crazy fast. 180+t/s in concurrent 1 on TG and 6000t/s in PP.
Scores per Benchmark
Bold = best in row.
Benchmark Qwen3.5-122B Laguna Solar-250B Qwen3.6-27B MiniMax-M2.7 Internal Tool Use 98.6 100.0 100.0 96.9 96.6 OpenClaw Agentic 87.7 93.0 81.6 85.1 85.1 MATH Hard 83.0 57.0 80.0 78.0 78.0 MuSR 91.0 76.0 72.0 88.0 88.0 GPQA Diamond 57.0 59.0 62.0 64.0 58.0 MMLU-Pro 82.0 68.7 79.0 82.3 60.3 HumanEval 94.0 93.0 91.0 81.0 81.0 BFCL 93.0 90.0 94.0 84.0 92.0 HellaSwag 96.7 90.0 94.0 96.7 82.7 GSM8K 96.0 88.0 91.0 92.0 93.0
Which sampler settings? This is a big point of contention currently.
Sample sizes vary per benchmark (1,079 total): Internal Tool Use 10, OpenClaw Agentic 19, MATH/MuSR/GPQA/HumanEval/BFCL/GSM8K 100 each, HellaSwag 150, MMLU-Pro 300.
M2.7 was 4 bit. Others 8.
I think the model are sensetive on quant and setup. Reults sems to vary.
I don't give a fuck about those fucking benchmarks. I tested it with some prompts on my own app, and it's definitely worse than Qwen (Thinking Cap fine tune) 3.6 27b and Qwen (base model) 3.6 35b a3b. My tests mostly involve Blender MCP, Unity MCP, and HTML game code. The model overthinks too much and has almost zero sense of design.
I don't give a fuck about those fucking benchmarks. I tested it with some prompts on my own app, and it's definitely worse than Qwen (Thinking Cap fine tune) 3.6 27b and Qwen (base model) 3.6 35b a3b. My tests mostly involve Blender MCP, Unity MCP, and HTML game code. The model overthinks too much and has almost zero sense of design.
And i guess you applied the fixes and ran 8 bits?
I don't give a fuck about those fucking benchmarks. I tested it with some prompts on my own app, and it's definitely worse than Qwen (Thinking Cap fine tune) 3.6 27b and Qwen (base model) 3.6 35b a3b. My tests mostly involve Blender MCP, Unity MCP, and HTML game code. The model overthinks too much and has almost zero sense of design.
And i guess you applied the fixes and ran 8 bits?
I use open router for this kind of test.
You dont need defend this much. Here some of test on youtube community. Everyone knows this model worse than qwen 3.6 27b (iq4 xs) run on my 16gb vram machine.
https://www.youtube.com/watch?v=SoRla9IxttE
https://www.youtube.com/watch?v=t6uhTLEOpzU&t=1800s
Defend? I was just asking. I see very mixed reviews. And i know they have done a lot of changes. My first results was not good. One thing I have noticed is that the models does not seem to quant well, at least not at 4 bits.
Without doing any real testing (aka real world scenario), people try a couple of prompts and claim a 120B parameter model is the best—without even checking if it actually performs worse than a 27B or 35b a3b model. Some of the most downloaded models include nonsensical fine-tunes like fable vibe. We need to cut through the hype and focus on what AI can actually do for us and which model is genuinely better. Everyone just jumps on the hype train, looks at benchmark scores without proper testing, and calls it a day.
Edit: I forget to say this model even can't talk my native language's.
I don't give a fuck about those fucking benchmarks. I tested it with some prompts on my own app, and it's definitely worse than Qwen (Thinking Cap fine tune) 3.6 27b and Qwen (base model) 3.6 35b a3b. My tests mostly involve Blender MCP, Unity MCP, and HTML game code. The model overthinks too much and has almost zero sense of design.
And i guess you applied the fixes and ran 8 bits?
There is no fixes, it behaves the same on API as it does from the repo. The model is hot garbage in the regular ecosystem. Its good in a vacuum in their custom harness that somehow coaxed it to perform an eval.
It does really depend on the task. I did some light testing for design (html/react) and it did underperform a lot. However for my main work needs, namely ML engineering and framework dev (mostly ASR, some SLM tests on the side), it is performing quite a bit better than both ds v4 flash and 3.6 27B. If you need frontend/design, i would look towards other models, but for tricky math/algorithms and research, this model does perform quite well (im running the nvfp4 on a dgx spark with repetition penalty at 1.15, which has essentially fixed the looping issues i've had).
Models are trained for different tasks on different data, I understand the frustration for your target domain, however it is genuinely a good model if it fits your needs. I still run both 27B on my 3090 and laguna on the spark, both have their usecases. I do agree it really doesn't feel like a "general use/jack of all trades" model. I haven't done extensive testing outside the stuff i work with and the small design test, but it was enough to convince me that it is indeed a 'specialized' model.
It does really depend on the task. I did some light testing for design (html/react) and it did underperform a lot. However for my main work needs, namely ML engineering and framework dev (mostly ASR, some SLM tests on the side), it is performing quite a bit better than both ds v4 flash and 3.6 27B. If you need frontend/design, i would look towards other models, but for tricky math/algorithms and research, this model does perform quite well (im running the nvfp4 on a dgx spark with repetition penalty at 1.15, which has essentially fixed the looping issues i've had).
Models are trained for different tasks on different data, I understand the frustration for your target domain, however it is genuinely a good model if it fits your needs. I still run both 27B on my 3090 and laguna on the spark, both have their usecases. I do agree it really doesn't feel like a "general use/jack of all trades" model. I haven't done extensive testing outside the stuff i work with and the small design test, but it was enough to convince me that it is indeed a 'specialized' model.
The model can't speak my two main languages, Turkish and Kurdish. It can't write HTML, JS, or CSS, which are relatively easy languages. It can't code a simple game by connecting to Unity MCP. It can't create a simple model using Blender MCP. The biggest problem is that it keeps getting stuck in a thinking loop. And Qwen 3.8 27b is incoming. Qwen 3.8 IQ4_XSS with no mmproj gonna fit my 16gb vram machine.
Check video from Luke's Dev Lab if you dont believe my test. I tested 14 days ago and 5 days after this man tested the model.
https://www.youtube.com/watch?v=6XwE5NjYPfk
I never said I don't believe your test, and I do follow Luke, but again, it solely depends on what the model is trained for, and for better or worse, this model (from your findings) seems to mostly (or only?) have been trained using english data, and on non-web dev languages. You are correct that the core web stack is "easy" (to a degree), but if it didn't have a lot of raw html in the training data, it is absolutely normal for it to not be good at writing html/js/css. But not having one thing (web stack/languages and maybe general historic data etc) in the training data means it was able to internalize other stuff better, like C++ and low-level algorithms, which is what I have found it excel at.
It is not a "general use" model, like how gemma, qwen, deepseek etc. are, as there are obvious domains where it is bad, and they do state in the model card it "is designed for agentic coding and long-horizon work.".
It does not have vision, which leads me to think that any 3d work (like unity and blender) will also underperform at, as you found, as it never had to learn "visual feedback", only text feedback.
However, it is the smallest model that I have found to not lose itself in a big low-level codebase and actually be able to program in and debug issues properly. My setup hasn't changed in the last week, I still run both 27B and Laguna locally on separate machines, with Laguna being the main model to explore and debug, and 27B be a subagent to implement specific stuff (since its faster on my setup and the code is usually good enough), or just "give its opinion" whenever work is delicate. But I keep hitting the limits of 27B in terms of knowledge, as much as I love that model, which is simply a size thing.
I never said I don't believe your test, and I do follow Luke, but again, it solely depends on what the model is trained for, and for better or worse, this model (from your findings) seems to mostly (or only?) have been trained using english data, and on non-web dev languages. You are correct that the core web stack is "easy" (to a degree), but if it didn't have a lot of raw html in the training data, it is absolutely normal for it to not be good at writing html/js/css. But not having one thing (web stack/languages and maybe general historic data etc) in the training data means it was able to internalize other stuff better, like C++ and low-level algorithms, which is what I have found it excel at.
It is not a "general use" model, like how gemma, qwen, deepseek etc. are, as there are obvious domains where it is bad, and they do state in the model card it "is designed for agentic coding and long-horizon work.".
It does not have vision, which leads me to think that any 3d work (like unity and blender) will also underperform at, as you found, as it never had to learn "visual feedback", only text feedback.However, it is the smallest model that I have found to not lose itself in a big low-level codebase and actually be able to program in and debug issues properly. My setup hasn't changed in the last week, I still run both 27B and Laguna locally on separate machines, with Laguna being the main model to explore and debug, and 27B be a subagent to implement specific stuff (since its faster on my setup and the code is usually good enough), or just "give its opinion" whenever work is delicate. But I keep hitting the limits of 27B in terms of knowledge, as much as I love that model, which is simply a size thing.
I think you have valid points, but the actual logic of MoE models is to work faster while containing more information. For example, a model with 27b parameters must fit its entire knowledge base into those 27b parameters. It works slower but is smarter. The logic behind MoE models, on the other hand, is that by activating 8b parameters out of a 118b parameter knowledge base, it has broader knowledge from a wider perspective, even if it's not smarter. What I don't understand is why a model with 118b parameters would have no general knowledge at all. I believe this model knowledge base that could be achieved with fewer parameters.
Footnote: I'm not an AI engineer.
I am curious, what hardware do you run when you have better performance with Qwen36 27B? I have the opposite and are just curious. I get close to 200t/s on Laguna and 120 on 27B (TG).
I am curious, what hardware do you run when you have better performance with Qwen36 27B? I have the opposite and are just curious. I get close to 200t/s on Laguna and 120 on 27B (TG).
Performance isn't about speed; it's about using fewer tokens, cheaper hardware, and writing better code.
Thinking cap is 50% faster with same quality. Grug is way way faster but quality is lower. Laguna in my test faster on openrouter but dumber than most models and it is not cheapest for it size.
Sorry, I was asking Sleepy because of this statement: "My setup hasn't changed in the last week, I still run both 27B and Laguna locally on separate machines, with Laguna being the main model to explore and debug, and 27B be a subagent to implement specific stuff (since its faster on my setup and the code is usually good enough)".
I dont at all agrre with speed is not performance. There are many parameters, but speed is of course one. And probably the only one that is pretty easy to measure. For me, speed is very important, using a model like Laguna att 200t/s TG and 6000t/s PP makes it much easier for me to focus. Other models (even frontier) usually are pretty slow and I start doing other stuff at the same time. At 200t/s I feel like I am almost not able to give instructions fast enough.
Also I hear that you are dissapointed with the model, maybe thats why I enjoy it. Many models are trained on alot of knowledge that is useless for me, I dont care about it beeing able to do chinese, Turkish or Kurdish. It does Swedish decent. That is probably useless for you. For me frontend development is much less useful then the agentic capabilitys. I don´t primarely develope with opensource models but rather automate. Models in this size need to prioritize. This models suits me better than it suits you. I dint think its perfect but for me its one of the best out there. Maybe if i Start using it for other task I will like it less.
@TawusGames On the MoE thing, if it helps with the intuition, take something big like the 2.8T Qwen Max or kimi k3, or the 760B GLM (i don't recall the exact number, its not too relevant for this). say you have 10 trillion training tokens of literature, 10 trillion tokens of medical texts, 10 trillion tokens of coding examples etc. (if you are curious about exact counts, some labs disclose how many tokens they've used to train a model, the 'minimum' is considered 20 tokens of training data per param, so a 1B model needs at a minimum 20B tokens, but usually everyone trains on as much as they can within a deadline).
Giving more training tokens to a model tends to help, but there are clear diminishing returns, as you can only fit so much knowledge (using current architectures) in parameters, no matter how much tokens it sees. If I recall correctly, deepseek v4 flash and pro are trained on the same amount of training data (which is a good example of under-saturating, at least imo, given that, while pro tended to be better, it wasn't by a big margin, while being 1.6T params vs 290B for flash), and the XS laguna and S are also trained on the same training set, but the difference in what they are capable of is massive.
My bet, as an outsider to poolside, is that they wanted to bet on instead of training on 10T tokens of lit, 10T of medical etc, they have swayed the training data (and RL) heavily towards non-graphical coding (as without vision, while possible its a little awkward to try and teach a model to understand 2d/3d), with the other domains (including, I guess, language, which probably would fall in the 'literature' domain) being represented to the minimal degree they've decided is needed to help with logic/reasoning.
So put simply, the reason why a 118B model can exist without having a lot of general knowledge, is because general knowledge was just a small part of the training data, and even if it memorized all of it, if its 1/10th of the general knowledge used in other model's training, it will lack there. Whether that was a good or bad idea, I think is subjective to each user.
On my setup @Danne980 , I am currently running Qwen3.6 27B (llama.cpp IQ4_NL, 256k ctx @ q4 , mtp=3 @q4 , no vision) on a 3090 (24gb ram, ~970 GB/s bandwidth), and I run Laguna S 2.1 NVFP4 on a DGX Spark (128gb vram, ~270 GB/s bandwidth). The massive delta in bandwidth is the reason for the odd speed numbers. From playing around with both, without drafters the 27B on 3090 starts off around 40 t/s and is around 25 t/s beyond 128k ctx (with mtp=3 starts around 70 and is around 50 beyond 128k), laguna on the spark on short contexts it was around 15-18 iirc without a drafter, and with a drafter starts at ~25 t/s, drops to as low as 12-13 during thinking, however jumps to ~40 t/s during coding (the drafter is very much code optimized, but given that this model spends 80% of its time thinking, im happy poolside are looking into optimizing those predictions too currently). On multiple concurrent requests on the spark, the aggregate t/s while they all think is ~35 t/s in my xp and i've seen it burst up to 80 t/s (aggregate) when multiple of them are coding. I don't have prefill numbers, but its pretty fast for this model on the spark in vllm, qwen with llama.cpp on the 3090 feels slower (I assume due to params and not compute, as iirc 3090 has more compute than the spark).
Naturally, take everything I say with a grain of salt, my professional xp is mostly in speech recognition ML (both framework dev and modeling), I've been tinkering with LLMs (both using and training/finetuning) mostly in the last 5-6 months in my spare time, as I find them fascinating, how changing up what you train for and your goals can make model you can run on your phone be better than "frontier" models for a task you need and stuff like that. I think optimizing for 'general' knowledge is good and important, but am currently more interested in models that are optimized for reasoning in any context (what I feel like 27B is, and why its good in many environments even if it doesn't "know" more than other models) instead of "knowing" a lot of information (as i consider that a little DoA, since knowledge gets outdated and the model will be useless in a year), and models that are specialized for a certain use case (like this model, or VibeThinker, or to a degree gemma models).
Thinking cap is 50% faster with same quality.
I would be a little careful with thinking cap, it was on my 3090 for about 2 weeks and seemed good, but I noticed it not being able to progress on some tasks, went back to the base 27B and it solved them without a problem. I then ran my ML coding tasks bench and it did perform worse than the base 27B. Its a fascinating model and I love the idea of it, but I would not fully delete the base one, as it may be worth booting if thinkingcap fails to solve something. In my xp it's at least 95% as good as the base one, and it being faster (by thinking less) does make it appealing, I would just suggest that, if you have the disk space for it, to also keep a backup base one.
Thinking cap is 50% faster with same quality.
I would be a little careful with thinking cap, it was on my 3090 for about 2 weeks and seemed good, but I noticed it not being able to progress on some tasks, went back to the base 27B and it solved them without a problem. I then ran my ML coding tasks bench and it did perform worse than the base 27B. Its a fascinating model and I love the idea of it, but I would not fully delete the base one, as it may be worth booting if thinkingcap fails to solve something. In my xp it's at least 95% as good as the base one, and it being faster (by thinking less) does make it appealing, I would just suggest that, if you have the disk space for it, to also keep a backup base one.
No matter what new model comes out, I'm not taking the base model out of my toolbox.
Some of the big models offer a free tier, so I take the text I get from those and feed it into Qwen 3.6 27B's 'thinking cap' to browse files and make changes. Qwen 3.6 35 A3B sometimes struggles to find the function that needs to be deleted when it's described in text without the actual code being provided.
@TawusGames @sleepyeldrazi You can think of MoEs as either bad for their total parameter count or absurdly good for their active parameter count. Which perspective people take often depends on their hardware and how educated they are on how LLMs work. People a limited amount of high memory bandwidth memory (like a premium consumer grade GPU) and limited familiarity with MoEs (like your average "home gamer") tend to dismiss MoEs as inefficient or underperforming for their (total) size. Because when measured by quality per GB of VRAM, they are.
When memory bandwidth constrained, decode speed tends to be directly proportional to active parameter count. However, the quality of a model roughly scales with the geometric mean of the active * total parameter count. So a 110B A8B parameter MoE is equivalent to about a 30B dense model. So theoretically, Qwen 27B and Laguna-S-2.1 have the potential to be of the same quality, but if you have enough vRAM or unified memory to run it without offloading to system RAM, Laguna-S-2.1 is way faster.
So, MoEs thrive is when you have a ton of mediocre memory bandwidth vRAM, or a hybrid CPU/GPU machine with a ton of system memory. In these memory rich, bandwidth limited scenarios, an MoE that can run inference at the same tok/sec as a given dense model will have much better quality than the dense model. So I view the inactive parameters as leverage: they allow my server to run a much larger model at a practical speed.
There's no longer any need for debate. Qwen 3.8 27b is out, and it's better than all mid-size LLMs to date. I don't think there's any need to talk about Laguna anymore. Qwen 3.8 27b and Laguna aren't just in different leagues; they aren't even in the same universe.
There's no longer any need for debate. Qwen 3.8 27b is out, and it's better than all mid-size LLMs to date. I don't think there's any need to talk about Laguna anymore. Qwen 3.8 27b and Laguna aren't just in different leagues; they aren't even in the same universe.
You had a great post above yours about the benefits of dense vs MoE but you still wanted a last word. You are making an argument that is wrong but right now, exactly at this time when Qwen 3.8 is just released it looks kind of logical. If there are a qwen 3.8 122B the flaws in your arguments will be obvious. Also models like Laguna is still great even if you understand its use. It you want a model primarly for agentic use and have a low bandwidth high memory system it can be amazing. It can probably challenge qwen 3.8 on that usecase but with certain systens it will be much faster. In my system with relatively fast mem speed it is still almost 100% faster than qwen 3.8 27b.
MoE models also have cheaper KV. I have not measured exact numbers for Laguna to be fair, however I did a comparison between deepseek v4 flash (284B-A14B) and 1M tokens of that is 10 gb, while ~19gb for 256k for 27B (7gb at q4). When you do the math between those two, if you host more than 6 concurrent 256k agents, dsv4 flash, even if a little over 3x the size in GB, uses less ram than 27B.
Both have many benefits and downsides. Saying one is better than the other is pointless.
There's no longer any need for debate. Qwen 3.8 27b is out, and it's better than all mid-size LLMs to date. I don't think there's any need to talk about Laguna anymore. Qwen 3.8 27b and Laguna aren't just in different leagues; they aren't even in the same universe.
You had a great post above yours about the benefits of dense vs MoE but you still wanted a last word. You are making an argument that is wrong but right now, exactly at this time when Qwen 3.8 is just released it looks kind of logical. If there are a qwen 3.8 122B the flaws in your arguments will be obvious. Also models like Laguna is still great even if you understand its use. It you want a model primarly for agentic use and have a low bandwidth high memory system it can be amazing. It can probably challenge qwen 3.8 on that usecase but with certain systens it will be much faster. In my system with relatively fast mem speed it is still almost 100% faster than qwen 3.8 27b.
I 100% agree with you Qwen 3.8 122b moe absoutly beat qwen 3.8 27b but laguna is trash.
There's no longer any need for debate. Qwen 3.8 27b is out, and it's better than all mid-size LLMs to date. I don't think there's any need to talk about Laguna anymore. Qwen 3.8 27b and Laguna aren't just in different leagues; they aren't even in the same universe.
You had a great post above yours about the benefits of dense vs MoE but you still wanted a last word. You are making an argument that is wrong but right now, exactly at this time when Qwen 3.8 is just released it looks kind of logical. If there are a qwen 3.8 122B the flaws in your arguments will be obvious. Also models like Laguna is still great even if you understand its use. It you want a model primarly for agentic use and have a low bandwidth high memory system it can be amazing. It can probably challenge qwen 3.8 on that usecase but with certain systens it will be much faster. In my system with relatively fast mem speed it is still almost 100% faster than qwen 3.8 27b.
Lol. They marketed laguna as specifically a agentic worker. In most cases it thinks itself into another universe and cant complete a single workflow in our harnesses. We have 30b MOEs that can at least attempt to complete our workflows in our harnesses. Laguna simply does not. Its really that simple. We have ran laguna from the API, and unquantized in our mulit RTX6000 environments.
So what is a VAST MAJORITY of the people saying this model is non functional not understanding? Its unanimous, its benchmaxxed and laguna released a defective model marketed as an agentic worker. Thats all there is to it. Weve given it unlimited resources, it fails at every attempt. So into the bin it went.
If you have some "understanding" how about you share a reproducible test and environment that proves your point otherwise, I dont understand what you are defending.
guys, no debating anymore, Qwen3.8-27B was out for days, It matches the performance of GPT-5.6 Luna (max), a small model in the room of large models.
I used laguna, it's a bit better than Qwen3.6-27B in programming, please keep evolving.
@Weisunding I suspect that GPT-5.6 luna and Qwen3.8-27B are of a similar parameter count. This just means that Qwen is catching up with the closed weight/ closed source labs, which is great. The initial success of OpenAI and the narrowing of their lead attests to the power of community involvement, and the stagnation that results from a group of paranoid OpenAI scientists jerking each other off behind closed doors for several years.
And Qwen3.8-27B still outperforms, or is on par, with several much larger SOTA models.
Qwen3.8-27B is not a great counter example for “benchmaxxing”. Qwen3.8-27B scores higher than Opus 4.6 max on benchmarks in almost all applications. Don’t get me wrong, Qwen’s models do perform well in real world applications. But not Opus 4.6-well.
@Weisunding I suspect that GPT-5.6 luna and Qwen3.8-27B are of a similar parameter count. This just means that Qwen is catching up with the closed weight/ closed source labs, which is great. The initial success of OpenAI and the narrowing of their lead attests to the power of community involvement, and the stagnation that results from a group of paranoid OpenAI scientists jerking each other off behind closed doors for several years.
And Qwen3.8-27B still outperforms, or is on par, with several much larger SOTA models.
What an insightful assessment! The paranoid OpenAI and Anthropic leads hate the chinese open models very much with various conspiracy theories.
With regard to benchmaxxing, the DeepSeek-V4-Flash-0731 should be counted, it created lot of bugs in a project and I use the Qwen3.8-27B to fix them quickly.
Have a nice day, and keep moving Laguna, open-weight model is not easy for startup!