Better metrics needed
Hi! The architecture looks promising, but I have a couple of questions regarding the experimental results.
Pretraining efficiency
Avey-B Base was pretrained on 180B FineWeb tokens, which is kinda low compared to ModernBERT. Despite this difference, the reported downstream performance is competitive. Have you conducted any controlled comparisons using the same pretraining data and compute budget , or measured downstream performance as a function of training tokens? I'm particularly interested in whether the improvement comes from the architecture itself or differences in pretraining and evaluation setups.
As in my own experiments, training on FineWeb solely results in pretty bad model quality.
ModernBERT score (and other encoders)
I noticed that ModernBERT Base performs considerably worse in your evaluation than in the original ModernBERT paper:
- MNLI: 86.72 vs. 89.1
- SST-2: 92.32 vs. 96.0
I understand that the fine-tuning protocols differ, but this seems like a substantial gap. Have you tried reproducing the original ModernBERT results or allowing each model a larger, architecture-specific hyperparameter search? In ModernBERT paper they took the best scores known for each model.
Thanks!
- No, we didn't pre-train a transformer model on the same dataset with the same pre-training protocol. Avey-B was already performing on par with existing checkpoints, so we didn't see the need to. Typically (for instance with decoder models) we'd need to do that since pre-training on 100-200B tokens wouldn't get anywhere close to the performance of existing production-grade checkpoints, and the comparison wouldn't make sense. But in this case, it does.
- I agree that for a pure architecture comparison, it would be necessary to train the architectures under the same conditions. In this particular case though, performance comparison with established models is more compelling in my opinion. That doesn't take away from the value of having a matched pre-training, and I'd be happy to see future work which addresses that.
- In my experience, downstream performance of encoder models seems to have a much lower pre-training threshold for saturation than decoder models. I don't have any experiments to prove that at this time, but I don't expect the downstream performance to change much past maybe ~300-500B pre-training tokens. The paper reports 180B; the Avey-B checkpoint here on HF is trained longer, on ~220B tokens (details in the model card).
- That being said, this checkpoint was not made for production use; there's a lot that can be done to further improve the pre-training setup and downstream performance.
- About the benchmarks: every model reported in the paper was fine-tuned under the exact same setup, and each model got its own hyperparameter sweep for each individual benchmark (4 learning rates and 10 seeds per model per benchmark per learning rate, with median for the best learning rate reported). Details in Section 5.1 of the paper.
- The discrepancy in the benchmark scores of the models in the paper is most likely because we fine-tuned the models under pure bf16, including weights and optimizer states. I just ran a small experiment to test this for ModernBERT on MNLI (30k-example subset, 1 epoch): with fp32 weights (bf16 autocast), ModernBERT-base reaches 84.6% MNLI-matched accuracy at lr 6e-5, but only 48.6% with weights and optimizer states in bf16 at the same lr, and at best 70.1% (at lr 1e-4). A full-data run with fp32 weights under this setup gives 89.0%, in line with the 89.1 reported by the ModernBERT authors. More testing is required, but this suggests that fine-tuning with fp32 weights should recover the originally reported performance (at least for ModernBERT on MNLI, per the run above).
- Running 10 seeds × 4 learning rates × 9 models × 9 benchmarks for the paper was a substantial amount of compute, and from what I remember, the speed difference with bf16 added up, so it made sense to use it. Of course, in a practical case where one would be fine-tuning a single model (or a few models) for a single task, it would make sense to use full precision.
- Another thing to note is that with full bf16 training, most of the models seem to prefer higher learning rates than usual, likely because most of each step's updates fall below the bf16 resolution (which probably also explains the worse performance). For the results reported in the paper, among the fine-tuning learning rates tested, the best learning rate for ModernBERT was either 1e-4 or 5e-4 depending on the benchmark, which is quite a bit higher than what people typically use. For Avey-B itself, it was mostly 5e-4 that did the best across all benchmarks.
- Ideally, I'd like to run all benchmarks in fp32, but I don't have the time or compute for it at the moment.
- All of the evaluation code and configs used for the experiments reported in the paper are available at github.avey.ai/avey-b; feel free to go through it.