Spaces:
Running on CPU Upgrade
Differentiate between models that do and don't disclose their training data
As the title suggests, it is very hard to figure out if the quality of a system is due to technological advancements or simply due to the type and amount of training data. Making it explicit if a system discloses its training data or not would be of great help to the community, imho. Back in the 90's, Darpa/NIST had separate tracks for open and closed-set ASR evaluations.
Very supportive of this idea;
I think it could also be nice to indicate if the model is an Transformer Decoder + Speech Feature Extractor. Now self-supervised models (W2V2, D2V) are being compared to large decoder models without context, and it undermines their success.
I also noticed that some models seem to have transformer language models attached to them (speechbrain-wav2vec2 , 1.77 WER) and some use probing (wav2vec2 gets 11.13 on libri-speech clean).
hi all! small update, we've recently added a (recommended) checkpoint to the submission process that asks submitters to disclose training data. As we collect this information, we'll be able to add such a column to the leaderboard.
@GokseninYuksel it is possible toggle on/off the "Encoder" and "Decoder" columns to reveal the architectures and then sort the architectures to see how each category performs. We could add another toggle similar to the existing "Show proprietary (API) models", which for example is called "Show LLM-based models" to be able to include/exclude models with a large decoder? If there's enough interest for that we could go forward with it
Hey @bezzam
Thank you for adding such a toggle! Appreciate it a lot.
I had a quick check and it seems accurate. I'll let you know if I find some mistakes, I can also add a PR maybe :)
