---
license: apache-2.0
datasets:
- HuggingFaceFW/fineweb-2
language:
- aeb
- amp
- ape
- arb
- azj
- bak
- bbc
- bdq
- bew
- bmr
- bss
- bzj
- caf
- cak
- cmn
- cnh
- crn
- crs
- ctd
- efi
- ell
- eng
- ewe
- gug
- hau
- hil
- hin
- ibo
- ind
- kmr
- kua
- kum
- lat
- lus
- mps
- mri
- mzn
- nrm
- ory
- pan
- pol
- pon
- qug
- qxh
- rus
- sat
- sin
- sme
- smo
- sna
- swh
- tur
- ubu
- uig
- vie
- wal
- wol
- wrk
- yon
- zai
---

All data is taken from Fineweb-2 which consists of web-crawls (cutoff April 2024). The exact data can be obtained by running the code in the repository that will be released later.

Sizes of training data in number of UTF-8 characters:


| Language | Size |
|---------|------|
| abt     | 6608062 |
| aeb     | 49814258 |
|amp|3737645|
|ape|4980833|
|arb|3242372164|
|azj|3242372164|
|bak|486333599|
|bbc|35774794|
|bdq|41451|
|bew|3242372164|
|bmr|2439163|
|bss|1731272|
|bzj|4036823|
|caf|909293|
|cak|14765286|
|cmn|3242372164|
|cnh|141157675|
|crn|5912477|
|crs|88583543|
|ctd|28218128|
|efi|17408332|
|ell|3242372164|
| eng     | 3242372164 |
|ewe|35627506|
|gug|38458467|
|hau|3242372164|
|hil|123755905|
|hin|3242372164|
|ibo|333471679|
|ind|3242372164|
|kmr|3242372164|
|kua|14827393|
|kum|20332449|
|lat|3242372164|
|lus|230520812|
|mps|6855963|
|mri|534077417|
|mzn|2835443|
|nrm|3242372164|
|ory|3242372164|
|pan|3242372164|
|pol|3242372164|
|pon|12919138|
|qug|15887852|
|qxh|2452330|
|rus|3242372164|
|sat|4507904|
|sin|3242372164|
|sme|174411164|
|smo|417316517|
|sna|359593003|
|swh|3242372164|
|tur|3242372164|
|ubu|7941636|
|uig|603363013|
|vie|3242372164|
|wal|11207859|
|wol|18292867|
|wrk|2419613|
|yon|2356518|
|zai|6148374|
|total| 68719476736 |


contact:
- Rob van der Goot (robv@itu.dk)