๐Ÿ‚ domestic-yak, a Macedonian LM (base version)

Model Summary

This model is a Macedonian language adaptation of the Llama 3.1 8B model. It has undergone continued pretraining on a deduplicated version of the Macedonian Corpus Raw dataset, containing approximately 1.6 billion tokens. The model has been pretrained for one epoch on this corpus, making it well-suited for tasks involving the Macedonian language, such as text classification, language generation, and translation.

๐Ÿ“Š Results

The table below compares the performance of our model, domestic-yak-8B, with its foundational model, Llama 3.1-8B Instruct evaluated using the macedonian-llm-eval benchmark.

As shown in the table, domestic-yak-8B consistently outperforms its foundational model on all tasks.

Task (mk-version) domestic-yak-8B Llama 3.1-8B Instruct
ARC Easy 0.5244 ยฑ 0.0102 0.4453 ยฑ 0.0102
ARC Challenge 0.3183 ยฑ 0.0136 0.2824 ยฑ 0.0132
BoolQ 0.7676 ยฑ 0.0074 0.7639 ยฑ 0.0074
HellaSwag 0.4324 ยฑ 0.0049 0.3740 ยฑ 0.0048
Openbook QA 0.2920 ยฑ 0.0204 0.2520 ยฑ 0.0194
PIQA 0.6687 ยฑ 0.0110 0.5865 ยฑ 0.0115
NQ Open 0.0416 ยฑ 0.0033 0.0335 ยฑ 0.0030
WinoGrande 0.6259 ยฑ 0.0136 0.5683 ยฑ 0.0139

Quantitative evaluation is just one part of the story. We also conducted qualitative tests, interacting with the model to assess its ability to generate coherent and relevant responses in Macedonian. The results were satisfying: the model not only mastered the language's grammar but also demonstrated a natural and fluent style.

Hereโ€™s a comparison showing the answers from Llama-3.1-8B-Instruct and domestic-yak-8B:

  • Llama-3.1-8B-Instruct:
    "ะ—ะดั€ะฐะฒะพ! ะกะตะบะพั˜ ะฟะฐั‚ ะบะพะณะฐ ะณะพ ะบะพั€ะธัั‚ะธัˆ ะพะฒะฐะฐ ะฟะปะฐั‚ั„ะพั€ะผะฐ, ัะพ ั‚ะพะฑะต ัะธ ะฒะพ ะบะพะฝั‚ะฐะบั‚ ัะพ ะผะพั˜ะพั‚ ัะธัั‚ะตะผ ะฝะฐ ะฒะตัˆั‚ะฐั‡ะบะฐ ะธะฝั‚ะตะปะธะณะตะฝั†ะธั˜ะฐ. ะœะพั˜ะพั‚ ะทะฐะดะฐั‡ะฐ ะต ะดะฐ ั‚ะธ ะณะพ ะพะฑะตะทะฑะตะดะฐะผ ะธะฝั„ะพั€ะผะฐั‚ะธะฒะตะฝ ะธ ะบะพั€ะธัะตะฝ ะพะดะณะพะฒะพั€ ะฝะฐ ัะตะบะพะต ะฟั€ะฐัˆะฐัšะต ัˆั‚ะพ ะผะพะถะตัˆ ะดะฐ ะผะธ ะณะพ ะฟะพัั‚ะฐะฒะธัˆ. ะกะต ะฝะฐั€ะตะบัƒะฒะฐะผ 'ะัะธัั‚ะตะฝั‚', ะธ ััƒะผ ั‚ัƒะบะฐ ะทะฐ ะดะฐ ั‚ะธ ะณะพ ะฟั€ะฐะฒะฐะผ ะถะธะฒะพั‚ ะฟะพะปะตัะตะฝ ะธ ะธะฝั‚ะตั€ะตัะฝะธะพั‚. ะจั‚ะพ ั‚ะฒะพะตั‚ะพ ะฟั€ะฐัˆะฐัšะต ะต?"

  • domestic-yak-8B:
    "ะ—ะดั€ะฐะฒะพ, ั˜ะฐั ััƒะผ ะฐัะธัั‚ะตะฝั‚ ะดะธะทะฐั˜ะฝะธั€ะฐะฝ ะดะฐ ะพะฑะตะทะฑะตะดะธ ะธะฝั„ะพั€ะผะฐั†ะธะธ ะธ ะพะดะณะพะฒะพั€ะธ ะฝะฐ ะฟั€ะฐัˆะฐัšะฐ ะพะด ะบะพั€ะธัะฝะธั†ะธั‚ะต. ะœะพะถะฐะผ ะดะฐ ะพะดะณะพะฒะฐั€ะฐะผ ะฝะฐ ัˆะธั€ะพะบ ัะฟะตะบั‚ะฐั€ ะฝะฐ ะฟั€ะฐัˆะฐัšะฐ, ะพะด ะพะฟัˆั‚ะธ ั„ะฐะบั‚ะธ ะธ ะธะฝั„ะพั€ะผะฐั†ะธะธ ะดะพ ัะฟะตั†ะธั˜ะฐะปะธะทะธั€ะฐะฝะธ ะฟั€ะฐัˆะฐัšะฐ ะฒะพ ั€ะฐะทะปะธั‡ะฝะธ ะพะฑะปะฐัั‚ะธ. ะœะพั˜ะฐั‚ะฐ ั†ะตะป ะต ะดะฐ ะพะฑะตะทะฑะตะดะฐะผ ั‚ะพั‡ะฝะธ ะธ ะบะพั€ะธัะฝะธ ะพะดะณะพะฒะพั€ะธ, ะบะฐะบะพ ะธ ะดะฐ ะณะพ ะพะปะตัะฝะฐะผ ะฟั€ะพั†ะตัะพั‚ ะฝะฐ ะฟั€ะตะฑะฐั€ัƒะฒะฐัšะต ะธ ะดะฐ ะทะฐัˆั‚ะตะดะฐะผ ะฒั€ะตะผะต ะทะฐ ะบะพั€ะธัะฝะธั†ะธั‚ะต. ะกะปะพะฑะพะดะฝะพ ะฟะพัั‚ะฐะฒะธ ะผะธ ะณะพ ั‚ะฒะพะตั‚ะพ ะฟั€ะฐัˆะฐัšะต ะธ ัœะต ัะต ะพะฑะธะดะฐะผ ะดะฐ ั‚ะธ ะฟะพะผะพะณะฝะฐะผ."

๐Ÿ”‘ Key Details

โš ๏ธ Limitations

  • Biases: The model may show biases present in the training dataset. Efforts were made to clean and deduplicate the corpus, but further bias mitigation might be necessary for sensitive applications.
  • Domain Specificity: While the dataset covers diverse domains, performance may vary for niche or underrepresented topics. For example, the dataset is heavily skewed toward 'news'-themed texts, while domains such as 'science' or 'medicine' are less represented.
  • Chat Capabilities: This version is the base model so its chat capabilities might be limited. If you would like to chat use the instruct version.

๐Ÿ“ฌ Contact

For inquiries, feedback, or contributions, please feel free to reach out to the core team:

Citation

@article{krsteski2025towards,
  title={Towards Open Foundation Language Model and Corpus for Macedonian: A Low-Resource Language},
  author={Krsteski, Stefan and Tashkovska, Matea and Sazdov, Borjan and Gjoreski, Hristijan and Gerazov, Branislav},
  journal={arXiv preprint arXiv:2506.09560},
  year={2025}
}

Paper

The model was presented in the paper Towards Open Foundation Language Model and Corpus for Macedonian: A Low-Resource Language.

Downloads last month
66
Safetensors
Model size
8B params
Tensor type
BF16
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for LVSTCK/domestic-yak-8B

Finetuned
(3,320)
this model
Finetunes
1 model
Quantizations
1 model

Paper for LVSTCK/domestic-yak-8B