PartAI
/

TookaBERT-Base

@@ -12,197 +12,58 @@ widget:
 - text: "سید ابراهیم رییسی در سال <mask> رییس جمهور ایران شد."
 - text: "دیگر امکان ادامه وجود ندارد. باید قرارداد را <mask> کنیم."
 ---
-# Model Card for Model ID
-<!-- Provide a quick summary of what the model is/does. -->
-This modelcard aims to be a base template for new models. It has been generated using [this raw template](https://github.com/huggingface/huggingface_hub/blob/main/src/huggingface_hub/templates/modelcard_template.md?plain=1).
-## Model Details
-### Model Description
-<!-- Provide a longer summary of what this model is. -->
-- **Developed by:** [More Information Needed]
-- **Funded by [optional]:** [More Information Needed]
-- **Shared by [optional]:** [More Information Needed]
-- **Model type:** [More Information Needed]
-- **Language(s) (NLP):** [More Information Needed]
-- **License:** [More Information Needed]
-- **Finetuned from model [optional]:** [More Information Needed]
-### Model Sources [optional]
-<!-- Provide the basic links for the model. -->
-- **Repository:** [More Information Needed]
-- **Paper [optional]:** [More Information Needed]
-- **Demo [optional]:** [More Information Needed]
-## Uses
-<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
-### Direct Use
-<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
-[More Information Needed]
-### Downstream Use [optional]
-<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
-[More Information Needed]
-### Out-of-Scope Use
-<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
-[More Information Needed]
-## Bias, Risks, and Limitations
-<!-- This section is meant to convey both technical and sociotechnical limitations. -->
-[More Information Needed]
-### Recommendations
-<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
-Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
-## How to Get Started with the Model
-Use the code below to get started with the model.
-[More Information Needed]
-## Training Details
-### Training Data
-<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
-[More Information Needed]
-### Training Procedure
-<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
-#### Preprocessing [optional]
-[More Information Needed]
-#### Training Hyperparameters
-- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
-#### Speeds, Sizes, Times [optional]
-<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
-[More Information Needed]
 ## Evaluation
-<!-- This section describes the evaluation protocols and provides the results. -->
-### Testing Data, Factors & Metrics
-#### Testing Data
-<!-- This should link to a Dataset Card if possible. -->
-[More Information Needed]
-#### Factors
-<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
-[More Information Needed]
-#### Metrics
-<!-- These are the evaluation metrics being used, ideally with a description of why. -->
-[More Information Needed]
-### Results
-[More Information Needed]
-#### Summary
-## Model Examination [optional]
-<!-- Relevant interpretability work for the model goes here -->
-[More Information Needed]
-## Environmental Impact
-<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
-Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
-- **Hardware Type:** [More Information Needed]
-- **Hours used:** [More Information Needed]
-- **Cloud Provider:** [More Information Needed]
-- **Compute Region:** [More Information Needed]
-- **Carbon Emitted:** [More Information Needed]
-## Technical Specifications [optional]
-### Model Architecture and Objective
-[More Information Needed]
-### Compute Infrastructure
-[More Information Needed]
-#### Hardware
-[More Information Needed]
-#### Software
-[More Information Needed]
-## Citation [optional]
-<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
-**BibTeX:**
-[More Information Needed]
-**APA:**
-[More Information Needed]
-## Glossary [optional]
-<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
-[More Information Needed]
-## More Information [optional]
-[More Information Needed]
-## Model Card Authors [optional]
-[More Information Needed]
-## Model Card Contact
-[More Information Needed]

 - text: "سید ابراهیم رییسی در سال <mask> رییس جمهور ایران شد."
 - text: "دیگر امکان ادامه وجود ندارد. باید قرارداد را <mask> کنیم."
 ---
+# Model Details
+TukaBERT models are a family of encoder models trained on Persian in two sizes of base and large.
+These Models pre-trained on over 300GB Persian data including variety of topics such as News, Blogs, Forums,
+Books, etc. They were pre-training with the MLM (WWM) objective using two context lengths.
+## How to use
+You can use this model directly for Masked Language Modeling using the provided code below.
+```Python
+from transformers import AutoTokenizer, AutoModelForMaskedLM
+tokenizer = AutoTokenizer.from_pretrained("PartAI/PartBert-Base")
+model = AutoModelForMaskedLM.from_pretrained("PartAI/PartBert-Base")
+# prepare input
+text = "شهر برلین در کشور <mask> واقع شده است."
+encoded_input = tokenizer(text, return_tensors='pt')
+# forward pass
+output = model(**encoded_input)
+```
+It is also possible to use inference pipelines such as below.
+```Python
+from transformers import pipeline
+inference_pipeline = pipeline('fill-mask', model="PartAI/PartBert-Base")
+inference_pipeline("شهر برلین در کشور <mask> واقع شده است.")
+```
+You can use this model to fine-tune it over your dataset and prepare it for your task.
 ## Evaluation
+TukaBERT models are evaluated on a wide range of NLP downstream tasks, such as Sentiment Analysis (SA), Text Classification, Multiple-choice, Question Answering, and Named Entity Recognition (NER).
+Here are some key performance results:
+| Model name       | DeepSentiPers (f1/acc) | MultiCoNER-v2 (f1/acc) | PQuAD (best_exact/best_f1/HasAns_exact/HasAns_f1) | FarsTail (f1/acc) | ParsiNLU-Multiple-choice (f1/acc) | ParsiNLU-Reading-comprehension (exact/f1) | ParsiNLU-QQP (f1/acc) |
+|------------------|------------------------|------------------------|---------------------------------------------------|-------------------|-----------------------------------|-------------------------------------------|-----------------------|
+| TukaBERT-large   | **85.66/85.78**        | **69.69/94.07**        | **75.56/88.06/70.24/87.83**                       | **89.71/89.72**   | **36.13/35.97**                   | **33.6/60.5**                             | **82.72/82.63**       |
+| TukaBERT-base    | _83.93/83.93_          | _66.23/93.3_           | _73.18_/_85.71_/_68.29_/_85.94_                   | _83.26/83.41_     | 33.6/_33.81_                      | 20.8/42.52                                | _81.33/81.29_         |
+| Shiraz           | 81.17/81.08            | 59.1/92.83             | 65.96/81.25/59.63/81.31                           | 77.76/77.75       | _34.73/34.53_                     | 17.6/39.61                                | 79.68/79.51           |
+| ParsBERT         | 80.22/80.23            | 64.91/93.23            | 71.41/84.21/66.29/84.57                           | 80.89/80.94       | **35.34/35.25**                   | 20/39.58                                  | 80.15/80.07           |
+| XLM-V-base       | _83.43/83.36_          | 58.83/92.23            | _73.26_/_85.69_/_68.21_/_85.56_                   | 81.1/81.2         | **35.28/35.25**                   | 8/26.66                                   | 80.1/79.96            |
+| XLM-RoBERTa-base | _83.99/84.07_          | 60.38/92.49            | _73.72_/_86.24_/_68.16_/_85.8_                    | 82.0/81.98        | 32.4/32.37                        | 20.0/40.43                                | 79.14/78.95           |
+| FaBERT           | 82.68/82.65            | 63.89/93.01            | _72.57_/_85.39_/67.16/_85.31_                     | _83.69/83.67_     | 32.47/32.37                       | _27.2/48.42_                              | **82.34/82.29**       |
+| mBERT            | 78.57/78.66            | 60.31/92.54            | 71.79/84.68/65.89/83.99                           | _82.69/82.82_     | 33.41/33.09                       | _27.2_/42.18                              | 79.19/79.29           |
+| AriaBERT         | 80.51/80.51            | 60.98/92.45            | 68.09/81.23/62.12/80.94                           | 74.47/74.43       | 30.75/30.94                       | 14.4/35.48                                | 79.09/78.84           |
+\*Note because of the randomness in the fine-tuning process, results with less than 1% differences are italic together.
+## How to Cite