Update README.md
#5
by
reach-vb
- opened
README.md
CHANGED
|
@@ -23,19 +23,4 @@ The template used to build a prompt for the Instruct model is defined as follows
|
|
| 23 |
```
|
| 24 |
Note that `<s>` and `</s>` are special tokens for beginning of string (BOS) and end of string (EOS) while [INST] and [/INST] are regular strings.
|
| 25 |
|
| 26 |
-
As reference, here is the pseudo-code used to tokenize instructions during fine-tuning:
|
| 27 |
-
```python
|
| 28 |
-
def tokenize(text):
|
| 29 |
-
return tok.encode(text, add_special_tokens=False)
|
| 30 |
-
|
| 31 |
-
[BOS_ID] +
|
| 32 |
-
tokenize("[INST]") + tokenize(USER_MESSAGE_1) + tokenize("[/INST]") +
|
| 33 |
-
tokenize(BOT_MESSAGE_1) + [EOS_ID] +
|
| 34 |
-
…
|
| 35 |
-
tokenize("[INST]") + tokenize(USER_MESSAGE_N) + tokenize("[/INST]") +
|
| 36 |
-
tokenize(BOT_MESSAGE_N) + [EOS_ID]
|
| 37 |
-
```
|
| 38 |
-
|
| 39 |
-
In the pseudo-code above, note that the `tokenize` method should not add a BOS or EOS token automatically, but should add a prefix space.
|
| 40 |
-
|
| 41 |
## Run the model
|
|
|
|
| 23 |
```
|
| 24 |
Note that `<s>` and `</s>` are special tokens for beginning of string (BOS) and end of string (EOS) while [INST] and [/INST] are regular strings.
|
| 25 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 26 |
## Run the model
|