| --- |
| license: other |
| datasets: |
| - allenai/ai2_arc |
| - unalignment/spicy-3.1 |
| - codeparrot/apps |
| - facebook/belebele |
| - boolq |
| - jondurbin/cinematika-v0.1 |
| - drop |
| - lmsys/lmsys-chat-1m |
| - TIGER-Lab/MathInstruct |
| - cais/mmlu |
| - Muennighoff/natural-instructions |
| - openbookqa |
| - piqa |
| - Vezora/Tested-22k-Python-Alpaca |
| - cakiki/rosetta-code |
| - Open-Orca/SlimOrca |
| - spider |
| - squad_v2 |
| - migtissera/Synthia-v1.3 |
| - datasets/winogrande |
| - nvidia/HelpSteer |
| - Intel/orca_dpo_pairs |
| - unalignment/toxic-dpo-v0.1 |
| - jondurbin/truthy-dpo-v0.1 |
| - allenai/ultrafeedback_binarized_cleaned |
| - Squish42/bluemoon-fandom-1-1-rp-cleaned |
| - LDJnr/Capybara |
| - JULIELab/EmoBank |
| - kingbri/PIPPA-shareGPT |
| license_name: yi-license |
| license_link: https://huggingface.co/01-ai/Yi-34B-200K/blob/main/LICENSE |
| quantized_by: bartowski |
| pipeline_tag: text-generation |
| --- |
| |
| ## Exllama v2 Quantizations of bagel-dpo-34b-v0.2 |
|
|
| Using <a href="https://github.com/turboderp/exllamav2/releases/tag/v0.0.11">turboderp's ExLlamaV2 v0.0.11</a> for quantization. |
|
|
| Each branch contains an individual bits per weight, with the main one containing only the meaurement.json for further conversions. |
|
|
| Conversion was done using the default calibration dataset. |
|
|
| Default arguments used except when the bits per weight is above 6.0, at that point the lm_head layer is quantized at 8 bits per weight instead of the default 6. |
| |
| Original model: https://huggingface.co/jondurbin/bagel-dpo-34b-v0.2 |
| |
| <a href="https://huggingface.co/bartowski/bagel-dpo-34b-v0.2-exl2/tree/2_4">2.4 bits per weight</a> |
| |
| <a href="https://huggingface.co/bartowski/bagel-dpo-34b-v0.2-exl2/tree/3_0">3.0 bits per weight</a> |
| |
| <a href="https://huggingface.co/bartowski/bagel-dpo-34b-v0.2-exl2/tree/3_5">3.5 bits per weight</a> |
| |
| <a href="https://huggingface.co/bartowski/bagel-dpo-34b-v0.2-exl2/tree/4_25">4.25 bits per weight</a> |
| |
| <a href="https://huggingface.co/bartowski/bagel-dpo-34b-v0.2-exl2/tree/6_5">6.5 bits per weight</a> |
| |
| ## Download instructions |
| |
| With git: |
| |
| ```shell |
| git clone --single-branch --branch 4_0 https://huggingface.co/bartowski/bagel-dpo-34b-v0.2-exl2 |
| ``` |
| |
| With huggingface hub (credit to TheBloke for instructions): |
| |
| ```shell |
| pip3 install huggingface-hub |
| ``` |
| |
| To download the `main` (only useful if you only care about measurement.json) branch to a folder called `bagel-dpo-34b-v0.2-exl2`: |
| |
| ```shell |
| mkdir bagel-dpo-34b-v0.2-exl2 |
| huggingface-cli download bartowski/bagel-dpo-34b-v0.2-exl2 --local-dir bagel-dpo-34b-v0.2-exl2 --local-dir-use-symlinks False |
| ``` |
| |
| To download from a different branch, add the `--revision` parameter: |
| |
| ```shell |
| mkdir bagel-dpo-34b-v0.2-exl2 |
| huggingface-cli download bartowski/bagel-dpo-34b-v0.2-exl2 --revision 4_0 --local-dir bagel-dpo-34b-v0.2-exl2 --local-dir-use-symlinks False |
| ``` |
| |
