chargoddard

/

llama2-22b

like 46

yes please

Hey, thanks for the interest! I've added the script I used to generate the base model to the repo (frankenllama_22.py).
This actually came out of some experiments I was doing with attention head pruning. I decided to try going the other direction instead, and it's looking pretty promising so far.

For the fine tuning, I used axolotl: https://github.com/OpenAccess-AI-Collective/axolotl

@chargoddard Thanks for posting the script, I'm going to experiment with it. Do you know if it's possible to transplant heads from l2-70b instead of l1-33b like in the original script? And does the script need any changing other than pointing to the right donor?

I can't find this github repo, could you link it?

I can't find this github repo, could you link it?

@Vezora Do you mean the merge script? It's the .py file in the files section of this model.

I can't find this github repo, could you link it?

@Vezora Do you mean the merge script? It's the .py file in the files section of this model.

That's embarrassing, thank you!