tatsu-lab/alpaca
Viewer โข Updated โข 52k โข 92.3k โข 1.4k
How to use tog/TinyLlama-1.1B-alpaca-chat-v1.5 with Transformers:
# Use a pipeline as a high-level helper
from transformers import pipeline
pipe = pipeline("text-generation", model="tog/TinyLlama-1.1B-alpaca-chat-v1.5") # pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM
tokenizer = AutoTokenizer.from_pretrained("tog/TinyLlama-1.1B-alpaca-chat-v1.5")
model = AutoModelForCausalLM.from_pretrained("tog/TinyLlama-1.1B-alpaca-chat-v1.5", device_map="auto")How to use tog/TinyLlama-1.1B-alpaca-chat-v1.5 with vLLM:
# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "tog/TinyLlama-1.1B-alpaca-chat-v1.5"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
-H "Content-Type: application/json" \
--data '{
"model": "tog/TinyLlama-1.1B-alpaca-chat-v1.5",
"prompt": "Once upon a time,",
"max_tokens": 512,
"temperature": 0.5
}'docker model run hf.co/tog/TinyLlama-1.1B-alpaca-chat-v1.5
How to use tog/TinyLlama-1.1B-alpaca-chat-v1.5 with SGLang:
# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
--model-path "tog/TinyLlama-1.1B-alpaca-chat-v1.5" \
--host 0.0.0.0 \
--port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
-H "Content-Type: application/json" \
--data '{
"model": "tog/TinyLlama-1.1B-alpaca-chat-v1.5",
"prompt": "Once upon a time,",
"max_tokens": 512,
"temperature": 0.5
}'docker run --gpus all \
--shm-size 32g \
-p 30000:30000 \
-v ~/.cache/huggingface:/root/.cache/huggingface \
--env "HF_TOKEN=<secret>" \
--ipc=host \
lmsysorg/sglang:latest \
python3 -m sglang.launch_server \
--model-path "tog/TinyLlama-1.1B-alpaca-chat-v1.5" \
--host 0.0.0.0 \
--port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
-H "Content-Type: application/json" \
--data '{
"model": "tog/TinyLlama-1.1B-alpaca-chat-v1.5",
"prompt": "Once upon a time,",
"max_tokens": 512,
"temperature": 0.5
}'How to use tog/TinyLlama-1.1B-alpaca-chat-v1.5 with Docker Model Runner:
docker model run hf.co/tog/TinyLlama-1.1B-alpaca-chat-v1.5
This is the chat model finetuned on top of PY007/TinyLlama-1.1B-intermediate-step-715k-1.5T. The dataset used is tatsu-lab/stanford_alpaca.
Below is an instruction that describes a task. Write a response that appropriately completes the request.
### Instruction:
{instruction}
### Response:
You can use it with the transformers library:
from transformers import AutoTokenizer
import transformers
import torch
model = "tog/TinyLlama-1.1B-alpaca-chat-v1.5"
tokenizer = AutoTokenizer.from_pretrained(model)
pipeline = transformers.pipeline(
"text-generation",
model=model,
torch_dtype=torch.float16,
device_map="auto")
sequences = pipeline(
'###Instruction:\nWhat is a large language model? Be concise.\n\n### Response:\n',
do_sample=True,
top_k=10,
num_return_sequences=1,
eos_token_id=tokenizer.eos_token_id,
max_length=200)
for seq in sequences:
print(f"{seq['generated_text']}")
You should get something along those lines:
Setting `pad_token_id` to `eos_token_id`:2 for open-end generation.
Result: ###Instruction:
What is a large language model? Be concise.
### Response:
A large language model is a type of natural language understanding model that can learn to accurately recognize and interpret text data by understanding the context of words. Languages used for text understanding are typically trained on a corpus of text data.