Instructions to use zai-org/cogvlm2-llama3-chat-19B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use zai-org/cogvlm2-llama3-chat-19B with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="zai-org/cogvlm2-llama3-chat-19B", trust_remote_code=True)
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("zai-org/cogvlm2-llama3-chat-19B", trust_remote_code=True, device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use zai-org/cogvlm2-llama3-chat-19B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "zai-org/cogvlm2-llama3-chat-19B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "zai-org/cogvlm2-llama3-chat-19B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/zai-org/cogvlm2-llama3-chat-19B

SGLang

How to use zai-org/cogvlm2-llama3-chat-19B with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "zai-org/cogvlm2-llama3-chat-19B" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "zai-org/cogvlm2-llama3-chat-19B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "zai-org/cogvlm2-llama3-chat-19B" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "zai-org/cogvlm2-llama3-chat-19B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use zai-org/cogvlm2-llama3-chat-19B with Docker Model Runner:
```
docker model run hf.co/zai-org/cogvlm2-llama3-chat-19B
```

cogvlm2-llama3-chat-19B

Commit History

Update README.md

f592f29
verified

wenyi commited on Sep 3, 2024

Update of modeling_cogvlm.py for Transformers newer version (#15)

e45a3a3
verified

Qishuai commited on Aug 26, 2024

Update modeling_cogvlm.py

2bf7de6
verified

zR commited on May 25, 2024

Update README.md

f84e97e
verified

zR commited on May 23, 2024

Update README.md

43f2bc5
verified

zR commited on May 22, 2024

Update README.md

0d4869f
verified

zR commited on May 21, 2024

fix wrong image

9f600cf
verified

zR commited on May 20, 2024

readme

9296bc6

Ubuntu commited on May 18, 2024

readme

0fe7ab5

Ubuntu commited on May 18, 2024

Upload folder using huggingface_hub

1e1d266
verified

Yuxuan Zhang commited on May 17, 2024

initial commit

34824f6
verified

Yuxuan Zhang commited on May 16, 2024

Commit History

Update README.md f592f29 verified

Update of modeling_cogvlm.py for Transformers newer version (#15) e45a3a3 verified

Update modeling_cogvlm.py 2bf7de6 verified

Update README.md f84e97e verified

Update README.md 43f2bc5 verified

Update README.md 0d4869f verified

fix wrong image 9f600cf verified

readme 9296bc6

readme 0fe7ab5

Upload folder using huggingface_hub 1e1d266 verified

initial commit 34824f6 verified