Text Generation
Transformers
Safetensors
English
klm
custom_code
small-language-model
base
pretrained
hybrid
gated-deltanet
Instructions to use MinimaLabs/KLM-1.5 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use MinimaLabs/KLM-1.5 with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="MinimaLabs/KLM-1.5", trust_remote_code=True)# pip install -U transformers accelerate # Load model directly from transformers import AutoModelForCausalLM model = AutoModelForCausalLM.from_pretrained("MinimaLabs/KLM-1.5", trust_remote_code=True, device_map="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use MinimaLabs/KLM-1.5 with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "MinimaLabs/KLM-1.5" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "MinimaLabs/KLM-1.5", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker
docker model run hf.co/MinimaLabs/KLM-1.5
- SGLang
How to use MinimaLabs/KLM-1.5 with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "MinimaLabs/KLM-1.5" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "MinimaLabs/KLM-1.5", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "MinimaLabs/KLM-1.5" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "MinimaLabs/KLM-1.5", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }' - Docker Model Runner
How to use MinimaLabs/KLM-1.5 with Docker Model Runner:
docker model run hf.co/MinimaLabs/KLM-1.5
Download configuration_klm.py from MinimaLabs/KLM-1.5: direct link, hf CLI and curl.
- Browser
- Download file 2.02 kB
-
https://huggingface.co/MinimaLabs/KLM-1.5/resolve/main/configuration_klm.py
- Command line
-
hf download hf://MinimaLabs/KLM-1.5/configuration_klm.py
-
curl -L -o configuration_klm.py https://huggingface.co/MinimaLabs/KLM-1.5/resolve/main/configuration_klm.py
2.02 kB
| """KLMConfig: Transformers config for KLM 1.5.""" | |
| from __future__ import annotations | |
| from transformers import PretrainedConfig | |
| class KLMConfig(PretrainedConfig): | |
| model_type = "klm" | |
| def __init__( | |
| self, | |
| vocab_size: int = 4160, | |
| hidden_size: int = 512, | |
| num_hidden_layers: int = 24, | |
| attention_every: int = 4, | |
| gdn_heads: int = 4, | |
| gdn_head_dim: int = 128, | |
| num_attention_heads: int = 8, | |
| num_key_value_heads: int = 4, | |
| intermediate_size: int = 1408, | |
| max_position_embeddings: int = 2048, | |
| rope_theta: float = 10000.0, | |
| n_bigrams: int = 125000, | |
| n_trigrams: int = 38000, | |
| ngram_dim: int = 32, | |
| rms_norm_eps: float = 1e-6, | |
| gdn_norm_eps: float = 1e-5, | |
| decay_floor: float = -20.0, | |
| bos_token_id: int = 1, | |
| eos_token_id: int = 2, | |
| pad_token_id: int = 0, | |
| tie_word_embeddings: bool = True, | |
| **kwargs, | |
| ): | |
| self.vocab_size = vocab_size | |
| self.hidden_size = hidden_size | |
| self.num_hidden_layers = num_hidden_layers | |
| self.attention_every = attention_every | |
| self.gdn_heads = gdn_heads | |
| self.gdn_head_dim = gdn_head_dim | |
| self.num_attention_heads = num_attention_heads | |
| self.num_key_value_heads = num_key_value_heads | |
| self.intermediate_size = intermediate_size | |
| self.max_position_embeddings = max_position_embeddings | |
| self.rope_theta = rope_theta | |
| self.n_bigrams = n_bigrams | |
| self.n_trigrams = n_trigrams | |
| self.ngram_dim = ngram_dim | |
| self.rms_norm_eps = rms_norm_eps | |
| self.gdn_norm_eps = gdn_norm_eps | |
| self.decay_floor = decay_floor | |
| super().__init__(bos_token_id=bos_token_id, eos_token_id=eos_token_id, | |
| pad_token_id=pad_token_id, tie_word_embeddings=tie_word_embeddings, **kwargs) | |
| def is_attention(self, layer: int) -> bool: | |
| return layer % self.attention_every == self.attention_every - 1 | |