Instructions to use prompterminal/nanogpt-shakespeare-compressed with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use prompterminal/nanogpt-shakespeare-compressed with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="prompterminal/nanogpt-shakespeare-compressed")

# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("prompterminal/nanogpt-shakespeare-compressed", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use prompterminal/nanogpt-shakespeare-compressed with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "prompterminal/nanogpt-shakespeare-compressed"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "prompterminal/nanogpt-shakespeare-compressed",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/prompterminal/nanogpt-shakespeare-compressed

SGLang

How to use prompterminal/nanogpt-shakespeare-compressed with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "prompterminal/nanogpt-shakespeare-compressed" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "prompterminal/nanogpt-shakespeare-compressed",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "prompterminal/nanogpt-shakespeare-compressed" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "prompterminal/nanogpt-shakespeare-compressed",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use prompterminal/nanogpt-shakespeare-compressed with Docker Model Runner:
```
docker model run hf.co/prompterminal/nanogpt-shakespeare-compressed
```

NanoGPT Shakespeare - Compressed Model

This is a compressed version of nanoGPT trained on Shakespeare text using low-rank matrix decomposition.

Model Details

Parameters: 562,432 (1.4× compression from 804,096 original)
Compression Method: Low-rank decomposition (rank=8) on MLP layers 2 and 3
Architecture: GPT with compressed MLP layers (6 layers, 6 heads, 384 embedding dim)
Training Data: Shakespeare corpus
Tokenization: Character-level
Context Length: 256 tokens

Compression Results

Original Parameters: 804,096
Compressed Parameters: 562,432
Compression Ratio: 1.4× smaller
Parameter Reduction: 30.1% (241,664 fewer parameters)
Method: SVD-based low-rank approximation of weight matrices

Usage

from transformers import AutoModel, AutoTokenizer

# Load model and tokenizer
model = AutoModel.from_pretrained("prompterminal/nanogpt-shakespeare-compressed", trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained("prompterminal/nanogpt-shakespeare-compressed", trust_remote_code=True)

# Generate Shakespeare-style text
text = "ROMEO:"
inputs = tokenizer.encode(text, return_tensors="pt")
outputs = model.generate(inputs, max_length=100, temperature=0.8)
result = tokenizer.decode(outputs[0])
print(result)

LLM AutoEval Configuration

Use this model in LLM AutoEval:

MODEL_ID: prompterminal/nanogpt-shakespeare-compressed
BENCHMARK: nous
TRUST_REMOTE_CODE: true
GPU: RTX 3090

Technical Details

The compression technique uses Singular Value Decomposition (SVD) to decompose large weight matrices into smaller low-rank factors:

Original MLP layer: W ∈ R^(d×4d)
Compressed: W ≈ U·V where U ∈ R^(d×r), V ∈ R^(r×4d)
Rank: r = 8 (much smaller than d = 384)

This maintains model expressiveness while significantly reducing parameters.

Evaluation

This model demonstrates effective neural network compression with minimal performance loss. Perfect for:

Resource-constrained deployment
Edge computing applications
Research into compression techniques
Educational purposes

Citation

Based on nanoGPT by Andrej Karpathy. Compression technique inspired by low-rank approximation methods in deep learning.

Downloads last month: 7