Instructions to use peculiar-ragdoll/Sharp-Spark-X2.5-4B-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use peculiar-ragdoll/Sharp-Spark-X2.5-4B-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf peculiar-ragdoll/Sharp-Spark-X2.5-4B-GGUF:Q4_K_XL
# Run inference directly in the terminal:
llama cli -hf peculiar-ragdoll/Sharp-Spark-X2.5-4B-GGUF:Q4_K_XL

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf peculiar-ragdoll/Sharp-Spark-X2.5-4B-GGUF:Q4_K_XL
# Run inference directly in the terminal:
llama cli -hf peculiar-ragdoll/Sharp-Spark-X2.5-4B-GGUF:Q4_K_XL

Build from source code

git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-app
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama serve -hf peculiar-ragdoll/Sharp-Spark-X2.5-4B-GGUF:Q4_K_XL
# Run inference directly in the terminal:
./build/bin/llama cli -hf peculiar-ragdoll/Sharp-Spark-X2.5-4B-GGUF:Q4_K_XL

Use Docker

docker model run hf.co/peculiar-ragdoll/Sharp-Spark-X2.5-4B-GGUF:Q4_K_XL

Llama new
LM Studio
Jan

vLLM

How to use peculiar-ragdoll/Sharp-Spark-X2.5-4B-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "peculiar-ragdoll/Sharp-Spark-X2.5-4B-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "peculiar-ragdoll/Sharp-Spark-X2.5-4B-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/peculiar-ragdoll/Sharp-Spark-X2.5-4B-GGUF:Q4_K_XL

Ollama
How to use peculiar-ragdoll/Sharp-Spark-X2.5-4B-GGUF with Ollama:
```
ollama run hf.co/peculiar-ragdoll/Sharp-Spark-X2.5-4B-GGUF:Q4_K_XL
```
Unsloth Desktop

How to use peculiar-ragdoll/Sharp-Spark-X2.5-4B-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf peculiar-ragdoll/Sharp-Spark-X2.5-4B-GGUF:Q4_K_XL

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "peculiar-ragdoll/Sharp-Spark-X2.5-4B-GGUF:Q4_K_XL"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use peculiar-ragdoll/Sharp-Spark-X2.5-4B-GGUF with Docker Model Runner:
```
docker model run hf.co/peculiar-ragdoll/Sharp-Spark-X2.5-4B-GGUF:Q4_K_XL
```

Lemonade

How to use peculiar-ragdoll/Sharp-Spark-X2.5-4B-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull peculiar-ragdoll/Sharp-Spark-X2.5-4B-GGUF:Q4_K_XL

Run and chat with the model

lemonade run user.Sharp-Spark-X2.5-4B-GGUF-Q4_K_XL

List all available models

lemonade list

Hermes Agent

How to use peculiar-ragdoll/Sharp-Spark-X2.5-4B-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf peculiar-ragdoll/Sharp-Spark-X2.5-4B-GGUF:Q4_K_XL

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default peculiar-ragdoll/Sharp-Spark-X2.5-4B-GGUF:Q4_K_XL

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use peculiar-ragdoll/Sharp-Spark-X2.5-4B-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf peculiar-ragdoll/Sharp-Spark-X2.5-4B-GGUF:Q4_K_XL

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "peculiar-ragdoll/Sharp-Spark-X2.5-4B-GGUF:Q4_K_XL" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Awesome quant

by scarletkiss - opened Sep 25

Discussion

scarletkiss

Sep 25

•

edited Sep 25

--- Setup ---
CPU: Ryzen 7 8845HS
GPU: RTX 4060 8GB
RAM: 32GB DDR5-5600
beellama.cpp fork v0.4.6 with the following flags
-fitt 512 --jinja --temp 0.6 --top-p 0.95 --top-k 20 -np 1 -ctk kvarn6 -ctv kvarn6 --reasoning-budget 32768(ctx size fits to 198144)
Harness used: pi, hermes
Quant: Q6_K_XL
--- Review ---
After trying the model out for an afternoon, it is really neat for my use case (programming). Average of 1.75k±250 t/s prefills and 35±5 t/s decode, it gets the job done eventually. I did have to set a reasoning budget when using hermes otherwise it would reason for very long time, perhaps due to the size of the model, having reasoning budget removed likely helps it not confidently implement something badly.

The biggest issue I have seen so far is offloading, it tanks the prefill and decode to extremely egregious speeds with a prefill of ~100 t/s and 10 t/s decode. Perhaps it is my setup, but if you want to use this model with offloading I would suggest benchmarking it lest you use it overnight on autonomous work.

king0770

Sep 25

GPU: NVIDIA RTX 5070 Ti, 16 GB VRAM
RAM: 91 GiB total, approximately 87 GiB available
CPU: Ryzen 9 9950X
llama: 0.4.1-dev, build 11046, commit 60081bb2b

--models-dir ~/huggingface -c 250000 --parallel 1 --gpu-layers all --flash-attn on --cache-type-k q8_0 --cache-type-v q8_0 --batch-size 512 --ubatch-size 128 --jinja --reasoning-format deepseek --temp 0.6 --top-p 0.95 --top-k 0 --min-p 0 -ngl auto

I am still experimenting, and no expert by any means; but the model is very performant. I haven't seen the --reasoning-budget setting, going to give that a try!

peculiar-ragdoll

Owner Sep 25

Love to hear it! <3 Just because I want to learn more, is there any particular reason either of you are not using CyberTiel or TielCoder instead with partial GPU offloading? Should be just as fast (Due to MoE architecture) and twice as smart, small kv cache for full context, and just needs a bit of the RAM in addition to the GPU. Especially with the RTX 5070 I would think Tiel Q4 with partial offloading will be very fast, but it should also be decent on the 4060.

scarletkiss

Sep 25

I do use CyberTiel, it really just comes down to TTFT just for small tasks, since it would take about a minute to 2. At 128K context size my prefills are 200~300t/s with decodes at 40t/s on Cyber no MTP. I will be using Cyber for bigger tasks that the small model can't handle :3

peculiar-ragdoll

Owner Sep 25

Nice, thanks for the answer!:)

king0770

Sep 26

Love to hear it! <3 Just because I want to learn more, is there any particular reason either of you are not using CyberTiel or TielCoder instead with partial GPU offloading? Should be just as fast (Due to MoE architecture) and twice as smart, small kv cache for full context, and just needs a bit of the RAM in addition to the GPU. Especially with the RTX 5070 I would think Tiel Q4 with partial offloading will be very fast, but it should also be decent on the 4060.

You are giving me more credit than I deserve! LOL! But seriously, I haven't heard about CyberTiel or TielCoder. If you have settings to suggest, I can try them out. TBH, I am still learning.

peculiar-ragdoll

Owner Sep 26

@king0770 TielCoder and the slightly improved and uncensored CyberTiel are the two models are the smartest fast agentic coders you can run on your system currently! For running them I recommend using --n-cpu-moe 16 in order to activate partial GPU offloading, and --flash-attn on --jinja -c 262144 --cache-type-k q8_0 --cache-type-v q8_0 --temp 0.6 --top-p 0.95 --top-k 20 --min-p 0 for sampling parameters and context. IF you want to run CyberTiel, which is uncensored, you should be proficient in safely sandboxing your agents to reduce risk. TielCoder is almost as good and also safer, still better than any other coder model in the 35B-A3B MoE class, so if you're still learning I'd start there!

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment