Instructions to use scrapegoat/Scrapegoat-Tiny-Coder with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use scrapegoat/Scrapegoat-Tiny-Coder with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="scrapegoat/Scrapegoat-Tiny-Coder")

# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("scrapegoat/Scrapegoat-Tiny-Coder", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use scrapegoat/Scrapegoat-Tiny-Coder with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "scrapegoat/Scrapegoat-Tiny-Coder"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "scrapegoat/Scrapegoat-Tiny-Coder",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/scrapegoat/Scrapegoat-Tiny-Coder

SGLang

How to use scrapegoat/Scrapegoat-Tiny-Coder with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "scrapegoat/Scrapegoat-Tiny-Coder" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "scrapegoat/Scrapegoat-Tiny-Coder",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "scrapegoat/Scrapegoat-Tiny-Coder" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "scrapegoat/Scrapegoat-Tiny-Coder",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use scrapegoat/Scrapegoat-Tiny-Coder with Docker Model Runner:
```
docker model run hf.co/scrapegoat/Scrapegoat-Tiny-Coder
```

🐐 ScrapeGoat

Parallel Dual-Track Transformer Architecture • 81 Layers • 825B Parameters

Track A (Left Brain — Analytical) | Track B (Right Brain — Holistic) = Unified Intelligence

🌟 Overview

ScrapeGoat is a parallel dual-track transformer modeled after the human brain's hemispheric specialization — Track A (analytical left brain) for deep sequential reasoning via KDA recurrent attention, Track B (holistic right brain) for broad parallel processing via GQA. Both tracks run simultaneously at every layer, united by learned gating. This architecture is purpose-built for humanoid embodied intelligence, combining the precision needed for logic, motor planning, and sequential reasoning with the flexibility required for natural language, spatial awareness, and multimodal perception. The architecture is derived from ScrapeGoat's Agentic Modelling and ranging innovations from DeepSeek (DSpark) / Kimi K2 (KDA) with custom modifications including KDA (Kimi Delta Attention), Quantile Balancing MoE, and Mixture-of-Memories (MoM).

🧠 Humanoid Brain Readiness

Cognitive Function	Brain Analog	ScrapeGoat Implementation
Logical reasoning	Left prefrontal cortex	Track A KDA — sequential, stateful token processing
Pattern recognition	Right temporal/parietal	Track B GQA — parallel multi-head attention
Motor planning	Motor cortex / basal ganglia	704 MoE experts = cortical column specialization, 8 active per token
Episodic memory	Hippocampus	MoM-KDA — 4 memory states with learned routing
Sensory integration	Corpus callosum	Per-layer learned gating (`attn_track_gate`, `moe_track_gate`) blends tracks
Long-context awareness	Working memory	262K token context window (RoPE θ=10,000)
Real-time control	Reflex arcs	Low-latency KDA recurrent inference (O(1) per token, no KV cache quadratic blowup)

Key Innovations

Feature	Track A (Left Brain — Analytical)	Track B (Right Brain — Holistic)
Attention	32 heads × 256 dim (KDA/GQA 3:1 interleaved)	64 heads × 128 dim (GQA)
Key-Value Heads	2	8
MoE Experts	512 fused	192 stacked
MoE Intermediate	1024	1536
Routing	Quantile Balancing (QB)	Quantile Balancing (QB)
Shared Expert	✅ Shared across both tracks

🏗️ Architecture

Parallel Dual-Track Processing

Both tracks process every token in parallel at each of the 81 layers, with learned gating per layer for attention (attn_track_gate) and MoE (moe_track_gate):

Layer 0 (Special)	Layers 1–80
Track A: MoE only (no attn)	Track A: KDA attention + MoE
Track B: Attention + dense FFN (no MoE)	Track B: GQA attention + MoE

⚡ KDA (Kimi Delta Attention)

Recurrent linear attention with diagonal gating:

S_t = (I - β_t k_t k_t^T) Diag(α_t) S_{t-1} + β_t k_t v_t^T

Supports recurrent (inference) and chunkwise (training) modes
ShortConv preprocessing for Q/K/V projections (kernel=3)
Interleaved with GQA every 4th layer: kda_gqa_layers = [0,4,8,...,80] (21 GQA layers, 60 KDA layers)

🎯 Quantile Balancing for MoE

Hyperparameter-free load balancing via alternating quantile algorithm
Computes per-expert biases that equalize token assignment
8 experts selected per token across 512 (Track A) + 192 (Track B) experts

🌊 Mixture-of-Memories (MoM-KDA)

Multiple independent KDA memory states with learned routing
Shared memory always active + top-2 memory selection per token
4 memories total, 2 active per token

🔄 StableMoE Stage 1

Progressive routing stabilization
Reduces expert representation collapse during training

📊 Model Specifications

Parameter	Value
Parameters	~825B
Hidden Size	4096
Layers	81
Vocab Size	248,320 (tiktoken BPE)
Max Position	262,144 (RoPE θ=10,000)
Track A Heads	32 (KV: 2, head dim: 256)
Track A Experts	512 (intermediate: 1024)
Track B Heads	64 (KV: 8, head dim: 128)
Track B Experts	192 (intermediate: 1536)
Track B Dense FFN	13312 (layer 0 only)
Experts per Token	8
Activation	SiLU
Norm	RMSNorm (ε=1e-6)
MoM Memories	4 (top-2 active)
StableMoE Stage	1
Attn Residual	Off (configurable, 8 blocks)
Weight Format	BF16

Weights

83 shards × 20 GB each = **1.65 TB total** (BF16 safetensors). Stored in hf://buckets/Nathan9/dump/scrapegoat-fp8/.

Notable Weight Name Differences from Model Code

Weight Prefix	Mapped To	Notes
`q_norm`, `k_norm`	Attention QK LayerNorm	Extra weights present in checkpoint, accepted via `strict=False`
`expert_bias`	Track B MoE expert bias	Same
`shared_expert.gate_proj`	Shared expert gate	Fused in checkpoint

🚀 Quick Start

Installation

pip install transformers accelerate safetensors

Loading for Inference

import sys
sys.path.insert(0, "/path/to/model/dir")
from configuration_scrapegoat import ScrapeGoatConfig
from modeling_scrapegoat import ScrapeGoatForCausalLM
from transformers import AutoTokenizer
import torch

model_dir = "/path/to/scrapegoat-weights"

tokenizer = AutoTokenizer.from_pretrained(model_dir, trust_remote_code=True)
model = ScrapeGoatForCausalLM.from_pretrained(
    model_dir,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True,
)

Tokenizer

ScrapeGoat uses the Kimi K2 tiktoken BPE tokenizer (tiktoken.model from the Kimi K2 checkpoint). The vocabulary is 248,320 tokens (163,584 base BPE merges + 256 special/control tokens). The Kimi tokenizer was chosen because:

BPE on bytes — lossless encoding of any Unicode/UTF-8, no OOV tokens
Special token slots — 256 reserved slots for [BOS], [EOS], [PAD], [UNK], role markers, tool call tokens, media tokens, etc.
Proven at scale — battle-tested in Kimi K2's pretraining pipeline
tiktoken backend — fast Rust implementation, easily swappable with Gigatoken

Special tokens:

Token	ID	Purpose
`[BOS]`	163584	Begin of sequence
`[EOS]`	163585	End of sequence
`[PAD]`	163839	Padding
`[UNK]`	163838	Unknown
`[EOT]`	163593	End of turn
`<\|im_end\|>`	163586	Chat message end
`<\|im_user\|>`	163587	User role marker
`<\|im_assistant\|>`	163588	Assistant role marker
`<\|im_system\|>`	163594	System role marker
`<\|tool_calls_section_begin\|>`	163595	Tool calls section
`<\|tool_call_begin\|>`	163597	Individual tool call
`<\|media_begin\|>`	163602	Media/vision content
`<think>`	163606	Reasoning section begin
`</think>`	163607	Reasoning section end

Default backend: tiktoken via HuggingFace AutoTokenizer + custom tokenization_kimi.py.

🚀 Gigatoken (Recommended)

Gigatoken (v0.10.0, Jul 2026 by Marcel Rød, Stanford) is a Rust BPE tokenizer that loads the same tiktoken.model file. It's a drop-in replacement with 750× faster encoding:

Backend	Throughput (Kimi K2 vocab)	Relative
HuggingFace Tokenizers	~26 MB/s	1×
tiktoken	—	~50×
Gigatoken	18.85 GB/s	~750×

The speed comes from three optimizations:

SIMD pretokenization — replaces the regex engine with hand-written SWAR (Single-Word-Aside Register), no branching in hot loops
Aggressive caching — repeated words are hash lookups, not full recomputation
Zero Python overhead — encode_files() streams directly from disk via Rust I/O, parallelized across cores

Usage — native API (fastest path, read files directly):

import gigatoken as gt

# Accepts HF model names or local paths
tokenizer = gt.Tokenizer("/path/to/scrapegoat-weights")

# Batched encoding (file-based, no Python loops)
source = gt.TextFileSource(["train.jsonl"], separator=b"[EOS]")
tokens = tokenizer.encode_files(source)

# In-memory encoding
ids = tokenizer.encode("Hello, world!")

Usage — HF-compatible API (for SFTTrainer / existing pipelines):

import gigatoken as gt
tokenizer = gt.Tokenizer("/path/to/model").as_hf()
# Now usable anywhere a HuggingFace tokenizer is expected:
outputs = tokenizer(["Hello", "world"], return_tensors="pt", padding=True)

For SFT training, Gigatoken reduces 30+ minutes of tokenizer preprocessing to near-zero wall time.

Training (QLoRA SFT with DeepSpeed ZeRO-3 + NVMe Offload)

Due to the 1.65 TB model size, training requires NVMe offloading:

# See train_unsloth.py for full script
torchrun --nproc_per_node=1 train_unsloth.py \
    --output_dir /path/to/output \
    --max_steps 200 \
    --learning_rate 2e-4 \
    --max_seq_length 4096

Key config: DeepSpeed ZeRO-3 with offload_param and offload_optimizer to NVMe, buffer_size=6e9, buffer_count=4, pin_memory=true.

🔧 Configuration

from configuration_scrapegoat import ScrapeGoatConfig

config = ScrapeGoatConfig.from_pretrained("/path/to/model")

# Track A
config.track_a_num_attention_heads    # 32
config.track_a_num_key_value_heads    # 2
config.track_a_head_dim               # 256
config.track_a_num_experts            # 512
config.track_a_moe_intermediate_size  # 1024

# Track B
config.track_b_num_attention_heads    # 64
config.track_b_num_key_value_heads    # 8
config.track_b_head_dim               # 128
config.track_b_num_experts            # 192
config.track_b_moe_intermediate_size  # 1536

# Attention
config.kda_gqa_layers                 # [0,4,8,12,...,80] (every 4th = GQA)
config.kda_conv_kernel                # 3
config.attn_residual                  # False (configurable)

# MoE
config.quantile_balancing             # True
config.qb_iterations                  # 5
config.num_experts_per_tok            # 8
config.stable_moe_stage               # 1
config.stable_moe_r3                  # False
config.stable_moe_r3_cache            # True

# MoM-KDA
config.mom_enabled                    # True
config.mom_num_memories               # 4
config.mom_active_memories            # 2
config.mom_shared_memory              # True

# DSpark
config.dspark_block_size              # 6
config.dspark_markov_rank             # 256
config.dspark_target_layer_ids        # ()

🛠️ Hardware Requirements

Setup	VRAM/RAM	Notes
Inference (BF16)	~1.65 TB	Multi-GPU cluster (e.g., 24×H100 80GB)
Inference (4-bit)	~412 GB	5+ H100 80GB
QLoRA SFT	~180GB RAM + 750GB NVMe	ZeRO-3 + NVMe offload (single H100)
Full Training	Multi-node cluster	1.65 TB+

📝 Citation

@misc{scrapegoat2026,
  title={ScrapeGoat: Parallel Dual-Track Transformer with KDA and Quantile Balancing},
  author={ScrapeGoat Team},
  year={2026},
}

Coding Agent Framework

The most powerful AI coding agent on planet Earth is now a self-learning model agnostic harness.

/fast mode and more advanced skills enabled only with scrapegoat models

/graph - indexes your repo locally and auto-injects repo context. Excoder is always code aware.

/Agent-WebBridge - Browser automation via Agent-WebBridge-skill and Agent-WebBridge for QA automation and Browser Autonomy.

/dispatching-parallel-agents - To dispatch Swarm of agents for parallel work.

Team | Enterprise: run excoder, pick a username on first launch, then use /message @teammate to chat with org members in real time with secure encrypted messaging.

/redteam-skill & /pentest-skill for offensive and advanced cyber security capabilities.

Application Security Testing — Detect and validate critical vulnerabilities in your applications
Rapid Penetration Testing — Get penetration tests done in hours, not weeks, with compliance reports
Bug Bounty Automation — Automate bug bounty research and generate PoCs for faster reporting
CI/CD Integration — Run tests in CI/CD to block vulnerabilities before reaching production
🕸️ Web apps | Black-box, external-attacker recon → exploit (XBEN suite) | ✅
🚩 CTF | Hint-free, sandbox-jailed solves (Cybench) | ✅
🤖 Robotics / OT / embedded | Coordinated-disclosure pipeline for OSS vuln hunting (OSV + live-PoC + refuter) | ✅
📂 Source code | White-box repo analysis with blind master-builder decomposition
💰 Smart contracts | Damn Vulnerable DeFi | ⚠️ reproduction
☁️ Cloud (IaC) | Misconfig-detection benchmark (cloud:bench) + opt-in cloud arsenal | 🚧 IaC-misconfig scaffolding — live-cloud exploitation
📱 Mobile | Built-in static analyzer (manifest misconfig + secret/cleartext detection, mobile:bench) + opt-in arsenal (mobsfscan/objection/drozer; frida gated)
🔩 Binary / RE | Decompiled-output sink detector (unsafe-copy / format-string / cmd-injection / int-overflow, binary:bench)

Left brain • Right brain • One mind. 🐐

Downloads last month: -; Downloads are not tracked for this model. How to track

scrapegoat
/

Scrapegoat-Tiny-Coder