pcalhoun/gpt-j-6b-8bit-pun-generator

GPT-J-6B Pun Generator

A GPT-J-6B fine-tuned to invent puns that don't already exist — and to explain them. It's a base GPT-J-6B (no chat template) that was quantized to 8-bit and fine-tuned with low-rank adapters, trained alongside phonetic tasks (grapheme↔phoneme conversion and homophone lists) so it can build puns that turn on words which merely sound alike.

Background: "This is the moment I've been training for," said the pun-generating AI.

Some of its output:

• Two guys argued about a painting. There was a rupture in the peace. (peace → piece)
• Old pianists never die they just get tuned away.
• A successful soft drink company is full of cool executives.
• The story of a boy who was born with one eye in the wrong place was told from an unexpected angle.
• "I love kiwis," said Tom kiwwisely. (they're not all zingers)

Files in this repo

There are two ready-to-run formats:

File	Format	Use
model.safetensors (+ config.json, tokenizer)	fp16 safetensors	loads with a plain from_pretrained on modern transformers — no custom code, no 8-bit tricks
ggml-model-q4_0.bin	ggml, 4-bit	CPU-only inference (~3.4 GB, ~4 GB RAM) via ggml's gpt-j
eval.py	reference	the original 2023 inference script — documents the prompt schema

The fp16 safetensors was produced from the original 8-bit training by dequantizing the weights and folding the trained adapters into the dense weights — it is numerically equivalent to the original (matching argmax and ≥0.999999 cosine on held-out probes). The original 8-bit .pt pickle has been removed in favour of this safetensors (safe-format) equivalent.

The prompt format

This is a base model with no chat template. Its entire task interface is a set of control tokens built from GPT-J's unused <|extratoken_N|> vocabulary. To generate a pun unconditionally, prompt with:

<|extratoken_60|><|extratoken_61|>   # TASK_START
<|extratoken_46|><|extratoken_47|>   # generate pun → explanation from keywords
<|extratoken_10|><|extratoken_11|>   # PUN begin
<|extratoken_24|><|extratoken_25|>   # GRAPHEMES (written form) begin

i.e. the concatenated string <|extratoken_60|><|extratoken_61|><|extratoken_46|><|extratoken_47|><|extratoken_10|><|extratoken_11|><|extratoken_24|><|extratoken_25|>, then sample. The pun is emitted as written text; the model also produces an explanation (often as ARPABET phonemes — a quirk of how deeply the phonetic training pervades it). Strip the <|extratoken_N|> tags from the output.

Usage — modern (fp16 safetensors)

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "pcalhoun/gpt-j-6b-8bit-pun-generator"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, torch_dtype=torch.float16).cuda().eval()

pre = ("<|extratoken_60|><|extratoken_61|>"
       "<|extratoken_46|><|extratoken_47|>"
       "<|extratoken_10|><|extratoken_11|>"
       "<|extratoken_24|><|extratoken_25|>")
ids = tok(pre, return_tensors="pt").to(model.device)
out = model.generate(ids.input_ids, do_sample=True, temperature=0.9,
                     top_k=50, top_p=0.98, max_new_tokens=128, pad_token_id=50256)
text = tok.decode(out[0])
import re
print(re.sub(r"<\|extratoken_\d+\|>", " ", text).strip())

Usage — CPU, 4-bit (ggml)

Note: mainline llama.cpp does not support GPT-J — the architecture is only implemented in the gpt-j example of the ggml repo. Build that, and patch its tokenizer to treat the <|extratoken_N|> tags as atomic special tokens (otherwise they get BPE-split and the model isn't conditioned):

git clone https://github.com/ggml-org/ggml && cd ggml
cmake -B build -DCMAKE_BUILD_TYPE=Release && cmake --build build --target gpt-j -j
# download ggml-model-q4_0.bin from this repo, then:
./build/bin/gpt-j -m ggml-model-q4_0.bin -n 128 --top_k 50 --top_p 0.98 --temp 0.9 \
  -p '<|extratoken_60|><|extratoken_61|><|extratoken_46|><|extratoken_47|><|extratoken_10|><|extratoken_11|><|extratoken_24|><|extratoken_25|>'

Runs in ~3.4 GB on disk / ~4 GB RAM, CPU only.

License

Apache-2.0. The model was trained on data derived from public pun corpora; note that some of those corpora carry NonCommercial terms, so treat pun generation for redistribution accordingly.