omargrist/gemma-4-e2b-besstie-sarcasm-en-au

Gemma 4 E2B BESSTIE Sarcasm - en-AU

This repository contains a LoRA adapter for google/gemma-4-E2B, fine-tuned for binary sarcasm detection on the en-AU portion of surrey-nlp/BESSTIE-CW-26.

The adapter is intentionally variety-specific. During training and validation it only saw en-AU examples, rather than examples from the other BESSTIE varieties. This makes it useful for cross-variety transfer experiments: for example, evaluating an en-AU adapter on en-IN and en-UK test examples.

Model summary

• Base model: google/gemma-4-E2B
• Adapter method: QLoRA PEFT adapter
• Task: binary sarcasm detection
• Output labels: Yes = sarcastic, No = not sarcastic
• Training variety: en-AU
• Training seed used for this adapter: 100
• Dataset: surrey-nlp/BESSTIE-CW-26

This is not a merged standalone model. It contains the PEFT adapter weights and configuration, and should be loaded on top of the Gemma 4 E2B base model.

Dataset and training data

The adapter was fine-tuned on surrey-nlp/BESSTIE-CW-26, a benchmark for sentiment and sarcasm classification across English varieties. The dataset includes text, variety, source, sentiment, and sarcasm fields.

For this adapter, only the en-AU subset was used for training and validation.

Prompt format

The adapter was trained as a next-token prompted classifier. Use the same prompt format at inference time:

Detect sarcasm in the following English user-generated text.
Return Yes if sarcastic and No if not sarcastic.

Text: <text>

Label:

The model was trained to generate one of two single-token labels:

• No
• Yes

Training procedure

The adapter was trained using the following setup:

Setting	Value
Base model	google/gemma-4-E2B
Dataset	surrey-nlp/BESSTIE-CW-26
Training variety	en-AU
Seed used for release	100
Epochs	4
Learning rate	2e-4
Per-device batch size	4
Gradient accumulation	4
Warmup ratio	0.05
Weight decay	0.01
Scheduler	cosine
Max sequence length	512
LoRA rank	8
LoRA alpha	16
LoRA dropout	0.10
Quantisation	4-bit NF4 with double quantisation
Compute dtype	bfloat16
Optimised metric during training	validation loss
Checkpoint selection	best checkpoint by validation loss

Loading the adapter

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import PeftModel

base_model = "google/gemma-4-E2B"
adapter_id = "omargrist/gemma-4-e2b-besstie-sarcasm-en-au"

tokenizer = AutoTokenizer.from_pretrained(adapter_id, use_fast=True)

if tokenizer.pad_token is None:
    tokenizer.pad_token = tokenizer.eos_token

quant_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True,
    bnb_4bit_compute_dtype=torch.bfloat16,
)

model = AutoModelForCausalLM.from_pretrained(
    base_model,
    quantization_config=quant_config,
    dtype=torch.bfloat16,
    device_map="auto",
)

model = PeftModel.from_pretrained(model, adapter_id)
model.eval()

Inference example

import torch

NEGATIVE_LABEL = " No"
POSITIVE_LABEL = " Yes"

def make_prompt(text):
    return (
        "Detect sarcasm in the following English user-generated text.\n"
        "Return Yes if sarcastic and No if not sarcastic.\n\n"
        f"Text: {text}\n\n"
        "Label:"
    )

def label_token_ids(tokenizer):
    no_ids = tokenizer(NEGATIVE_LABEL, add_special_tokens=False)["input_ids"]
    yes_ids = tokenizer(POSITIVE_LABEL, add_special_tokens=False)["input_ids"]

    if len(no_ids) != 1 or len(yes_ids) != 1:
        raise ValueError("Expected single-token labels for No/Yes.")

    return no_ids[0], yes_ids[0]

@torch.inference_mode()
def predict_sarcasm_prob(model, tokenizer, text):
    id_no, id_yes = label_token_ids(tokenizer)
    prompt = make_prompt(text)

    batch = tokenizer(
        prompt,
        return_tensors="pt",
        truncation=True,
        max_length=512,
        add_special_tokens=False,
    ).to(model.device)

    outputs = model(**batch, logits_to_keep=1)
    logits = outputs.logits[:, -1, :]
    label_logits = logits[:, [id_no, id_yes]]

    return torch.softmax(label_logits.float(), dim=-1)[0, 1].item()

text = "Oh brilliant, another meeting that could have been an email."
prob_sarcastic = predict_sarcasm_prob(model, tokenizer, text)

print({"prob_sarcastic": prob_sarcastic})

Results

The training script evaluates all 3 adapters on:

1. the full BESSTIE test set;
2. each individual test variety: en-AU, en-IN, and en-UK.

The decision threshold is tuned on the target validation split for the adapter variety.

Adapter	Test subset	Macro-F1
en-AU	en-AU	0.767
en-AU	en-IN	0.495
en-AU	en-UK	0.602

Citation

If you use this adapter, please cite the BESSTIE dataset paper referenced by the dataset card:

@misc{besstie2024,
  title={BESSTIE: A Benchmark for Sentiment and Sarcasm Classification for Varieties of English},
  year={2024},
  eprint={2412.04726},
  archivePrefix={arXiv}
}