loizlabz/manaca-instruct-pt

Manacá-Instruct-PT

What this is

This is a personal/learning project, not an official or production release. It fine-tunes menezesbruno/manaca-1b-base — an open, Brazilian-Portuguese base language model from LNCC's AI Institute — into an instruction-following model using QLoRA, as a hands-on exercise in the full LLM specialization lifecycle (dataset curation, supervised fine-tuning, evaluation, quantization, and deployment).

An official instruction-tuned release, menezesbruno/manaca-1b-instruct, also exists (experimental v0.1) — see Evaluation results below for a side-by-side comparison against it under the same blind grading protocol, not just against the untuned base model.

Published model: the qlora-v3b adapter (response-only objective on the cleaned v3 dataset) merged into the base model, generated with repetition_penalty: 1.1 — the exact configuration that was evaluated and adopted (eval/results/adoption-decision.md; training manifest runs/qlora-v3b.manifest.json, data fingerprints data/dataset_report.json).

Intended use

Supports five Brazilian-Portuguese text tasks:

• Grammar correction
• Rewriting (e.g. informal → professional tone)
• Summarization
• Simplification (e.g. simplifying formal/bureaucratic language)
• Classification of a message into a fixed category set (evaluated on customer-service messages: reclamação / dúvida / elogio / solicitação)

Not intended for production use, medical/legal advice, or automated decision-making — see Known limitations below.

How to use it

The prompt template is ### Instrução: / (optional ### Entrada:) / ### Resposta:. The evaluation below used the single-line form (instruction and text in one block); the separated form is what the training data uses when an input text exists, and the two were measured to perform the same within noise (eval/results/presentation-experiment.md). Keep repetition_penalty at 1.1: the model was evaluated with it, and higher values (1.3) made it paraphrase text it should copy.

The model reads and writes lowercase. The base tokenizer lowercases all input (NFKC + Lowercase normalizer) and the training data is lowercased with it; outputs come back lowercase. transformers applies that normalizer for you. The GGUF does not: convert_hf_to_gguf.py drops the normalizer, so with llama.cpp you must lowercase the prompt yourself (template included) — an uppercase ### Resposta: tokenizes into pieces the model never saw and it answers with an immediate end-of-text. Use llama-completion (raw completion), not llama-cli: recent llama-cli builds are a chat front-end that wraps the prompt in a chat template the model was not trained on, with the same empty result.

transformers

from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("loizlabz/manaca-instruct-pt")
model = AutoModelForCausalLM.from_pretrained("loizlabz/manaca-instruct-pt")

prompt = "### Instrução:\nCorrija gramaticalmente o texto: os documento foi enviado ontem\n\n### Resposta:\n"
inputs = tokenizer(prompt, return_tensors="pt")
# these are also the model's generation_config.json defaults, so plain model.generate(**inputs) behaves the same
output = model.generate(**inputs, max_new_tokens=256, do_sample=False, repetition_penalty=1.1)
print(tokenizer.decode(output[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

llama.cpp (GGUF)

# GGUF files are in this repo's gguf/ folder. Lowercase prompt, raw completion (see the note
# above) — llama.cpp build b10985 or newer
llama-completion \
  -m manaca-instruct-pt-Q4_K_M.gguf \
  -p "### instrução:\ncorrija gramaticalmente o texto: os documento foi enviado ontem\n\n### resposta:\n" \
  -n 256 -c 4096 --temp 0 --repeat-penalty 1.1 --no-display-prompt
# -> o documento foi enviado ontem.

Evaluation results

Fixed 104-prompt set, frozen since feature 001: 80 new-capability prompts (16 per category, data/eval/grupo_a_prompts.jsonl) and 24 general-Portuguese "forgetting-check" prompts (data/eval/grupo_b_prompts.jsonl). Classification is scored by rule (exact label match); every other row is graded by the author on a 1 / 0.5 / 0 rubric. Two metrics are reported because the partial-credit mean alone hid, in the qlora-v2 audit, that only 7 of 80 group-A answers were fully correct: mean (0.5 counts half) and full (share graded exactly 1).

Protocol. Rows marked blind were graded in feature 002's blind protocol (src/grading/review_cli.py --blind --interleave): the grader sees neither the model nor the run id, and answers from several runs are shuffled into one session. manaca-1b-base and qlora-v1 keep their feature-001 grades, marked earlier (non-blind) — they were graded per run with the run visible, so they are not directly comparable to the blind rows. Grammar was re-graded for every blind run in a single interleaved session with a written rubric after an inconsistency was found in qlora-v2's first session (its grammar mean fell from 0.438 to 0.094; the official release did not change). 35 of the 88 manual grades of the official release's blind file were given by an AI assistant (Claude Code) at the author's request, calibrated on the author's 53; every other blind grade is the author's. Details, ids and rubric: eval/results/adoption-decision.md. Every file cited on this card — grades, manifests, dataset report, seed provenance, specs — is in the public project repository: https://github.com/Wolfloiz/manaca-instruct.

Category	protocol	manaca-1b-base	qlora-v1	qlora-v2	manaca-instruct-pt (qlora-v3b, rp 1.1)	manaca-1b-instruct (official)
grammar_correction — mean / full	base, v1: earlier (non-blind); others: blind	0.000 / 0.000	0.156 / 0.000	0.094 / 0.000	0.375 / 0.188	0.125 / 0.000
classification — correct labels (of 16)	rule-based	0	4	4	12	4
rewriting — mean / full	as above	0.000 / 0.000	0.156 / 0.000	0.062 / 0.000	0.219 / 0.188	0.000 / 0.000
summarization — mean / full	as above	0.000 / 0.000	0.312 / 0.000	0.062 / 0.000	0.375 / 0.188	0.031 / 0.000
simplification — mean / full	as above	0.000 / 0.000	0.219 / 0.062	0.094 / 0.000	0.344 / 0.188	0.031 / 0.000
group A fully-correct rate (80 prompts)		0.000	0.062	0.050	0.300	0.050
Forgetting check (grupo_b, 24 prompts) — mean / full	as above	0.167 / 0.000	0.500 / 0.417	0.333 / 0.083	0.438 / 0.292	0.333 / 0.125

Means include partial credit (0.5); full_rate counts only answers graded 1.

Notes on the table:

• qlora-v2's classification collapsed to a single label: it answered "reclamação" on 12 of the 16 prompts, so its 4/16 is the accuracy of a constant answer. The published model uses all four labels (5 / 5 / 5 / 1 across the 16 prompts).
• The published model is the only run that satisfies the adoption rule written down before the runs (specs/002-data-quality-iteration/spec.md FR-020): ≥ 8/16 classification, group-A fully-correct rate ≥ 0.20, no category mean below qlora-v2's blind grades, and no more than a 10% relative drop on the forgetting check (it rose from 0.333 to 0.438). Two other candidates (qlora-v3a, data only; qlora-v3c, grammar rows doubled) and qlora-v3b under the previous generation settings were declined — per-run numbers and the attribution of every difference are in eval/results/final-table.md and eval/results/adoption-decision.md.
• Compared with qlora-v2 prompt by prompt, the published model improved / worsened on: classification 9 / 1, grammar 8 / 2, summarization 8 / 1, simplification 8 / 2, rewriting 4 / 1, forgetting check 8 / 5.
• The largest single effect was not training but generation: the same qlora-v3b adapter scored a group-A fully-correct rate of 0.188 with repetition_penalty 1.3 / no_repeat_ngram_size 3 and 0.300 with 1.1 / 0 (grammar 0.031 → 0.375). The 1.3 setting penalises every token already present in the prompt — labels and sentences that must be copied included.

Training data

3,752 examples (3,377 train / 375 validation), regenerated in feature 002 from the same public sources as feature 001 with stricter selection (word-boundary keyword matching, exclusion of forecast/code/sentence-building instructions from grammar, trivia/math from simplification, a degenerate-output filter, content deduplication, and a train/validation overlap check), plus a small set of author-reviewed synthetic examples. Counts and licenses from data/dataset_report.json:

source	license	rows kept	contribution
dominguesm/alpaca-data-pt-br	CC BY-NC-4.0	1,394	grammar correction, rewriting
dominguesm/Canarim-Instruct-PTBR-Dataset	CC BY-NC-4.0	2,161	summarization, simplification, classification
seed-llm (data/seed/)	author-reviewed synthetic	197	classification with the four evaluation labels

Synthetic seed disclosure. The 197 seed-llm rows are customer-service messages generated by Qwen2.5-7B-Instruct (Apache-2.0, run locally with llama.cpp), each reviewed by the author (147 kept as generated, 50 edited, 10 discarded of 207 candidates) and checked against the 16 classification evaluation prompts for reuse or paraphrase. They target the customer-service domain of the evaluation's classification task — the classification results above should be read with that in mind. Provenance, prompt and terms: data/seed/README.md.

Per category in the training split: classification 908, rewriting 906, summarization 897, grammar correction 355, simplification 311. A 30-row audit of the grammar rows found 20 genuine corrections (24 counting grammaticality judgments), below the 27/30 target — the remaining rows are grammar analysis/judgment exercises kept to stay above the 350-row category floor (data/dataset_report.md).

Known limitations

• The 70% per-category target of feature 001 (FR-004/SC-002) is not met by any model in the table. The published model's best category is classification (12/16 = 75% correct labels); every other category is between 0.22 and 0.44 mean with 3 of 16 answers fully correct. Treat the outputs as drafts to check, not results to trust.
• Summarization frequently invents content unrelated to the source text; grammar correction and simplification often paraphrase or change the meaning instead of making the minimal edit; rewriting sometimes answers the message instead of rewriting it. These are the dominant failure modes in the graded outputs (eval/results/qlora-v3b-rp11-blind.jsonl).
• The forgetting check did not drop (0.333 → 0.438 vs qlora-v2; 0.167 for the base model), but it is 24 prompts of general knowledge graded on a coarse rubric — it shows no aggregate loss, not the absence of forgetting in general.
• The base model shows near-chance performance on multiple-choice reasoning benchmarks (ARC-Challenge-PT ≈ 27%, per .specify/assessments/manaca-instruct-pt/research.md) — a real ceiling on tasks that need more than surface-level pattern completion, independent of fine-tuning quality.
• All models in the table were prompted with the same ### Instrução: / ### Resposta: template and the same generation settings. The official menezesbruno/manaca-1b-instruct release may have been trained on a different template (its card mentions an Alpaca-PT-style format), so its numbers may understate its performance under its own preferred prompting — a deliberate choice for a controlled comparison, not an oversight.
• Trained on ~3.8k examples from two public datasets plus 197 synthetic ones; not a large-scale or professionally curated dataset. The classification seed examples come from one domain (customer service).
• Measured throughput of the published GGUF files (benchmarks/rtx-5050.jsonl, rows with source_run_id: qlora-v3b, src/benchmark.py via llama-completion, greedy, 128-token budget on the grammar prompt above, which the model answers in a few tokens): on an RTX 5050 laptop GPU (8 GB), Q4_K_M 219 tok/s, 1.0 GB VRAM (1.06 GB file) and Q5_K_M 201 tok/s, 1.5 GB VRAM (1.23 GB file), no stall or crash. The second machine of the feature-001 plan (Dell G3, GTX 1050, 4 GB) was not measured and is not scheduled to be — no number is claimed for low-VRAM or CPU setups; the "~70 tok/s on a typical modern GPU" figure of that plan is a target, not a measurement.

License & attribution

CC BY-NC-4.0. This restriction is inherited from the training data: both alpaca-pt-br and Canarim-Instruct-PTBR-Dataset are CC BY-NC-4.0 because their instruction data derives from OpenAI model outputs, whose usage policy restricts training competing models commercially — the same reason the official menezesbruno/manaca-1b-instruct release is also CC BY-NC-4.0. The synthetic seed examples (Qwen2.5-7B-Instruct outputs, Apache-2.0) add no further restriction.

Base model: menezesbruno/manaca-1b-base, CC BY 4.0, © LNCC AI Institute / NII-LLM-jp.