crp4222/privaite-pii-minilm

PrivAiTe PII MiniLM

A small multilingual token classifier that finds personal data in text so it can be masked before the text leaves your machine, for example before a prompt is sent to an LLM. It was built for PrivAiTe, a local proxy that masks personal data in LLM traffic.

• 117M parameters, of which 96M are the frozen multilingual embedding table (MiniLM L12 H384, XLM-R vocabulary).
• 15 entity types, BIO tagging (31 labels).
• Ships in three forms: PyTorch (model.safetensors), ONNX for CPU (onnx/model.onnx, onnx/model_int8.onnx) and Core ML for the Apple Neural Engine (coreml/pilot_ane_256.mlpackage).
• On 6 of the 9 unseen test sets below it masks as many persons as the best of five open PII models (2.5 to 7 times its size on disk) or more, usually with higher precision, at 9 ms per 1,000 characters on an M1 Pro Neural Engine (23 ms on CPU with int8).

Masking replaces what the model finds; it is pseudonymisation, not anonymisation, and the model will miss some personal data (see Limitations).

Entity types

Type	Examples
PERSON	first names, last names, full names
CONTACT	email addresses, phone and fax numbers, URLs, user names
ADDRESS	street addresses, building numbers
LOCATION	cities, regions, countries, postcodes, coordinates
ORGANIZATION	companies and other organizations
DATE	dates and times
DATE_OF_BIRTH	dates of birth
AGE	ages
DEMOGRAPHIC	gender, ethnicity, religion, political view, occupation and similar
GOV_ID	national IDs, passport, driver licence, tax and social security numbers
FINANCIAL	IBAN, account and card numbers, routing and SWIFT codes
NETWORK_ID	IP and MAC addresses, device identifiers
RECORD_ID	customer, employee, medical record and similar identifiers, licence plates
SECRET	passwords, PINs, API keys, card security codes, cookies
OTHER	other personal data in the training sets (for example titles)

Usage

In PrivAiTe

Since PrivAiTe 0.8.0, two opt-in presets use this model in place of the Privacy Filter:

pii:
  preset: minilm      # ONNX on CPU, any platform
  # preset: ultrafast # Core ML on the Apple Neural Engine (macOS, pip install 'privaite[ane]')

Transformers

from transformers import pipeline

pii = pipeline("token-classification", model="crp4222/privaite-pii-minilm",
               aggregation_strategy="simple", stride=64)
pii("Hi, I'm Marie Dupont, call me at +33 6 12 34 56 78 or write to marie.dupont@example.fr.")
# PERSON 'Marie', PERSON 'Dupont', CONTACT '+33 6 12 34 56 78', CONTACT 'marie.dupont@example.fr'

The model was trained and evaluated on windows of 256 tokens with 64 tokens of overlap; the tokenizer is set to 256 so that stride=64 reproduces that on long text. Adjacent spans of the same type (a first and a last name) can be merged before masking.

ONNX (CPU)

import numpy as np, onnxruntime as ort
from huggingface_hub import snapshot_download
from transformers import AutoConfig, AutoTokenizer

path = snapshot_download("crp4222/privaite-pii-minilm", allow_patterns=["*.json", "onnx/model_int8.onnx"])
tok = AutoTokenizer.from_pretrained(path)
labels = AutoConfig.from_pretrained(path).id2label
sess = ort.InferenceSession(f"{path}/onnx/model_int8.onnx", providers=["CPUExecutionProvider"])

def find_pii(text):
    enc = tok(text, return_offsets_mapping=True, truncation=True, max_length=256, stride=64,
              return_overflowing_tokens=True)
    best = {}  # each token keeps the prediction of the window where it sits furthest from an edge
    for ids, offsets in zip(enc["input_ids"], enc["offset_mapping"]):
        a = np.array([ids], dtype=np.int64)
        pred = sess.run(None, {"input_ids": a, "attention_mask": np.ones_like(a)})[0][0].argmax(-1)
        for k, (s, e) in enumerate(offsets):
            dist = min(k, len(ids) - 1 - k)
            if s < e and ((s, e) not in best or dist > best[(s, e)][0]):
                best[(s, e)] = (dist, labels[int(pred[k])])
    spans, cur = [], None
    for (s, e), (_, label) in sorted(best.items()):
        if label == "O":
            cur = None
        elif cur and label == "I-" + cur[2]:
            cur[1] = e
        else:
            cur = [s, e, label[2:]]
            spans.append(cur)
    return spans

onnx/model.onnx (fp32) gives exactly the PyTorch predictions; onnx/model_int8.onnx is 4 times smaller and about twice as fast, with the same accuracy within a point (table below).

Core ML (Apple Neural Engine)

coreml/pilot_ane_256.mlpackage is the same encoder rewritten in Apple's layout for the Neural Engine, fp16, with a fixed input of 256 tokens: inputs input_ids and attention_mask (int32, shape 1 x 256, padded with the tokenizer's pad id and a zero mask), output logits (1 x 256 x 31). Use the same windows as above and pad each one:

import coremltools as ct
ml = ct.models.MLModel(f"{path}/coreml/pilot_ane_256.mlpackage", compute_units=ct.ComputeUnit.CPU_AND_NE)
ids_pad = np.full((1, 256), tok.pad_token_id, dtype=np.int32); mask = np.zeros((1, 256), dtype=np.int32)
ids_pad[0, :len(ids)], mask[0, :len(ids)] = ids, 1
logits = ml.predict({"input_ids": ids_pad, "attention_mask": mask})["logits"][0, :len(ids)]

Evaluation

All numbers come from the scorer of privaite-bench, with the competitors run as that benchmark runs them:

• Persons fully masked: share of person mentions whose every non-space character is masked (a half-masked name counts as missed).
• Precision: share of masked characters that fall inside an annotated span of any class. Where a corpus annotates only some classes (persons, places, organizations), correct masks of other personal data count against precision, for every system alike.
• Verdicts use a paired bootstrap of the difference (2,000 resamples, 95%): bold marks the best system, and a model is called ahead or behind only when the interval excludes zero.

Test sets never seen in training

Each suite was built from public datasets that were never used for training or model selection, before the training it was meant to judge, with every training document sharing a 10-word sequence with a test document removed. Lower-cased suites test text typed without capitals (chat, voice transcripts).

Persons fully masked (%)

Test set	This model	GLiNER PII	NVIDIA GLiNER-PII	GLiNER2 PII	Piiranha	OpenAI privacy-filter
WikiNER fr, GermEval de, UNER de and pt	93.6	90.6	86.9	82.6	36.3	15.8
Europarl de, en, es, it and SwissNER de, fr, it	97.0	95.0 (tie)	94.1	86.2	61.7	44.4
VoxPopuli, Few-NERD, UNER en and da (lower case)	89.2	87.9 (tie)	82.0	83.0	71.8	20.7
WikiGold (lower case)	92.1	88.4	84.6	76.4	73.9	19.9
CrossNER, UNER English PUD (lower case)	92.6	92.3 (tie)	91.1 (tie)	87.7	74.3	9.2
MASSIVE de, en, es, fr, it, pt (voice requests)	82.7	87.9	70.3	79.2	22.9	13.1
MTOP de, en, es, fr and SGD en, natural and lower case	97.9	98.9	91.5	88.7	32.2	43.4
TAB, European Court of Human Rights (en)	63.5	62.4 (tie)	74.6	57.1	45.0	52.6
PIILO, student essays (en)	91.6	90.9 (tie)	86.2	83.4	41.9	69.7

"(tie)" means no significant difference with this model. Against GLiNER PII, this model is ahead on two sets, tied on five and behind on two (short voice and assistant requests, where GLiNER PII also masks more text that is not personal data); NVIDIA GLiNER-PII is ahead on TAB.

Precision (%)

Test set	This model	GLiNER PII	NVIDIA GLiNER-PII	GLiNER2 PII	Piiranha	OpenAI privacy-filter
WikiNER fr, GermEval de, UNER de and pt	91.1	69.9	74.0	80.2	87.1	82.6
Europarl and SwissNER	82.9	64.0	69.8	79.7	98.7	75.4
VoxPopuli, Few-NERD, UNER en and da (lower case)	83.0	62.9	70.1	78.5	85.6	79.2
CrossNER, UNER English PUD (lower case)	91.1	76.3	79.4	83.1	91.0	89.0
MASSIVE (voice requests)	85.4	75.4	87.7	89.3	87.3	92.6
MTOP and SGD	95.0	95.0	97.7	98.5	98.3	95.3
TAB	94.9	67.8	86.1	92.2	98.5	98.5

High precision with low recall (Piiranha, privacy-filter on several sets) means a system masks little; the useful comparison is precision at a given level of persons masked.

privaite-bench, the corpora this model never saw (TAB, PIILO, Privy; 627 documents)

Here the measure covers every direct identifier (names, emails, phone numbers, IDs, card and account numbers, secrets, addresses), not only persons. The benchmark's 5 other corpora are left out: this model trained on their train splits.

	This model (ONNX fp32)	GLiNER PII	NVIDIA GLiNER-PII	GLiNER2 PII	Piiranha	OpenAI privacy-filter
Direct identifiers fully masked	82.7	72.3	81.3	73.6	42.9	64.8
Precision	86.4	52.6	63.8	65.5	77.6	68.6

Precision is on TAB and Privy (PIILO annotates only part of its personal data). On direct identifiers this model is tied with NVIDIA GLiNER-PII (difference +1.4, interval -1.3 to +4.1) and ahead of the others; on precision it is ahead of all five.

By build

The same nine test sets through the detector PrivAiTe ships (256-token windows, spans extended to whole words, which is why fp32 here can sit up to a point above the table above), persons fully masked (%):

Test set	ONNX fp32	ONNX int8	Core ML fp16 (Neural Engine)
WikiNER fr, GermEval de, UNER de and pt	93.9	92.7	93.9
Europarl and SwissNER	97.5	97.1	97.5
VoxPopuli, Few-NERD, UNER en and da (lower case)	90.1	88.7	90.2
WikiGold (lower case)	92.4	91.9	92.4
CrossNER, UNER English PUD (lower case)	93.6	92.5	93.6
MASSIVE (voice requests)	83.6	80.6	83.7
MTOP and SGD	98.0	97.6	98.0
TAB	63.5	63.0	63.5
PIILO	91.9	92.5	91.9

Core ML is tied with fp32 on all nine sets. int8 is behind on five of them (paired bootstrap), by 0.4 to 3 points, most on short voice requests.

Secrets in code

Samsung's CredData: real code from public repositories, credentials reviewed by hand, split by repository so 154 repositories (7,852 credentials, 4,649 values labeled not-a-credential) were never seen in training. A credential counts only if every non-space character is masked.

	Credentials fully masked	False alarms
This model alone	22.0%	15.6%
PrivAiTe minilm preset (model + rules, 0.8.1)	35.2%	25.1%
PrivAiTe onnx preset (openai/privacy-filter + rules)	59.8%	30.6%

Alone, the model is fine on passwords (61%) and tokens (56%), near zero on private keys, JWTs and generic keys. PrivAiTe's rules take care of private keys, JWTs and Basic headers; generic keys and secrets assigned under arbitrary names stay far behind the Privacy Filter (keys 7% against 34%, secrets 13% against 86%). For code full of keys, use the onnx preset.

Speed

Measured by the benchmark's own worker on all 1,607 documents of privaite-bench (a warm-up call first, then wall time around detection for each whole document, windows included), Apple M1 Pro, 8 CPU threads.

	Median per document	Median per 1,000 characters	Size
This model, Core ML (Neural Engine)	5.6 ms	9.3 ms	224 MB
This model, ONNX int8 (CPU)	14.8 ms	22.7 ms	113 MB
This model, ONNX fp32 (CPU)	29.5 ms	41.2 ms	448 MB
OpenAI privacy-filter	134.6 ms	186 ms	3.1 GB
Piiranha	113.1 ms	233 ms	1.1 GB
GLiNER PII	184.8 ms	258 ms	1.1 GB
GLiNER2 PII	218.3 ms	305 ms	1.2 GB
NVIDIA GLiNER-PII	413.5 ms	621 ms	1.7 GB

The competitors run on CPU through their own libraries, as in the benchmark.

Training

• Base: microsoft/Multilingual-MiniLM-L12-H384 (revision 6e8c1ec), word embeddings frozen.
• Stage 1: 2 epochs on real and synthetic annotated text (table below), batches of 256-token windows. Stage 2: 0.35 epoch at a learning rate of 3e-5 on the same data plus structured records (the same annotated values laid out as JSON, YAML, SQL, HTML, .env, CSV and log lines under field names, with look-alike fields that hold no personal data), so that values in agent traffic are read with their keys.
• 20% of documents lower-cased and 5% upper-cased, character by character.
• MultiCoNER sentences in which an earlier checkpoint found a person that the dataset left unannotated were dropped (MultiCoNER only annotates the mentions linked in Wikipedia).
• Checkpoint selected on the mean token F1 over the dev sets of every source, plus GermEval 2014 dev and a lower-cased copy of the multilingual dev sets.
• This checkpoint is training seed 20261028, the first of three seeds trained with this recipe. On the suites above the two others score from 2.6 points lower (MASSIVE) to 1.6 points higher (TAB); the decision to use this recipe was taken on the mean of the three seeds.

Data	Text	Languages	License
nvidia/Nemotron-PII	synthetic documents	en	CC BY 4.0
ai4privacy/pii-masking-openpii-1m	synthetic text	de, en, es, fr, it, nl, pt	CC BY 4.0
gretelai/gretel-pii-masking-en-v1	synthetic documents	en	Apache 2.0
gretelai/synthetic_pii_finance_multilingual	synthetic financial documents	de, en, es, fr, it, nl	Apache 2.0
MAPA (joelniklaus/mapa)	EU legal text	21 EU languages	CC BY 4.0
MultiCoNER v2	Wikipedia sentences, lower case	de, en, es, fr, it, pt, sv, uk	CC BY 4.0
NewsEye (HIPE-2022 subset)	historical newspapers	de, fi, fr, sv	CC BY 4.0
Europeana Newspapers NER	historical newspapers	de, fr, nl	CC0
LeNER-Br	court decisions	pt	MIT
SmartData	news, tweets	de	CC BY 4.0
MEDDOCAN	clinical cases	es	CC BY 4.0
NERDE	competition law documents	pt	CC BY 4.0
SEI	university administrative acts	pt	CC BY 4.0
InLegalNER	Indian court judgments	en	MIT
Broad Twitter Corpus	tweets	en	CC BY 4.0
WNUT-17	social media	en	CC BY 4.0

Revisions and attributions are in NOTICE. None of these licenses restricts commercial use; the CC BY 4.0 ones require the attribution given there.

Limitations

• Short voice or chat requests that name a person without context ("call maria", "how old is tom cruise"), especially without capitals: GLiNER PII finds 1 to 5 more names in 100 there.
• Long legal judgments (TAB): NVIDIA GLiNER-PII masks more persons.
• Persons, contacts and identifiers were evaluated most; dates and demographics follow the training data and were not tested on unseen data. Secrets were tested on unseen code (see Secrets in code): the model alone masks 22% of real credentials.
• Names are often tagged word by word (first name and last name as two spans); mask both, or merge adjacent spans of the same type.
• Do not rely on it as the only safeguard for sensitive data: review what it masks, and combine it with rules for structured identifiers where needed.

License

BSD 3-Clause (see LICENSE). The base model is MIT (Microsoft) and the training data licenses are listed in NOTICE.