Sazanami-AGC-310m

English | 日本語

A Japanese retrieval model trained from sbintuitions/modernbert-ja-310m that simultaneously learns late-interaction retrieval and document compression. It aggregates each document into 32 vectors, while using token-level representations on the query side. Similarity computation uses MaxSim.

Compared to Sazanami-ColBERT-310m, which retains vectors for all tokens, the volume of stored vectors is about 1/23, and similarity computation time is about 1/11. On tasks handling short documents, accuracy barely drops. On the other hand, on tasks searching for long documents among many candidates, accuracy decreases. It is a model whose suitability clearly depends on the use case.

1. Model Overview

Property Value
Base model sbintuitions/modernbert-ja-310m
Architecture Transformer with aggregation → token-level linear projection (768 → 128, no bias) → L2 normalization
Parameters About 315M (24,576 for importance estimation tokens, 98,304 for the projection layer)
Vector dimensions 128
Number of vectors per document Up to 32. For short documents with fewer than 32 non-padding tokens, up to that number. Averaging 30.31 on JaGovFaqs-22k
Number of vectors per query Equal to the number of non-padding tokens (same as the original model)
Recommended input length 256 tokens for queries / 1,024 tokens for documents + 32 importance estimation tokens
Scoring function MaxSim (sum over query tokens of the maximum similarity)
Prefix 検索クエリ: ("search query: ") for queries, 32 importance estimation tokens + 検索文書: ("search document: ") for documents
Main framework sentence-transformers 6.0.0 or later (MultiVectorEncoder)
Mixed precision during training bf16

Related Models

All four models were trained from sbintuitions/modernbert-ja-310m on the same training data. What differs is how many vectors represent a single document. Sazanami-Embed-310m is a single-vector model, trained under the same conditions for comparison.

Model Vectors per document When to choose
Sazanami-ColBERT-310m Same as the number of non-padding tokens in the text Highest accuracy when searching for long documents among many candidates
Sazanami-AGC-310m (this model) Up to 32 When you want to reduce storage size
Sazanami-MetaEmbed-310m Up to 64 (the number used can be reduced) When you want faster similarity computation
Sazanami-Embed-310m 1 (768 dimensions) Single-vector. When you want to minimize storage size and computational cost

2. Underlying Research

Late Interaction

Late interaction is a retrieval method that encodes queries and documents separately and matches them token by token only during similarity computation. It was introduced in ColBERT (Khattab and Zaharia, 2020, Santhanam et al., 2022). The score is MaxSim: for each query token, the maximum similarity with any document token is computed, and these are summed across all query tokens.

MaxSim(q,d)=∑imax⁡j  qi⋅dj \text{MaxSim}(q, d) = \sum_{i} \max_{j} \; q_i \cdot d_j

While it can preserve fine-grained matches, one needs to store as many vectors as there are tokens for each document, which increases storage size and similarity computation time. How to reduce this storage size is the main focus of this model.

AGC (Attention-Guided Compression)

AGC is a compression method that estimates which tokens are important from the model's own attention weights, and aggregates the remaining tokens around the important tokens as centers (Multi-Vector Index Compression in Any Modality, arXiv:2602.21202). While the original paper also covers images and video, this model applies it only to Japanese text.

The aggregation procedure is as follows:

  1. Place importance estimation tokens — 32 dedicated tokens are prepended to the document and encoded. The embeddings for these tokens do not expand the vocabulary table, but are provided as trainable parameters of the model.
  2. Read importance — In the final layer, the attention weights that these 32 tokens direct toward each token of the main text are averaged to determine per-token importance. Instead of storing attention weights across all layers, only the necessary rows from the final layer are computed.
  3. Select centers — The top 32 tokens by importance are chosen as cluster centers.
  4. Assign and aggregate the rest — Each token in the main text is assigned to the center with the closest representation, and a weighted average by importance is taken for each cluster. After that, projection and L2 normalization are applied.

Because the query side uses token-level representations just like the original model, the similarity computation mechanism itself does not change. Only the number of vectors in the index is reduced.

3. Usage

Load the model with MultiVectorEncoder in sentence-transformers 6.0.0 or later.

from sentence_transformers import MultiVectorEncoder

model = MultiVectorEncoder("mahiyama/Sazanami-AGC-310m", trust_remote_code=True)

# Queries: "Where can I get a certificate of residence?", "I want to know the enrollment procedure for National Health Insurance"
queries = [
    "住民票はどこで取れますか",
    "国民健康保険の加入手続きを知りたい",
]
# Documents: on obtaining a certificate of residence, National Health Insurance enrollment, and oversized garbage collection
documents = [
    "住民票の写しは、市区町村の窓口またはコンビニ交付サービスで取得できます。",
    "国民健康保険の加入手続きは、転入や退職から 14 日以内に市区町村の窓口で行ってください。",
    "粗大ごみの収集は、事前の申し込みが必要です。",
]

# Prefixes and importance estimation tokens are stored in the model and do not need to be added to the text
query_vectors = model.encode_query(queries)
document_vectors = model.encode_document(documents)

# The document side is aggregated into up to 32 vectors
print([tuple(v.shape) for v in document_vectors])

scores = model.similarity(query_vectors, document_vectors)
print(scores)

4. Training

Base Model

The model is initialized with sbintuitions/modernbert-ja-310m. It has undergone only masked language modeling pre-training and has not been trained for retrieval. Both the projection layer required for late-interaction retrieval and the importance estimation tokens used for aggregation were newly initialized. The importance estimation tokens start from values obtained by adding small random noise to the mean of the vocabulary embeddings.

In other words, this model carries out late-interaction training and document compression together in a single training run. Initializing from a trained late-interaction model was also tested, but showed no difference on this task suite.

Training Data

The same data as the original model was used. Two formats were used: n-tuples, pairing one query with one positive and five negatives, and pairs, pairing queries with only a positive.

Source Format Count
mahiyama/auto-wiki-qa n-tuples 100,000
mahiyama/mqa-ja n-tuples 100,000
mahiyama/mmarco-ja n-tuples 50,000
mahiyama/civicqa-ja (non-public) n-tuples 43,382
mahiyama/quiz-no-mori n-tuples 13,422
mahiyama/quiz-works n-tuples 12,502
mahiyama/amagasaki-qna n-tuples 11,069
mahiyama/miracl-retrieval n-tuples 4,603
mahiyama/mrtydi n-tuples 3,602
mahiyama/anlp-meeting-retrieval (title-abs) n-tuples 1,926
mahiyama/anlp-meeting-retrieval (title-intro) n-tuples 1,952
mahiyama/civicqa-ja (non-public) pairs 28,653
mahiyama/amagasaki-qna pairs 7,323
mahiyama/kosodate-faq-pairs-ja pairs 595
Total 379,029

mahiyama/civicqa-ja consists of questions and answers collected from FAQs published by Japanese government agencies and local governments. It is currently non-public.

Data used for evaluation was excluded from training. JaGovFaqs-22k was not used in training at all. Rows where the query after NFKC normalization and whitespace removal matched a query in the evaluation split of JaGovFaqs-22k, or where the positive after the same processing matched a document in that evaluation split, were excluded at each stage.

Training Procedure

The loss is the same as the original model. For n-tuples, the sum of a contrastive loss and a distillation loss is used. Both are computed from a single score computation.

  • Contrastive loss — Cross-entropy with all documents in the batch (its own 6 candidates and documents from other rows in the same batch) as candidates.
  • Distillation loss — KL divergence between the probability distribution of MaxSim scores over its own 6 candidates and the probability distribution of teacher scores. The teacher temperature is 2.0 and the student temperature is 1.0.

For pairs, a contrastive loss is used with documents from other rows in the batch as negatives.

Learning rates were separated by parameter type. This is to update the new importance estimation tokens and projection layer faster, while updating the pre-trained Transformer more slowly.

Setting Value
learning_rate (Transformer) 2e-5
learning_rate (importance estimation tokens) 1e-4
learning_rate (projection layer) 1e-4
num_train_epochs 1
per_device_train_batch_size 64
lr_scheduler_type linear
warmup_steps 5% of total
weight_decay 0.01
Input length during training 128 tokens for queries / 512 tokens for documents + 32 importance estimation tokens
Mixed precision bf16
batch_sampler NO_DUPLICATES
Gradient checkpointing Enabled
Teacher temperature / student temperature 2.0 / 1.0
Random seed 42

Training took about 4 hours 46 minutes (5,923 steps) on a single NVIDIA RTX PRO 6000 Blackwell Server Edition (96 GB) GPU, with peak GPU memory usage of 23.6 GB.

5. Training Code with Sentence Transformers

The training workflow is the same as Sazanami-ColBERT-310m, with the only difference being how the model is constructed. Only that difference is shown here.

AGCTransformer is a class in agc_modules.py, included in this repository. Inheriting from sentence-transformers' Transformer, it performs aggregation from attention weights of importance estimation tokens on the document side, while returning standard token-level representations on the query side.

import torch
from sentence_transformers import MultiVectorEncoder
from sentence_transformers.base.modules import Dense, Normalize

from agc_modules import AGCTransformer

BASE = "sbintuitions/modernbert-ja-310m"
BASE_DIR = "base-mve"
QUERY_PREFIX = "検索クエリ: "
DOCUMENT_PREFIX = "検索文書: "

# Load the raw base model once into MultiVectorEncoder and save it.
# This writes the folder with the projection layer (768 -> 128, no bias) initialized.
MultiVectorEncoder(
    BASE,
    model_kwargs={"torch_dtype": torch.float32},
    similarity_fn_name="maxsim",
    device="cpu",
).save_pretrained(BASE_DIR)

# Transformer that performs aggregation. Add the number of importance estimation tokens to the document input length.
transformer = AGCTransformer(
    BASE_DIR,
    num_probes=32,
    num_centers=32,
    model_kwargs={"torch_dtype": torch.float32},
    query_length=256,
    document_length=1024 + 32,
)

# For the document prefix, the sequence of importance estimation tokens comes first.
prompts = {
    "query": QUERY_PREFIX,
    "document": transformer.document_prompt_prefix + DOCUMENT_PREFIX,
}

model = MultiVectorEncoder(
    modules=[
        transformer,
        Dense.load(BASE_DIR, subfolder="1_Dense"),
        Normalize(module_input_name="token_embeddings"),
    ],
    prompts=prompts,
    similarity_fn_name="maxsim",
)

From this point on, the training code from the Sazanami-ColBERT-310m card can be used as-is. The loss (ContrastiveDistillLoss), enabling gradient checkpointing, specifying per-column prefixes, calling MultiVectorEncoderTrainer, and saving are all shared. The only two differences in training arguments are the following:

  • Add an entry for the importance estimation tokens to learning_rate_mapping: {r"0\.probe$": 1e-4, r"\.1\.linear\.": 1e-4}.
  • Add the 32 importance estimation tokens to the document side of max_length: {"query": 128, "document": 512 + 32}.

6. Evaluation Results

Among the 11 Retrieval tasks included in JMTEB(v2) in MTEB 2.21.8, three tasks differing in document length and candidate count were evaluated. The metric is NDCG@10. The setting uses 1,024 tokens for documents and 256 tokens for queries. For this model, 32 importance estimation tokens are added to the document side.

Benchmark Size

Number of documents, queries, and document lengths for the three tasks.

Task Documents Queries Avg. document tokens
Mintaka 1,592 2,312 10.0
JaGovFaqs 22,794 2,048 100.5
Jaqket 114,229 997 741.4

Average document tokens is the value calculated from the number of vectors actually generated by Sazanami-ColBERT-310m, which retains all tokens (1 vector = 1 token). It was measured with documents set to 1,024 tokens.

Retrieval Accuracy

Model Mintaka JaGovFaqs Jaqket
Sazanami-ColBERT-310m 0.18070 0.77986 0.77868
Sazanami-AGC-310m (this model) 0.17302 0.78513 0.72986
Sazanami-MetaEmbed-310m 0.22697 0.79316 0.72855
Sazanami-Embed-310m 0.21771 0.78883 0.69682

Storage Size and Similarity Computation Time

These are measured values on Jaqket with 114,229 documents. Documents were set to 1,024 tokens. Encoding is the process of converting document text into vectors, and is the total time to process all documents divided by 114,229 (as it is a batch process when building the index, it is averaged over GPU batch processing). Similarity computation time is the total time to process all 997 evaluation queries divided by 997.

Model Vectors per document Dimensions Storage ratio Encoding per document Similarity computation per query
Sazanami-ColBERT-310m 741.4 128 1 11.57 ms 43.03 ms
Sazanami-AGC-310m (this model) 32.0 128 About 1/23 12.22 ms 4.01 ms
Sazanami-MetaEmbed-310m 64.0 128 About 1/12 12.56 ms 0.70 ms
Sazanami-Embed-310m 1.0 768 About 1/124 12.83 ms 0.90 ms

7. Discussion of the Results

  • On tasks with short documents, accuracy does not drop even with compression. On JaGovFaqs, where documents average 100 tokens, this model's NDCG@10 is 0.78513, comparable to Sazanami-ColBERT-310m (0.77986), which retains vectors for all tokens. This can be interpreted as little information being lost even when aggregating into 32 vectors.
  • On tasks where documents are long and there are many candidates, accuracy decreases. On Jaqket, with an average document length of 741 tokens and 114,229 documents, NDCG@10 is 0.72986, which is 0.049 lower than Sazanami-ColBERT-310m (0.77868). Because an average of 741 vectors per document is reduced to 32, the parts that a query needs can get buried in the aggregation, and this shows up in the ranking.
  • Compared with the single-vector model, differences emerge only with long documents. On JaGovFaqs, where documents average 100 tokens, the difference from Sazanami-Embed-310m is within 0.004, and on Mintaka, where documents average 10 tokens, it is 0.045 lower. On the other hand, on Jaqket it is 0.033 higher. For corpora consisting mainly of short documents, a single vector is sufficient; differences with multi-vector representations appear when searching for long documents among many candidates. However, on Jaqket, Sazanami-ColBERT-310m outperforms this model by an additional 0.049, so if the primary use case involves long documents, an uncompressed configuration is more suitable.

8. License and Training Data Restrictions

The model weights are released under the MIT License. The base model, sbintuitions/modernbert-ja-310m, is also released under the MIT License.

The training data includes a Japanese translation of MS MARCO (mahiyama/mmarco-ja). MS MARCO is released for non-commercial research purposes, and its license for commercial use is not clear. If you use this model commercially, please check the MS MARCO terms of use at your own responsibility.

Part of the training data includes questions and answers collected from FAQs published by Japanese government agencies and local governments. This data is subject to the terms of use of each source and is not redistributed in this model's repository. If you use this model, please determine the suitability for your application at your own responsibility.

Downloads last month
42
Safetensors
Model size
0.3B params
Tensor type
F32
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for mahiyama/Sazanami-AGC-310m

Finetuned
(25)
this model

Datasets used to train mahiyama/Sazanami-AGC-310m

Collection including mahiyama/Sazanami-AGC-310m

Papers for mahiyama/Sazanami-AGC-310m