Haruka-EVA02-base-448

A general-purpose anime & illustration vision backbone (that also happens to work unreasonably well on real-life photos).

Compute (1 Node of 8x H100) graciously sponsored by alpindale from dphn.ai.

What's this?

So while I was waiting for my 3.7TB of Danbooru to download over the server for another round of haruka-tagger tuning, I played around with the idea of just using the backbone of Haruka (i.e. the pretrained Stage 1 eva02_base_patch14_448) for some unsupervised clustering fun.

Turns out, it seems quite good at it, so I've decided to release the pretrained EVA-02 backbone on its own for you guys to use as a feature extractor, clustering encoder, or drop-in base model for downstream classifiers.

Why use this over stock timm EVA-02?

Stock eva02_base_patch14_448.mim_in22k_ft_in22k_in1k is already a beast of a vision transformer (2D RoPE + SwiGLU + MIM-CLIP pretraining), but out of the box it has only ever seen ImageNet photo labels.

This checkpoint has been fine-tuned across 9.1 million deduplicated Danbooru illustrations and 98,726 tags (32.8k characters, 40.2k artists, 7.3k copyrights, 22.3k general tags) using Two-Sided Instance Repeat Factor Sampling (IRFS) and Asymmetric Loss:

• Fine-grained visual representations: Because it had to distinguish 40,000+ artist styles and 32,000+ characters down to 45-image long-tail tags, the 32x32 spatial patch tokens (1,024 x 768) capture brushwork, lineart, outfits, accessories, and facial traits rather than just coarse ImageNet nouns.
• Works on real-life photos oddly enough: Chalking this to "divine benevolence". Even after 5 epoches of danbooru, its 768-d embeddings still cleanly separate outfits, poses, objects, and people on real-world photos.

Usage

1. Loading via timm

Because the backbone is a standard eva02_base_patch14_448 architecture, timm can be diretly used like so:

import timm
import torch
import torch.nn.functional as F
from PIL import Image
import numpy as np

device = "cuda" if torch.cuda.is_available() else "cpu"

# Load the standalone backbone (num_classes=0 strips any classification head)
model = timm.create_model(
    "hf_hub:KaraKaraWitch/Haruka-EVA02-base-448",
    pretrained=True,
    num_classes=0,
    global_pool="",  # Returns full (B, 1025, 768) token sequence
).to(device).eval()

(Or if loading from a local model.safetensors file on top of timm's architecture definition):

import timm
from safetensors.torch import load_file

model = timm.create_model(
    "eva02_base_patch14_448.mim_in22k_ft_in22k_in1k",
    pretrained=False,
    num_classes=0,
    global_pool="",
)
state_dict = load_file("model.safetensors")
# Strip "backbone." prefix if loading from a full HarukaTagger checkpoint
state_dict = {k.removeprefix("backbone."): v for k, v in state_dict.items() if not k.startswith(("head.", "head_pool."))}
model.load_state_dict(state_dict, strict=True)
model.to(device).eval()

2. Preprocessing & Extracting Embeddings

During training, images were alpha-composited onto a white (255, 255, 255) canvas, resized with aspect-ratio preservation (BICUBIC) to 448x448, padded with white, and normalized with standard ImageNet mean/std:

IMAGENET_MEAN = np.array([0.485, 0.456, 0.406], dtype=np.float32).reshape(3, 1, 1)
IMAGENET_STD = np.array([0.229, 0.224, 0.225], dtype=np.float32).reshape(3, 1, 1)

def preprocess_448(img: Image.Image, target_size: int = 448) -> torch.Tensor:
    if img.mode != "RGB":
        img = img.convert("RGBA")
        bg = Image.new("RGBA", img.size, (255, 255, 255, 255))
        bg.alpha_composite(img)
        img = bg.convert("RGB")

    w, h = img.size
    if w != target_size or h != target_size:
        scale = min(target_size / w, target_size / h)
        nw, nh = max(1, int(w * scale)), max(1, int(h * scale))
        img = img.resize((nw, nh), Image.Resampling.BICUBIC)
        canvas = Image.new("RGB", (target_size, target_size), (255, 255, 255))
        canvas.paste(img, ((target_size - nw) // 2, (target_size - nh) // 2))
        img = canvas

    arr = np.asarray(img, dtype=np.float32).transpose(2, 0, 1) / 255.0
    arr = (arr - IMAGENET_MEAN) / IMAGENET_STD
    return torch.from_numpy(arr).unsqueeze(0)

@torch.no_grad()
def encode_image(img_path: str) -> dict[str, torch.Tensor]:
    """
    Extracts both:
      - "global": (1, 768) global mean-pooled vector
      - "windows": (9, 768) 3x3 overlapping spatial window vectors (16x16 token sub-grids)
    """
    x = preprocess_448(Image.open(img_path)).to(device)
    with torch.autocast(device_type="cuda", dtype=torch.bfloat16, enabled=(device == "cuda")):
        tokens = model.forward_features(x)[:, 1:, :].float()  # (1, 1024, 768) -> 32x32 spatial grid

    # 1. Global vector (good for single-subject clustering)
    global_emb = F.normalize(tokens.mean(dim=1), dim=-1)  # (1, 768)

    # 2. 3x3 Overlapping Spatial Window Pooling
    # Prevents a large background/poster/second character from diluting the subject by 50%
    grid = tokens.view(1, 32, 32, 768)
    win_embs = []
    for r in (0, 8, 16):
        for c in (0, 8, 16):
            win = grid[:, r : r + 16, c : c + 16, :].mean(dim=(1, 2))
            win_embs.append(F.normalize(win, dim=-1))
    windows_emb = torch.cat(win_embs, dim=0)  # (9, 768)

    return {"global": global_emb, "windows": windows_emb}

Note: We strongly encourage using Sliding-Window Token Pooling over Flat Global Pooling if you're looking for a specific object or person within the scene.

forward_features() provides you with the 32 x 32 spatial patch grid (which works out to 1024 tokens), slicing the grid into a 3 x 3 set of overlapping 16 x 16 tokens doesn't require additional compute and allows you to take the max cosine similarity across windows so that local subjects don't get diluted by other objects in the background.