Liiesl/lama-manga-onnx-quant

LaMa Manga ONNX — INT8 & FP16 Quant (Single-Input 512)

Weight-only quantized exports of LaMa (Large Mask Inpainting) for manga inpainting, derived from mayocream/lama-manga-onnx (lama-manga.onnx 197 MB, dual-input) and inspired by g-ronimo/lama quantization approach. Both variants are single-input [1,4,512,512] (0-2 = masked RGB 0-1 zeroed, 3 = mask 0/1) and keep all compute in FP32 via dequant/cast — no activation quantization, no calibration data, no FFC/BN modification needed for correctness.

File	Size	Inputs	Method	Quality vs FP32
lama-manga_int8.onnx	57 MB	input [1,4,512,512]	Per-channel asymmetric UINT8 — 222 Conv weights as UINT8 + DequantizeLinear(axis=0) before each Conv (scale [OC] FLOAT, zp [OC] UINT8)	max 60/255, 0.007% pixels >50 (symmetric INT8 was 185 / 70 bad)
lama-manga_fp16.onnx	109 MB	input [1,4,512,512]	Weight-only FP16 — 222 Conv weights as FLOAT16 + Cast(FLOAT16->FLOAT) before each Conv; BN/FFC untouched	max 3-11/255

Full-FP16 compute (lama_512_fp16.onnx, BN re-param var->1 + DFT 64x64 mats, ~5× on WebGPU) is available as reference in g-ronimo/lama and is not required here — the two files above already cover CPU/WASM (INT8) and size-halved (FP16) use.

Figure: Original / Masked (red) / FP32 4ch / INT8 w-only (~60 MB). Quant keeps surrounding context via 32 px LAMA_CONTEXT_PAD + mirror padding; 77 manga pages in calibration_manga_images/ used only for this figure, not for quantization.

Input / Output

Input

input [1,4,512,512] FLOAT
  0-2: masked RGB 0-1 (img * (1 - mask), hole zeroed)
  3:   mask 0 or 1

Pack via masked = img * (1 - mask) then input = concat(masked, mask, dim=1) (see inference.py:pack_input_4ch and inpaint/src/lib.rs:compose_inputs).

Output

output [1,3,512,512] FLOAT 0-1 (inpainted RGB)

Clip *255 to uint8.

Quantization Approach

INT8 follows g-ronimo/lama weight-only (avoids BatchNormalization running_var ~675k >> 65504 and FFC spectral branch blow-up):

for c in 0..OC-1:
  w_c = W[c]
  min_c, max_c = min(w_c), max(w_c)
  scale_c = (max_c - min_c) / 255.0  # clamp 1e-8
  zp_c    = clip(round(-min_c / scale_c), 0, 255)
  q_c     = clip(round(w_c / scale_c + zp_c), 0, 255)  # UINT8 stored
# runtime: w ≈ (q - zp) * scale via DequantizeLinear(axis=0)

Graph surgery dual→single: removes /Constant_1,F32 1.0 -> Sub -> Mul -> Concat (4 nodes), rewires /Concat_output_0 -> input, keeps /Constant int64 1 (shared by 36 Gather_6 FFC nodes). Numerical max diff 6.8e-07 vs original. No CalibrationDataReader, O(1) memory, deterministic. Source notebook ../quant.py (Kaggle 2×T4: INT8 on CPUExecutionProvider, full FP16 on CUDAExecutionProvider).

Usage

pip install onnxruntime Pillow numpy  # T4: onnxruntime-gpu for FP16 full
python inference.py --model lama-manga_int8.onnx --image in.jpg --mask mask.png --out out.png --provider cpu
python inference.py --model lama-manga_fp16.onnx --image in.jpg --mask mask.png --out out.png --provider cpu
# full FP16 reference (if present): --provider cuda

inference.py handles pack_input_4ch + ort.InferenceSession(..., providers=[...]).

Citation

LaMa — Resolution-robust Large Mask Inpainting with Fourier Convolutions
Suvorov et al., WACV 2022 — https://arxiv.org/abs/2109.07161
ONNX port: https://huggingface.co/g-ronimo/lama (Apache-2.0)
Manga FP32 source: https://huggingface.co/mayocream/lama-manga-onnx

Licensed Apache-2.0. INT8 scales FLOAT [OC], zp UINT8 [OC], axis=0.