silv4souz4/florence-2-large-coreai-w8a16

Model Card for Florence-2-large — Apple Core AI, int8 weights / fp16 activations (W8A16)

Format conversion of Microsoft's microsoft/Florence-2-large to Apple Core AI (.aimodel) with 8-bit integer weights and fp16 activations, for on-device inference on Apple silicon. Tested on macOS 27.2 (Apple M5); not tested on iOS and not yet run from Swift (validated with the Core AI Python runtime).

Model Details

Model Description

Florence-2 is a sequence-to-sequence vision foundation model: a DaViT image encoder feeds a BART-style encoder-decoder that answers task prompts (captioning, object detection, OCR, grounding, region description…) as text, with coordinates encoded as location tokens. This repository contains the whole model as one Core AI asset with three functions (vision, prompt encoder, one-token decoder step); the generation loop, tokenizer and post-processing run in the host app.

• Original model developed by: Microsoft (Xiao et al., 2023).
• Conversion by: silv4souz4 (independent; not affiliated with or endorsed by Microsoft or Apple).
• Model type: vision-language encoder-decoder (DaViT + BART), 0.77 B parameters.
• Base model: microsoft/Florence-2-large, revision 21a599d4.
• Relation to base: quantized (int8 weights, fp16 activations); architecture re-authored for Core AI with identical math.
• License: MIT, inherited from Florence-2.

Model Sources

• Original model: https://huggingface.co/microsoft/Florence-2-large
• Paper: https://arxiv.org/abs/2311.06242

Uses

Direct Use

On-device image captioning (short, detailed, paragraph), object detection, dense region captioning, region proposals, OCR with or without regions, phrase grounding and other Florence-2 task prompts, on Apple silicon without a server.

Downstream Use

Auto-tagging and alt text in photo apps, pre-labelling boxes for datasets, reading text in images (for example as a second opinion next to a dedicated OCR), region descriptions for search.

Out-of-Scope Use

• Free-form chat or instructions outside Florence-2's task prompts (it is not a chat model).
• Safety-critical decisions without human review (for example legal identification from OCR).
• Surveillance or identification of individuals without consent, or any use that violates applicable law.

Bias, Risks, and Limitations

• Inherits the limitations and biases of Florence-2 (web-scale training data, English output, hallucinated details in long captions, errors on small text and dense scenes). See the Florence-2 paper and model card.
• Int8 weights change some decisions: on our check 6 of 32 greedy sequences differed from the fp32 original (late in the sequence or by one extra token, with equivalent text and boxes). Long outputs drift more than short ones.
• The original generation config uses beam search (3 beams); our reference loop is greedy, so outputs may differ from the transformers pipeline defaults even without quantization.
• fp16 numerics vary across compute units and OS releases; validate on the device and OS you ship. iOS is untested.
• Debug information embedded in the asset contains build-time file paths of the converting machine; it has no effect on inference.

Recommendations

Use the greedy loop exactly as described below (forced first token and trigram blocking); check outputs for consequential uses; compare against the original on your own images and target device.

How to Get Started with the Model

Download the whole repository (asset, tokenizer, host reference, config.json and checksums):

hf download silv4souz4/florence-2-large-coreai-w8a16 --local-dir florence-2-large-coreai-w8a16

config.json is a machine-readable summary of the functions, tensor shapes, preprocessing, generation settings and quantization below (it is also the file the Hub uses to count downloads of this repository).

File	Size	Notes
florence2_large_w8a16_g32.aimodel	906 MB	portable Core AI asset, 3 functions
tokenizer.json	1.7 MB	BART tokenizer + the 1,024 Florence tokens (<loc_0>…<loc_999>, task tokens) in the original processor's order
host_reference/gerar.py	4 KB	reference host loop: greedy decoding, trigram blocking, byte-level BPE decoding (comments in Portuguese)

P = 64 prompt tokens, T = 1,024 cache positions; float16 unless noted.

Function	Inputs	Outputs
florence2_vision	pixel_values [1,3,768,768]	image_features [1,577,1024]
florence2_encoder_P64	image_features, input_ids int32 [1,64], attention_mask int32 [1,64]	kv_cruzado [12,2,16,641,64] (cross-attention K/V), mascara_cruzada [1,641]
florence2_decoder_T1024	token int32 [1,1], posicao int32 [1], kv_passado [12,2,16,1024,64], mascara_passado [1,1024], kv_cruzado, mascara_cruzada	logits [1,51289], kv_novo [12,2,16,1,64]

1. Image: RGB, resize to 768×768 (bicubic), /255, ImageNet mean [0.485, 0.456, 0.406] and std [0.229, 0.224, 0.225], NCHW → florence2_vision once per image.
2. Prompt: expand the task token to its text as the original processor does, e.g. <CAPTION> → "What does the image describe?", <DETAILED_CAPTION> → "Describe in detail what is shown in the image.", <OD> → "Locate the objects with category name in the image.", <OCR> → "What is the text in the image?" (full list in processing_florence2.py of the original repo). Tokenize with tokenizer.json (adds <s>=0 … </s>=2), pad right with <pad>=1 to 64, attention_mask 0 on padding → florence2_encoder_P64 once per prompt; its outputs stay fixed during generation.
3. Decode loop: keep kv_passado (zeros) and mascara_passado (−1e4 everywhere). Step t = 0 feeds token = 2 at posicao 0; after each step write kv_novo into kv_passado[..., t, :] and set mascara_passado[0, t] = 0. The first generated token is forced to <s> = 0; afterwards take the argmax with trigram repetition blocked (no_repeat_ngram_size=3, part of Florence-2's generation config — without it the model loops on <s>). Stop at </s> or 1,024 positions.
4. Output: drop <s>/</s>; boxes are <loc_x1><loc_y1><loc_x2><loc_y2> on a 1,000-step grid over the image.

from coreai.runtime import AIModel, NDArray, SpecializationOptions
m = await AIModel.load("florence2_large_w8a16_g32.aimodel", SpecializationOptions.default())
vision = m.load_function("florence2_vision")
encoder = m.load_function("florence2_encoder_P64")
decoder = m.load_function("florence2_decoder_T1024")
# host_reference/gerar.py: gulosa(executar, pixel_values, prompt_ids) runs steps 2–3; executar(name, inputs) maps
# "visao" / "codificador" / "decodificador" to the three functions above and returns numpy arrays

Ahead-of-time compilation (.aimodelc) — tutorial

The asset here is a portable .aimodel. Apple's documentation, Compiling Core AI models ahead of time, explains that every Core AI model must be specialized to the device before inference; the coreai-build tool moves the most expensive part of that specialization, model compilation, to your build machine, so the model loads faster in your app. It produces one .aimodelc per device architecture; you ship (or download) only the one that matches the device. According to Apple, AOT compilation targets devices that support Apple Intelligence (iPhone/iPad with A17 Pro or later, Mac with M1 or later, Apple Vision Pro); it is not supported on tvOS and watchOS.

1. Install the Metal Toolchain (Xcode > Settings > Components > Metal Toolchain, or):

xcodebuild -downloadComponent MetalToolchain

2. Compile (all architectures of a platform, or one with --architecture):

xcrun coreai-build compile florence2_large_w8a16_g32.aimodel --platform macOS --min-deployment-version 27.0 --output compiled/
xcrun coreai-build compile florence2_large_w8a16_g32.aimodel --platform iOS --min-deployment-version 27.0 --output compiled/

Output files are named <asset>.<arch>.aimodelc (for example .h17g for the Apple M5 used here); each works on any OS at or above --min-deployment-version. Run coreai-build compile --help for --preferred-compute and the other options.

3. Load the matching bundle — the same API as for .aimodel:

let arch = AIModel.deviceArchitectureName
let url = compiledDir.appending(path: "florence2_large_w8a16_g32.\(arch).aimodelc")
let model = try await AIModel(contentsOf: url, options: .default)

Apple recommends hosting the per-architecture bundles and downloading only the matching one (for example with the BackgroundAssets framework). Some on-device specialization still remains after AOT, depending on the model and compute units.

What we measured (Apple M5, macOS 27.2, Swift runtime, florence2_vision, first load with an empty specialization cache, two rounds):

Step	Portable .aimodel	AOT .aimodelc
Compile on the Mac (coreai-build, h17g)	—	4.7 s
First load (model + function), cold cache	2.2–2.3 s	0.9–1.0 s
Later loads (cache warm)	0.2 s	0.2 s

• AOT changes load time, not the math: on the 32-case check below the M5 AOT build gave the same sequences and metrics as the portable asset. It is not an inference speed-up.
• The benefit is largest on iPhone, where on-device specialization is slow; the cache is also rebuilt after every OS update, so a portable asset pays the cold cost again.

Training Details

Training Data

No training or fine-tuning. Weights come from the official Florence-2-large checkpoint, trained by Microsoft on FLD-5B and other data as described in the Florence-2 paper.

Training Procedure (conversion)

1. Official model.safetensors (revision 21a599d4) loaded into a re-authored PyTorch model (DaViT + BART) with identical math; the fp32 re-authoring reproduced the transformers original on 32 of 32 greedy sequences.
2. Recipe chosen by layer analysis (fake-quant on 8 cases, then the real runtime): per-channel int8 gave 37.9 dB on the logits / 96.6 % top-1, groups of 64 gave 41.5 dB / 98.3 %, groups of 32 gave 41.8 dB / 98.5 %. Most of the error came from the vision tower, and within it from the depthwise 3×3 convolutions, so those stay in fp16.
3. Final recipe: int8 symmetric weights in blocks of 32 on every nn.Linear (vision, encoder, decoder, lm_head); embedding table int8 per row; DaViT convolutions in fp16; fp16 activations (LayerNorm scaled by its maximum, masks −1e4); decoder K/V cache passed as explicit inputs/outputs.
4. Converted with coreai-torch 0.4.3 / coreai-opt 0.3.0 (PyTorch 2.13) to one portable .aimodel with three functions.

A runtime pitfall found on the way (worth knowing for other conversions): int8 block-quantized nn.Linear layers whose input has a batch dimension > 1 (the window-attention linears, [256 windows, 144, C]) produced wrong results in the Core AI runtime (1.3 dB against 39 dB in PyTorch), while per-channel was fine; applying those linears on the token tensor [1, N, C] and partitioning into windows only around the attention fixed it with identical math.

Evaluation

Testing Data, Factors & Metrics

8 real images (5 from COCO val2017 and 3 private photos of vehicles with license plates) × 4 tasks (<CAPTION>, <DETAILED_CAPTION>, <OD>, <OCR>) = 32 cases. Reference: the original model in transformers, fp32, greedy decoding with trigram blocking. Candidate: this asset in the Core AI runtime (Python), Apple M5, macOS 27.2.

Results

Metric	W8A16 (this)
Identical greedy output sequence	26 / 32
Next-token agreement under teacher forcing	99.45 %
<OD> box IoU against the reference	0.999
Image-feature SNR	38.0 dB
AOT h17g and Neural-Engine-preferred runs	identical to the portable asset

Descriptive timings from the Python loop (not a controlled benchmark): vision ~0.45–0.55 s per image, prompt encoder ~28 ms, decoder ~17 ms per token.

Summary

Outputs equivalent to the fp32 original on this small check, with occasional late divergences in long sequences.

Environmental Impact

No training. Conversion and evaluation ran locally on one Apple M5 laptop over a few hours; negligible compared with the original training.

Technical Specifications

Compute Infrastructure

• Hardware: Apple M5 (Mac17,3), 24 GB unified memory.
• Software: macOS 27.2, PyTorch 2.13, coreai-torch 0.4.3, coreai-opt 0.3.0, coreai-core 1.0.0b3, coreai-build 3605.5.4.

Citation

If you use these assets, please cite Florence-2:

@article{xiao2023florence,
  title={Florence-2: Advancing a unified representation for a variety of vision tasks},
  author={Xiao, Bin and Wu, Haiping and Xu, Weijian and Dai, Xiyang and Hu, Houdong and Lu, Yumao and Zeng, Michael and Liu, Ce and Yuan, Lu},
  journal={arXiv preprint arXiv:2311.06242},
  year={2023}
}

The evaluation used COCO val2017 images (images subject to the Flickr Terms of Use):

@inproceedings{lin2014coco,
  title={Microsoft COCO: Common Objects in Context},
  author={Lin, Tsung-Yi and Maire, Michael and Belongie, Serge and Hays, James and Perona, Pietro and Ramanan, Deva and Doll{\'a}r, Piotr and Zitnick, C. Lawrence},
  booktitle={European Conference on Computer Vision (ECCV)},
  year={2014}
}

License and Attribution

• License: MIT — full text in LICENSE, identical to the license of microsoft/Florence-2-large.
• Copyright of the original model: Microsoft Corporation.
• Modifications: re-authoring for Core AI, int8 weight quantization, conversion to the .aimodel format and an extended tokenizer.json in the original processor's token order; no retraining. See NOTICE.
• "Apple", "Core AI" and "Microsoft" are trademarks of their respective owners; their use here is descriptive only.

Model Card Authors

silv4souz4

Model Card Contact

Please open a discussion in the Community tab of this repository.