Upload README.md with huggingface_hub

README.md

@@ -0,0 +1,95 @@
+---
+license: apache-2.0
+library_name: transformers
+pipeline_tag: image-text-to-text
+tags:
+- molmo
+- vision-language-model
+- olmo3
+base_model:
+- allenai/OLMo-3-1025-7B
+- amitha/clip-vit-b16-datacomp-medium
+---
+
+# amitha/molmo-clip-b16-olmo3
+
+A Molmo-style vision-language model: a **frozen CLIP ViT-B/16** vision encoder
+(pretrained on **DataComp-medium**, from [`amitha/clip-vit-b16-datacomp-medium`](https://huggingface.co/amitha/clip-vit-b16-datacomp-medium))
++ a trained multimodal connector + the **[OLMo-3-7B](https://huggingface.co/allenai/OLMo-3-1025-7B)**
+language model (`Olmo3ForCausalLM`).
+
+The vision encoder was **frozen** during training; only the connector (a SwiGLU image
+projector + a CLS projector) and the language model were trained, following the Molmo recipe.
+
+> **Vision weights are referenced, not stored.** This repo ships the connector + LLM weights
+> only. The vision tower is loaded at runtime from [`amitha/clip-vit-b16-datacomp-medium`](https://huggingface.co/amitha/clip-vit-b16-datacomp-medium),
+> so that repo must remain accessible. Loading requires `trust_remote_code=True`.
+
+## Checkpoints
+
+Training ran for 4 epochs. The **repo root is the final checkpoint** (`step14392`, 4 epochs).
+Three earlier checkpoints are available as subfolders:
+
+| Checkpoint | Subfolder | Notes |
+|---|---|---|
+| `step14392` | *(root)* | final (4 epochs) |
+| `step7196`  | `step7196`  | 2 epochs |
+| `step13000` | `step13000` | ~3.6 epochs |
+| `step14000` | `step14000` | ~3.9 epochs |
+
+Load an earlier checkpoint with `subfolder=`:
+
+```python
+model = AutoModelForImageTextToText.from_pretrained(
+    "amitha/molmo-clip-b16-olmo3", subfolder="step13000", trust_remote_code=True)
+```
+
+## Usage
+
+```python
+import torch, PIL.Image, requests
+from transformers import AutoModelForImageTextToText, AutoTokenizer
+from transformers import AutoImageProcessor, AutoProcessor
+
+repo = "amitha/molmo-clip-b16-olmo3"
+model = AutoModelForImageTextToText.from_pretrained(
+    repo, trust_remote_code=True, dtype=torch.float32).eval()
+processor = AutoProcessor.from_pretrained(repo, trust_remote_code=True)
+
+image = PIL.Image.open(requests.get(
+    "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/pipeline-cat-chonk.jpeg",
+    stream=True).raw).convert("RGB")
+inputs = processor(text="Describe this image in detail.", images=[image], return_tensors="pt")
+
+with torch.no_grad():
+    out = model.generate(**inputs, max_new_tokens=200, do_sample=False)
+print(processor.tokenizer.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))
+```
+
+### Prompt styles
+
+The model was trained with several caption/QA styles. The processor exposes an optional
+`style` argument (default: none) that prepends a `"{style}: "` prefix matching training:
+
+```python
+inputs = processor(text="Describe this image.", images=[image],
+                   style="long_caption", return_tensors="pt")
+```
+
+Known styles: `long_caption`, `transcript`, `user_qa`, `synthetic_qa`.
+
+## Architecture notes
+
+- **Image tokens:** single 224×224 crop, no pooling, CLS token included → 197 image tokens
+  (1 CLS + 196 patches) inserted into the text stream.
+- **LLM:** native `Olmo3ForCausalLM` (post-norm, YaRN RoPE), vocabulary padded to 100480;
+  the 128 image-placeholder logits are masked during generation.
+- **Image preprocessing:** resize so the short side is 224 (bicubic), center-crop 224,
+  normalize with OpenAI CLIP statistics.
+
+## Provenance
+
+Converted from native Molmo training checkpoints to the HuggingFace format with a converter
+verified to reproduce the original Molmo inference **bit-for-bit** (identical input ids and
+image token layout; vision features and logits matching to floating-point ordering noise;
+identical greedy generations).