Upload folder using huggingface_hub

.gitattributes

@@ -34,3 +34,9 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
 tokenizer.json filter=lfs diff=lfs merge=lfs -text
+assets/jay_chou_superman_cant_fly.mp3 filter=lfs diff=lfs merge=lfs -text
+assets/joe_hisaishi_summer.mp3 filter=lfs diff=lfs merge=lfs -text
+assets/llama4_hgf.png filter=lfs diff=lfs merge=lfs -text
+assets/mapo_tofu.mp4 filter=lfs diff=lfs merge=lfs -text
+assets/qwen2.5omni_hgf.png filter=lfs diff=lfs merge=lfs -text
+assets/zhajiang_noodle.mp4 filter=lfs diff=lfs merge=lfs -text

README.md

@@ -1,3 +1,132 @@
----
-license: mit
----
+---
+license: mit
+base_model:
+- Qwen/Qwen2.5-Omni-3B
+pipeline_tag: visual-document-retrieval
+library_name: peft
+---
+# Haon-Chen/e5-omni-3B
+
+**e5-omni-3B** is a powerful omni-modal embedding model built on [Qwen2.5-Omni-3B](https://huggingface.co/Qwen/Qwen2.5-Omni-3B).
+e5-omni-3B generates unified embeddings across text, images, audio, and video, enabling effective cross-modal retrieval for diverse applications. [Paper](https://arxiv.org/pdf/2505.02466v1).
+
+📝 Text   🖼️ Image   🎧 Audio   🎥 Video   🌐 Multilingual
+
+## Experimental Results
+Our model achieves SOTA performance on MMEB benchmark.
+<img width="900" alt="abs" src="https://raw.githubusercontent.com/haon-chen/mmE5/refs/heads/main/figures//exp_result.jpg">
+
+## Usage
+
+Below is an example we adapted from [Tevatron](https://huggingface.co/Tevatron/OmniEmbed-v0.1).
+
+```python
+# Import Library, Load Model and Processor
+import torch
+from transformers import AutoProcessor, Qwen2_5OmniThinkerForConditionalGeneration
+from qwen_omni_utils import process_mm_info
+
+device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
+
+processor = AutoProcessor.from_pretrained("Qwen/Qwen2.5-Omni-3B")
+model = Qwen2_5OmniThinkerForConditionalGeneration.from_pretrained(
+    "Haon-Chen/e5-omni-3B",
+    attn_implementation="flash_attention_2",
+    torch_dtype=torch.bfloat16
+).to(device).eval()
+
+processor.tokenizer.padding_side = "left"
+model.padding_side = "left"
+
+# Function to Encode Message
+def encode_message(message):
+    texts = processor.apply_chat_template(message, tokenize=False, add_generation_prompt=True)[0] + "<|endoftext|>"
+    audio_inputs, image_inputs, video_inputs = process_mm_info(message, use_audio_in_video=True)
+
+    inputs = processor(
+        text=texts,
+        audio=audio_inputs,
+        images=image_inputs,
+        videos=video_inputs,
+        return_tensors="pt",
+        padding="longest",
+    )
+    for k in inputs:
+        inputs[k] = inputs[k].to(device)
+
+    cache_position = torch.arange(0, inputs["input_ids"].shape[1], device=device)
+    inputs = model.prepare_inputs_for_generation(**inputs, use_cache=True, cache_position=cache_position)
+    model_outputs = model(**inputs, return_dict=True, output_hidden_states=True)
+
+    last_hidden_state = model_outputs.hidden_states[-1]
+    reps = last_hidden_state[:, -1]
+    reps = torch.nn.functional.normalize(reps, p=2, dim=-1)
+    return reps
+```
+
+### 🎬 Video Retrieval
+```python
+example_query = "Query: How to cook Mapo Tofu?"
+example_video_1 = "https://huggingface.co/Tevatron/OmniEmbed-v0.1/resolve/main/assets/mapo_tofu.mp4"
+example_video_2 = "https://huggingface.co/Tevatron/OmniEmbed-v0.1/resolve/main/assets/zhajiang_noodle.mp4"
+query = [{"role": "user", "content": [{"type": "text", "text": example_query}]}]
+video_1 = [{"role": "user", "content": [{"type": "video", "video": example_video_1}]}]
+video_2 = [{"role": "user", "content": [{"type": "video", "video": example_video_2}]}]
+
+sim1 = torch.cosine_similarity(encode_message(query), encode_message(video_1))
+sim2 = torch.cosine_similarity(encode_message(query), encode_message(video_2))
+
+print("Similarities:", sim1.item(), sim2.item())
+```
+
+### 🎵 Audio Retrieval
+```python
+example_query = "Query: A light piano piece"
+example_audio_1 = "https://huggingface.co/Tevatron/OmniEmbed-v0.1/resolve/main/assets/joe_hisaishi_summer.mp3"
+example_audio_2 = "https://huggingface.co/Tevatron/OmniEmbed-v0.1/resolve/main/assets/jay_chou_superman_cant_fly.mp3"
+query = [{"role": "user", "content": [{"type": "text", "text": example_query}]}]
+audio_1 = [{"role": "user", "content": [{"type": "audio", "audio": example_audio_1}]}]
+audio_2 = [{"role": "user", "content": [{"type": "audio", "audio": example_audio_2}]}]
+
+sim1 = torch.cosine_similarity(encode_message(query), encode_message(audio_1))
+sim2 = torch.cosine_similarity(encode_message(query), encode_message(audio_2))
+
+print("Similarities:", sim1.item(), sim2.item())
+```
+
+### 📈 Image Document Retrieval (Image, Chart, PDF)
+```python
+example_query = "Query: How many input modality does Qwen2.5-Omni support?"
+example_image_1 = "https://huggingface.co/Tevatron/OmniEmbed-v0.1/resolve/main/assets/qwen2.5omni_hgf.png"
+example_image_2 = "https://huggingface.co/Tevatron/OmniEmbed-v0.1/resolve/main/assets/llama4_hgf.png"
+query = [{"role": "user", "content": [{"type": "text", "text": example_query}]}]
+image_1 = [{"role": "user", "content": [{"type": "image", "image": example_image_1}]}]
+image_2 = [{"role": "user", "content": [{"type": "image", "image": example_image_2}]}]
+
+sim1 = torch.cosine_similarity(encode_message(query), encode_message(image_1))
+sim2 = torch.cosine_similarity(encode_message(query), encode_message(image_2))
+
+print("Similarities:", sim1.item(), sim2.item())
+```
+
+### 🌍 Multilingual Text Retrieval
+```python
+example_query = "Query: 氧气在空气中占比多少？"
+example_text_1 = "空气是指大气层中由不同气体和各类飘浮在其��的固体与液体颗粒（大气颗粒与气溶胶）所组成的气态混合物。地球大气层的空气主要由78.1%的氮气、20.9%氧气、0.9%的氩气和1~4%的水蒸气组成，其成分并不是固定的，随着高度、气压、温度的改变和对流情况不同，局部空气的组成比例也会改变。空气在大气层（特别是对流层）中的流动形成了风和曳流、气旋、龙卷等自然现象，而空气中飘浮的颗粒则形成了云、雾、霾和沙尘暴等短期天气情况。空气在海洋和陆地之间跨区域流动所承载的湿度和热能传导也是水循环和气候变率与变化的关键一环。"
+example_text_2 = "水（化学式：H2O）是一种无机化合物，在常温且无杂质中是无色[1]无味不导电的透明液体，也会通过蒸发产生气态的水蒸气（这种蒸发可以发生在任何温度下，同时取决于与空气接触的表面积和湿度差）。在标准大气压下，水的凝固点是0 °C（32 °F；273 K），沸点是100 °C（212 °F；373 K）。"
+query = [{"role": "user", "content": [{"type": "text", "text": example_query}]}]
+text_1 = [{"role": "user", "content": [{"type": "text", "text": example_text_1}]}]
+text_2 = [{"role": "user", "content": [{"type": "text", "text": example_text_2}]}]
+
+sim1 = torch.cosine_similarity(encode_message(query), encode_message(text_1))
+sim2 = torch.cosine_similarity(encode_message(query), encode_message(text_2))
+
+print("Similarities:", sim1.item(), sim2.item())
+```
+
+## Citation
+```
+@article{
+}
+```
+

assets/jay_chou_superman_cant_fly.mp3

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f54d08b9eecf4003b692daaff28f5387804a93a27deb1d5ce663e764a27de1d3
+size 4872832

assets/joe_hisaishi_summer.mp3

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:979c14285b90422965ecbce68c88b69dcb1e9328d78b5329c6a965eda0cfb938
+size 2638661

assets/mapo_tofu.mp4

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:23bd7a2a9a554bc09084cb74e584ca6129292073efcd2350f180e81975f96ec5
+size 5250889

assets/zhajiang_noodle.mp4

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:640b98fde893982dc0e866e72d537d9798a5a8432a6a7abc8f76d630c897a1b1
+size 3571831