Qwen3-VL PubMed Multimodal Assistant
This is a fine-tuned vision-language model designed for Medical Visual Question Answering (VQA) and multimodal document retrieval. It is optimized to process both clinical text and medical imaging queries within a Retrieval-Augmented Generation (RAG) architecture.
Model Details
Model Description
This model acts as the core reasoning engine for a highly specialized medical assistant. Fine-tuned over 8 hours on institute DGX hardware, it merges the vision-language capabilities of Qwen3-VL with domain-specific biomedical knowledge from PubMed. It is built to operate within a FAISS-backed RAG pipeline, allowing it to ground its diagnostic reasoning and medical image analysis in retrieved clinical literature.
- Developed by: Priyansh Saxena
- Model type: Multimodal Vision-Language Model (VLM)
- Language(s) (NLP): English
- License: Apache 2.0 (Inherited from Qwen)
- Finetuned from model:
Qwen/Qwen3-VL-4B-Instruct
Model Sources
Uses
Direct Use
The model is designed to be used via an API endpoint (e.g., LMDeploy or vLLM) as part of a larger medical RAG application. It directly takes user queries combining text and uploaded medical images (such as scans or diagrams), processes context retrieved from a vector database, and generates clinical summaries or answers.
Out-of-Scope Use
This model is a research and educational tool. It is strictly not intended for real-world clinical decision-making, patient diagnosis, or as a replacement for professional medical advice. The model may hallucinate or misinterpret complex medical imaging.
Bias, Risks, and Limitations
While fine-tuned on PubMed, the model's knowledge relies on the biases present in the retrieved literature. It may struggle with highly ambiguous imaging or rare medical conditions not well-represented in the fine-tuning dataset. Users should exercise extreme caution and verify all outputs against professional medical standards.
Recommendations
Users should implement robust guardrails and human-in-the-loop verification when deploying this model in any health-adjacent context.
How to Get Started with the Model
This model is intended to be served using lmdeploy or vLLM due to its multimodal architecture and RAG integration.
To serve the merged adapter weights locally:
lmdeploy serve api_server ./qwen-pubmed-merged --model-name pubmed-adapter --server-port 8000