Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering

Abstract: Problems at the intersection of vision and language are of significant importance both as challenging research questions and for the rich set of applications they enable. However, inherent structure in our world and bias in our language tend to be a simpler signal for learning than visual modalities, resulting in models that ignore visual information, leading to an inflated sense of their capability. We propose to counter these language priors for the task of Visual Question Answering (VQA) and make vision (the V in VQA) matter! Specifically, we balance the popular VQA dataset by collecting complementary images such that every question in our balanced dataset is associated with not just a single image, but rather a pair of similar images that result in two different answers to the question. Our dataset is by construction more balanced than the original VQA dataset and has approximately twice the number of image-question pairs. Our complete balanced dataset is publicly available at www.visualqa.org as part of the 2nd iteration of the Visual Question Answering Dataset and Challenge (VQA v2.0). We further benchmark a number of state-of-art VQA models on our balanced dataset. All models perform significantly worse on our balanced dataset, suggesting that these models have indeed learned to exploit language priors. This finding provides the first concrete empirical evidence for what seems to be a qualitative sense among practitioners. Finally, our data collection protocol for identifying complementary images enables us to develop a novel interpretable model, which in addition to providing an answer to the given (image, question) pair, also provides a counter-example based explanation. Specifically, it identifies an image that is similar to the original image, but it believes has a different answer to the same question. This can help in building trust for machines among their users.

IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoning

Abstract: Current visual question answering (VQA) tasks mainly consider answering human-annotated questions for natural images. However, aside from natural images, abstract diagrams with semantic richness are still understudied in visual understanding and reasoning research. In this work, we introduce a new challenge of Icon Question Answering (IconQA) with the goal of answering a question in an icon image context. We release IconQA, a large-scale dataset that consists of 107,439 questions and three sub-tasks: multi-image-choice, multi-text-choice, and filling-in-the-blank. The IconQA dataset is inspired by real-world diagram word problems that highlight the importance of abstract diagram understanding and comprehensive cognitive reasoning. Thus, IconQA requires not only perception skills like object recognition and text understanding, but also diverse cognitive reasoning skills, such as geometric reasoning, commonsense reasoning, and arithmetic reasoning. To facilitate potential IconQA models to learn semantic representations for icon images, we further release an icon dataset Icon645 which contains 645,687 colored icons on 377 classes. We conduct extensive user studies and blind experiments and reproduce a wide range of advanced VQA methods to benchmark the IconQA task. Also, we develop a strong IconQA baseline Patch-TRM that applies a pyramid cross-modal Transformer with input diagram embeddings pre-trained on the icon dataset. IconQA and Icon645 are available at https://iconqa.github.io.

Visual Spatial Reasoning

Abstract: Spatial relations are a basic part of human cognition. However, they are expressed in natural language in a variety of ways, and previous work has suggested that current vision-and-language models (VLMs) struggle to capture relational information. In this paper, we present Visual Spatial Reasoning (VSR), a dataset containing more than 10k natural text-image pairs with 66 types of spatial relations in English (such as: under, in front of, and facing). While using a seemingly simple annotation format, we show how the dataset includes challenging linguistic phenomena, such as varying reference frames. We demonstrate a large gap between human and model performance: the human ceiling is above 95%, while state-of-the-art models only achieve around 70%. We observe that VLMs' by-relation performances have little correlation with the number of training examples and the tested models are in general incapable of recognising relations concerning the orientations of objects.

OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge

Abstract: Visual Question Answering (VQA) in its ideal form lets us study reasoning in the joint space of vision and language and serves as a proxy for the AI task of scene understanding. However, most VQA benchmarks to date are focused on questions such as simple counting, visual attributes, and object detection that do not require reasoning or knowledge beyond what is in the image. In this paper, we address the task of knowledge-based visual question answering and provide a benchmark, called OK-VQA, where the image content is not sufficient to answer the questions, encouraging methods that rely on external knowledge resources. Our new dataset includes more than 14,000 questions that require external knowledge to answer. We show that the performance of the state-of-the-art VQA models degrades drastically in this new setting. Our analysis shows that our knowledge-based VQA task is diverse, difficult, and large compared to previous knowledge-based VQA datasets. We hope that this dataset enables researchers to open up new avenues for research in this domain.

Generation and Comprehension of Unambiguous Object Descriptions

Abstract: We propose a method that can generate an unambiguous description (known as a referring expression) of a specific object or region in an image, and which can also comprehend or interpret such an expression to infer which object is being described. We show that our method outperforms previous methods that generate descriptions of objects without taking into account other potentially ambiguous objects in the scene. Our model is inspired by recent successes of deep learning methods for image captioning, but while image captioning is difficult to evaluate, our task allows for easy objective evaluation. We also present a new large-scale dataset for referring expressions, based on MSCOCO. We have released the dataset and a toolbox for visualization and evaluation, see https://github.com/ mjhucla/Google_Refexp_toolbox.

Point and Ask: Incorporating Pointing into Visual Question Answering

Abstract: Visual Question Answering (VQA) has become one of the key benchmarks of visual recognition progress. Multiple VQA extensions have been explored to better simulate real-world settings: different question formulations, changing training and test distributions, conversational consistency in dialogues, and explanation-based answering. In this work, we further expand this space by considering visual questions that include a spatial point of reference. Pointing is a nearly universal gesture among humans, and real-world VQA is likely to involve a gesture towards the target region. Concretely, we (1) introduce and motivate point-input questions as an extension of VQA, (2) define three novel classes of questions within this space, and (3) for each class, introduce both a benchmark dataset and a series of baseline models to handle its unique challenges. There are two key distinctions from prior work. First, we explicitly design the benchmarks to require the point input, i.e., we ensure that the visual question cannot be answered accurately without the spatial reference. Second, we explicitly explore the more realistic point spatial input rather than the standard but unnatural bounding box input. Through our exploration we uncover and address several visual recognition challenges, including the ability to infer human intent, reason both locally and globally about the image, and effectively combine visual, language and spatial inputs. Code is available at: https://github.com/princetonvisualai/pointingqa .

Visual7W: Grounded Question Answering in Images

Abstract: We have seen great progress in basic perceptual tasks such as object recognition and detection. However, AI models still fail to match humans in high-level vision tasks due to the lack of capacities for deeper reasoning. Recently the new task of visual question answering (QA) has been proposed to evaluate a model's capacity for deep image understanding. Previous works have established a loose, global association between QA sentences and images. However, many questions and answers, in practice, relate to local regions in the images. We establish a semantic link between textual descriptions and image regions by object-level grounding. It enables a new type of QA with visual answers, in addition to textual answers used in previous work. We study the visual QA tasks in a grounded setting with a large collection of 7W multiple-choice QA pairs. Furthermore, we evaluate human performance and several baseline models on the QA tasks. Finally, we propose a novel LSTM model with spatial attention to tackle the 7W QA tasks.

MMBench: Is Your Multi-modal Model an All-around Player?

Abstract: Large vision-language models have recently achieved remarkable progress, exhibiting great perception and reasoning abilities concerning visual information. However, how to effectively evaluate these large vision-language models remains a major obstacle, hindering future model development. Traditional benchmarks like VQAv2 or COCO Caption provide quantitative performance measurements but suffer from a lack of fine-grained ability assessment and non-robust evaluation metrics. Recent subjective benchmarks, such as OwlEval, offer comprehensive evaluations of a model's abilities by incorporating human labor, but they are not scalable and display significant bias. In response to these challenges, we propose MMBench, a novel multi-modality benchmark. MMBench methodically develops a comprehensive evaluation pipeline, primarily comprised of two elements. The first element is a meticulously curated dataset that surpasses existing similar benchmarks in terms of the number and variety of evaluation questions and abilities. The second element introduces a novel CircularEval strategy and incorporates the use of ChatGPT. This implementation is designed to convert free-form predictions into pre-defined choices, thereby facilitating a more robust evaluation of the model's predictions. MMBench is a systematically-designed objective benchmark for robustly evaluating the various abilities of vision-language models. We hope MMBench will assist the research community in better evaluating their models and encourage future advancements in this domain. Project page: https://opencompass.org.cn/mmbench.

Summary of evaluation datasets.

This paper presents a novel framework for enhancing fine-grained image perception ability of MLLMs with RC tasks. The framework includes a method for constructing an instruction tuning dataset by converting annotations from existing datasets into diverse RC tasks. Additionally, a self-consistent bootstrapping method is proposed to extend object annotations to referring-expression-bounding-box pairs, enabling the acquisition of more instruction tuning data at a low cost. The visual encoder is tuned in a parameter-efficient way to gain fine-grained image understanding ability.
With fewer trainable parameters and less training data, our method achieves superior performance on both conventional multi-modal reasoning tasks and RC tasks compared with recently proposed models.

Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou.
Qwen-vl: A frontier large vision-language model with versatile abilities.
arXiv preprint arXiv:2308.12966, 2023.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei.
Language models are few-shot learners.
In NeurIPS, 2020.
Chi Chen, Ruoyu Qin, Fuwen Luo, Xiaoyue Mi, Peng Li, Maosong Sun, and Yang Liu.
Position-enhanced visual instruction tuning for multimodal large language models, 2023a.
Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao.
Shikra: Unleashing multimodal llm's referential dialogue magic.
arXiv preprint arXiv:2306.15195, 2023b.
Ting Chen, Saurabh Saxena, Lala Li, David J Fleet, and Geoffrey Hinton.
Pix2seq: A language modeling framework for object detection.
arXiv preprint arXiv:2109.10852, 2021.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi.
Instructblip: Towards general-purpose vision-language models with instruction tuning, 2023.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.
An image is worth 16x16 words: Transformers for image recognition at scale.
In ICLR. OpenReview.net, 2021.
Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh.
Making the v in vqa matter: Elevating the role of image understanding in visual question answering.
In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6904–6913, 2017.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick.
Mask r-cnn.
In Proceedings of the IEEE international conference on computer vision, pages 2961–2969, 2017.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly.
Parameter-efficient transfer learning for nlp.
In International Conference on Machine Learning, pages 2790–2799. PMLR, 2019.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen.
Lora: Low-rank adaptation of large language models.
In ICLR. OpenReview.net, 2022.
Drew A Hudson and Christopher D Manning.
Gqa: A new dataset for real-world visual reasoning and compositional question answering.
In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6700–6709, 2019.
Woojeong Jin, Subhabrata Mukherjee, Yu Cheng, Yelong Shen, Weizhu Chen, Ahmed Hassan Awadallah, Damien Jose, and Xiang Ren.
Grill: Grounded vision-language pre-training via aligning text and image regions.
arXiv preprint arXiv:2305.14676, 2023.
Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg.
Referitgame: Referring to objects in photographs of natural scenes.
In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 787–798, 2014.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al.
Visual genome: Connecting language and vision using crowdsourced dense image annotations.
International journal of computer vision, 123:32–73, 2017.
Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan.
Seed-bench: Benchmarking multimodal llms with generative comprehension.
arXiv preprint arXiv:2307.16125, 2023a.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi.
BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models.
CoRR, abs/2301.12597, 2023b.
Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, et al.
Grounded language-image pre-training.
In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10965–10975, 2022.
Fangyu Liu, Guy Emerson, and Nigel Collier.
Visual spatial reasoning.
Transactions of the Association for Computational Linguistics, 11:635–651, 2023a.
Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang.
Aligning large multi-modal model with robust instruction tuning.
arXiv preprint arXiv:2306.14565, 2023b.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee.
Improved baselines with visual instruction tuning.
arXiv preprint arXiv:2310.03744, 2023c.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee.
Visual instruction tuning.
CoRR, abs/2304.08485, 2023d.
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al.
Mmbench: Is your multi-modal model an all-around player?
arXiv preprint arXiv:2307.06281, 2023e.
Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mottaghi, and Aniruddha Kembhavi.
Unified-io: A unified model for vision, language, and multi-modal tasks.
arXiv preprint arXiv:2206.08916, 2022.
Pan Lu, Liang Qiu, Jiaqi Chen, Tony Xia, Yizhou Zhao, Wei Zhang, Zhou Yu, Xiaodan Liang, and Song-Chun Zhu.
Iconqa: A new benchmark for abstract diagram understanding and visual language reasoning.
arXiv preprint arXiv:2110.13214, 2021.
Arjun Mani, Nobline Yoo, Will Hinthorn, and Olga Russakovsky.
Point and ask: Incorporating pointing into visual question answering.
arXiv preprint arXiv:2011.13681, 2020.
Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L Yuille, and Kevin Murphy.
Generation and comprehension of unambiguous object descriptions.
In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 11–20, 2016.
Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi.
Ok-vqa: A visual question answering benchmark requiring external knowledge.
In Proceedings of the IEEE/cvf conference on computer vision and pattern recognition, pages 3195–3204, 2019.
Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei.
Kosmos-2: Grounding multimodal large language models to the world.
arXiv preprint arXiv:2306.14824, 2023.
Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik.
Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models.
In Proceedings of the IEEE international conference on computer vision, pages 2641–2649, 2015.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.
Learning transferable visual models from natural language supervision.
In International conference on machine learning, pages 8748–8763. PMLR, 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu.
Exploring the limits of transfer learning with a unified text-to-text transformer.
J. Mach. Learn. Res., 21:140:1–140:67, 2020.
Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilic, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, Jonathan Tow, Alexander M. Rush, Stella Biderman, Albert Webson, Pawan Sasanka Ammanamanchi, Thomas Wang, Benoît Sagot, Niklas Muennighoff, Albert Villanova del Moral, Olatunji Ruwase, Rachel Bawden, Stas Bekman, Angelina McMillan-Major, Iz Beltagy, Huu Nguyen, Lucile Saulnier, Samson Tan, Pedro Ortiz Suarez, Victor Sanh, Hugo Laurençon, Yacine Jernite, Julien Launay, Margaret Mitchell, Colin Raffel, Aaron Gokaslan, Adi Simhi, Aitor Soroa, Alham Fikri Aji, Amit Alfassy, Anna Rogers, Ariel Kreisberg Nitzav, Canwen Xu, Chenghao Mou, Chris Emezue, Christopher Klamm, Colin Leong, Daniel van Strien, David Ifeoluwa Adelani, and et al.
BLOOM: A 176b-parameter open-access multilingual language model.
CoRR, abs/2211.05100, 2022.
Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi.
A-okvqa: A benchmark for visual question answering using world knowledge.
In European Conference on Computer Vision, pages 146–162. Springer, 2022.
Shuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng, Gang Yu, Xiangyu Zhang, Jing Li, and Jian Sun.
Objects365: A large-scale, high-quality dataset for object detection.
In Proceedings of the IEEE/CVF international conference on computer vision, pages 8430–8439, 2019.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut.
Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning.
In ACL (1), pages 2556–2565. Association for Computational Linguistics, 2018.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample.
Llama: Open and efficient foundation language models.
CoRR, abs/2302.13971, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al.
Llama 2: Open foundation and fine-tuned chat models.
arXiv preprint arXiv:2307.09288, 2023b.
Vicuna.
Vicuna: An open chatbot impressing gpt-4.
https://github.com/lm-sys/FastChat, 2023.
Guangzhi Wang, Yixiao Ge, Xiaohan Ding, Mohan Kankanhalli, and Ying Shan.
What makes for good visual tokenizers for large language models?, 2023.
Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang.
Ofa: Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework.
In International Conference on Machine Learning, pages 23318–23340. PMLR, 2022.
Size Wu, Wenwei Zhang, Lumin Xu, Sheng Jin, Xiangtai Li, Wentao Liu, and Chen Change Loy.
Clipself: Vision transformer distills itself for open-vocabulary dense prediction.
arXiv preprint arXiv:2310.01403, 2023.
Canwen Xu, Daya Guo, Nan Duan, and Julian J. McAuley.
Baize: An open-source chat model with parameter-efficient tuning on self-chat data.
CoRR, abs/2304.01196, 2023.
Zhengyuan Yang, Linjie Li, Kevin Lin, Jianfeng Wang, Chung-Ching Lin, Zicheng Liu, and Lijuan Wang.
The dawn of lmms: Preliminary explorations with gpt-4v (ision).
arXiv preprint arXiv:2309.17421, 9, 2023.
Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al.
mplug-owl: Modularization empowers large language models with multimodality.
arXiv preprint arXiv:2304.14178, 2023.
Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen.
A survey on multimodal large language models.
arXiv preprint arXiv:2306.13549, 2023.
Shilong Zhang, Peize Sun, Shoufa Chen, Min Xiao, Wenqi Shao, Wenwei Zhang, Kai Chen, and Ping Luo.
Gpt4roi: Instruction tuning large language model on region-of-interest.
arXiv preprint arXiv:2307.03601, 2023.
Liang Zhao, En Yu, Zheng Ge, Jinrong Yang, Haoran Wei, Hongyu Zhou, Jianjian Sun, Yuang Peng, Runpei Dong, Chunrui Han, et al.
Chatspot: Bootstrapping multimodal llms via precise referring instruction tuning.
arXiv preprint arXiv:2307.09474, 2023.
Yiwu Zhong, Jianwei Yang, Pengchuan Zhang, Chunyuan Li, Noel Codella, Liunian Harold Li, Luowei Zhou, Xiyang Dai, Lu Yuan, Yin Li, et al.
Regionclip: Region-based language-image pretraining.
In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16793–16803, 2022.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny.
Minigpt-4: Enhancing vision-language understanding with advanced large language models, 2023.
Yuke Zhu, Oliver Groth, Michael Bernstein, and Li Fei-Fei.
Visual7w: Grounded question answering in images.
In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4995–5004, 2016.

Discussion of Limitations
Our method relies on the LLM. Therefore, it has some shortcomings from the LLM, such as bias or unfair response, and hallucination. Like most MLLMs with RC ability, we also find that our model is not good at object detection in complex scenarios, e.g., identifying multiple tiny objects in the image. It may be because the input resolution of the image is low. However, increasing the input resolution of the image is not a trivial task. Many efforts are still needed to deal with such tasks.

Statistics of Training and Evaluation Datasets
The statistics for the training datasets and evaluation datasets are summarized in Table {{table:d13b3745-554f-470b-97d1-235add3bc9a3}}  and Table {{table:99a5b68b-548b-428f-93d8-1c780ad6f290}} , respectively. We also provide an overview of the generated referring-expression-bounding-box pairs in Object365 in Table {{table:a09c24e9-79b4-4d94-9bb6-ec62f8ac0e1f}} .
To ensure the quality of data, we apply certain filters during preprocessing. Firstly, we exclude images containing more than 15 objects. Moreover, for the purpose of bounding box description bootstrapping, we only consider objects that cover an area of more than 2,000 pixels. As a result, our dataset comprises 1,063,034 images, with a total of 4,961,822 generated referring-expression-bounding-box pairs. To further enhance the reliability of our dataset, we perform a self-consistent method that filters out 2,528,619 low-quality referring-expression-bounding-box pairs.
Statistics of training datasets. PT and IT denote the pre-training and instruction tuning stage, respectively.
Stage
Dataset
Data Number
PT
LLaVA-CC3M-Pretrain-595K {{cite:6471ce1}}
595K
IT
LLaVA-158K {{cite:6471ce1}}
158K
IT
VQAv2 {{cite:940d361}}
83K
IT
A-OKVQA {{cite:5997a2b}}
17K
IT
Visual Genome {{cite:32ecb07}}
108K
IT
Flickr30K {{cite:7c38168}}
30K
IT
Object365{{formula:f1c8762e-fc18-4d3b-b6f8-bcf15d5d9c7c}}  {{cite:f640305}}
1M
{{table:1d32fde5-8101-4da2-b395-18de2131b20e}}{{table:d13b3745-554f-470b-97d1-235add3bc9a3}}Summary of evaluation datasets.
pxDataset
Split
Metric
VQAv2 {{cite:940d361}}
test-dev
VQA Score
IconQA {{cite:70663fc}}
multi-text-choice test
Accuracy
VSR {{cite:c617d94}}
zero-shot test
VQA Score
OK-VQA {{cite:578b8fe}}
val
VQA Score
GQA {{cite:9289280}}
test-dev
VQA Score
RefCOCO {{cite:3b6ed3f}}
val & testA & testB
Accuracy
RefCOCO+ {{cite:3b6ed3f}}
val & testA & testB
Accuracy
RefCOCOg {{cite:174a69a}}
val & test
Accuracy
PointQA-Local {{cite:19762f4}}
test-dev
VQA Score
PointQA-LookTwice {{cite:19762f4}}
test
VQA Score
Visual-7W {{cite:b859506}}
which box test
Accuracy
MMBench {{cite:10f46ae}}
test
Accuracy
{{table:34b46dd7-35ef-4398-8683-448f52cc89a2}}{{table:99a5b68b-548b-428f-93d8-1c780ad6f290}}Statistics of generated referring-expression-bounding-box pairs in Object365.
Images
Referring-expressions
Avg Expression Length
1,063,034
2,433,203
3.6
{{table:96f63b67-fada-4307-95ac-e704a5931f63}}{{table:a09c24e9-79b4-4d94-9bb6-ec62f8ac0e1f}}
More experimental results
The results that leverages a more advanced LLM LLaMA-2 {{cite:9e14458}} are shown in Table {{table:13059139-114b-4e75-92b8-c5df7fb31a4a}}  and Table {{table:f91754c3-93ba-483c-aec4-ba36dc76b30c}} . We can observe an improvement in performance on most datasets.
Similar to MMBench, SEED-Bench {{cite:fcd4cec}} is a recently proposed benchmark to evaluate the abilities of MLLMs on various evaluation dimensions including Scene Understanding (SU), Instance Identity (IId), Instance Attribute (IA), Instance Location (IL), Instance Counting (IC), Spatial Relation (SR), Instance Interaction (IIn), Visual Reasoning (VR), and Text Recognition (TR). We further conduct experiment on this benchmark. The results are shown in Table {{table:209df709-b8d9-405c-9b83-63b3f9b5dd0b}} . Pink also exhibits the best performance among compared MLLMs.
Our model has significant advantages in IIn, SR, and IL tasks, which require strong fine-grained image perception ability of MLLMs. These results further validate the effectiveness of our method to enhance the fine-grained perception ability of MLLMs.
Results on the conventional multi-modal reasoning tasks with LLaMA-2 {{cite:9e14458}} as the LLM. {{formula:14f8ba2d-c23f-452b-8474-bb0d85f526f5}}  denotes the training set of corresponding dataset is included.
pxModels
VQAv2
IconQA
VSR
OK-VQA
GQA
Pink
78.7{{formula:c6cd8ca6-3a47-4020-9ace-700429941ef2}}
47.8
66.3
59.5
52.6
Pink-LLaMA-2
78.8{{formula:324f4461-c9d0-4be1-ae0e-85fce2915784}}
49.1
67.9
60.2
52.0
{{table:43eda050-39e6-4ca9-9cd8-2fa4109a62ee}}{{table:13059139-114b-4e75-92b8-c5df7fb31a4a}}Fine-tuning results on RC tasks with LLaMA-2 {{cite:9e14458}} as the LLM.
px2*Models
RefCOCO
RefCOCO+
RefCOCOg
2*Visual-7W
2*LookTwice

val
testA
testB
val
testA
testB
val
test

Pink
88.3
91.7
84.0
81.4
87.5
73.7
83.7
83.7
85.1
73.5
Pink-LLaMA-2
89.0
92.1
84.6
82.6
88.3
74.5
83.8
84.4
85.1
73.8
{{table:d6826fc7-4d3a-40db-ab00-0b68a272f660}}{{table:f91754c3-93ba-483c-aec4-ba36dc76b30c}}Results on the image split of SEED-Bench {{cite:fcd4cec}}.
Models
Overall
SU
IId
IA
IL
IC
SR
IIn
VR
TR
Kosmos-2 {{cite:663ffe9}}
54.4
63.4
57.1
58.5
44.0
41.4
37.9
55.7
60.7
25.9
Instruct-BLIP {{cite:174502e}}
58.8
60.2
58.9
65.6
43.6
57.2
40.3
52.6
47.7
43.5
mPlug-Owl {{cite:e35d93d}}
37.9
49.7
45.3
32.5
36.7
27.3
32.7
44.3
54.7
28.8
Qwen-VL {{cite:094455d}}
62.3
71.2
66.4
67.7
53.5
44.8
43.8
62.9
74.9
51.2
gray!30 Pink
66.2
73.1
69.1
69.1
60.5
55.2
51.0
76.3
70.4
58.8
gray!30 Pink-LLaMA-2
67.0
75.2
70.1
70.1
63.3
53.8
50.2
69.1
74.3
50.0
{{table:49444809-a0c4-4219-ba27-f86c07520d97}}{{table:209df709-b8d9-405c-9b83-63b3f9b5dd0b}}
Qualitative Analysis
We show qualitative results on different types of vision-language tasks and RC tasks. The results of mPlug-Owl {{cite:e35d93d}}, Shikra {{cite:310443e}} and Qwen-VL {{cite:094455d}} are from their official online demos.
Knowledge-intensive QA. Fig {{figure:a7209c24-dce5-415c-8c15-8a49b6a42904}}  depicts an example of this task, where the model is required to identify an album cover and provide an introduction for the album. mPlug-Owl and Qwen-VL can identify the album {{formula:7cb36ba6-3bc0-4709-83c8-0bc83643c1d2}}  correctly but begins to make hallucination when introducing this album. Some introductions provide by mPlug-Owl do not align with the actual facts associated with the album.
The answer of Shikra focuses on the content of the image, indicating a poor instruction-following ability of Shikra.
In contrast, our model not only correctly identifies the album but also delivers an accurate and relevant introduction, showcasing its strong instruction-following capability.
A comparison of knowledge-intensive QA. We mark the hallucination part in redred.
{{figure:a7209c24-dce5-415c-8c15-8a49b6a42904}}Fine-grained QA. As shown in Fig. {{figure:9b4c3fe1-3326-4726-8010-4c9d5904c900}} , mPlug-Owl fails to output the correct number of people in the image. It may be because that its lack of RC ability hinders its fine-grained understanding of images.
Despite Shikra has RC ability, it also fails to give a correct answer. This result can validate that trained with limited RC tasks Shikra struggles to achieve a fine-grained perception of images.
In contrast, our model makes a correct answer to the question, demonstrating its strong capability in fine-grained image perception.
A comparison of fine-grained QA.
{{figure:9b4c3fe1-3326-4726-8010-4c9d5904c900}}Referential comprehension reasoning. We show a case of RC reasoning in Fig. {{figure:9647d80c-4d9a-4119-9ad9-b453061f7ff7}} . In this case, our model accurately locates Steve Nash. Shikra seems to be unaware of who is Steve Nash. Surprisingly, Shikra's output coordinates point to the face of Dirk Nowitzki. We also need to point out the result of Shikra is generated using the instruction template that is identical to the one used during training. Shikra shows limited instruction-following ability when using a template that is different from the one used during training.
Our model also shows a ability for multi-round conversation. It can understand what “him” refers to in the second question. As Shikra makes a wrong response for the first question, in the second question, we give the correct coordinates manually. However, Shikra fails to follow the instructions and begins to hallucination.
Qwen-VL fails to locate Steve Nash correctly. In the multi-round conversation, it also gives a wrong response.
These results can further validate the RC ability of our model.
A comparison of referential comprehension reasoning. We mark the hallucination part in redred.
{{figure:9647d80c-4d9a-4119-9ad9-b453061f7ff7}}More qualitative results of our model.
{{figure:e6eac037-1e93-4883-b8d8-f41a44bba947}}{{figure:85ccec24-65f6-4545-9ef5-74b4e6721ca3}}{{figure:2ef0da15-799e-4a46-9252-a40769ea5605}}Qualitative results of our model on multi-round conversation.
{{figure:38432b35-abac-4f31-baa8-6cb5f1a25fd2}}Qualitative results of our model on multi-round conversation with correction.
{{figure:ff979ded-84fa-4432-a1e0-066eb94363db}}Qualitative results of generated referring-expression-bounding-box pairs. The solid rectangle represents the ground-truth bounding box. The dashed rectangle represents the bounding box generated by our model with visual grounding according to the generated description. *{{figure:3bd8c352-45bb-4371-89f0-7ebfa55a4eec}}
{{figure:0ab2627e-69cb-4b13-a185-8953b8e8956b}}More qualitative results. More qualitative results of our model are shown in Fig. {{figure:2ef0da15-799e-4a46-9252-a40769ea5605}} . Our model demonstrates the ability to generate unique descriptions with contextual information when provided with coordinates of a specific area. For instance, instead of simply outputting “helmet”, our model uses “woman's” features for differentiation.
Moreover, our model successfully locates items mentioned in descriptions that require outside knowledge. For example, it can correctly identify what can be drunk in the image.
In the last two cases, our model not only provides correct answers but also locates the mentioned items in the image. This ability can achieve more applications.
We also present qualitative results of our model on multi-round conversation about RC in Fig. {{figure:38432b35-abac-4f31-baa8-6cb5f1a25fd2}}  and Fig. {{figure:ff979ded-84fa-4432-a1e0-066eb94363db}} . Our method can understand the complex referential relationships in the dialogue context, e.g., it, her and this instrument.
As shown in Fig. {{figure:ff979ded-84fa-4432-a1e0-066eb94363db}} , although the model initially gives the wrong answer about what is holding by the man, when it is told that its answer is incorrect, the model can make corrections and provide the correct answer. This result can further demonstrate the promising instruction-following ability of our model.
Additionally, these qualitative results highlight that the integration of RC ability significantly expands the range of tasks our model can successfully handle, thereby broadening its application potential.
Qualitative results of generated referring-expression-bounding-box pairs.
We illustrate some examples of generated referring-expression-bounding-box pairs in Fig. {{figure:0ab2627e-69cb-4b13-a185-8953b8e8956b}} . For each object in the image, the bounding box description bootstrapping method can generate a description related to that object. Most of generated descriptions are correct. However, some generated descriptions exhibit incorrect or ambiguous descriptions that fail to uniquely identify an object. As shown in Fig. {{figure:0ab2627e-69cb-4b13-a185-8953b8e8956b}}  (a) and (b), when leveraging visual grounding to locate these descriptions in the image, IOU between the predicted bounding box and the ground-truth bounding box is low. Our self-consistent filtering method can effectively filter them out. These results can further validate the effectiveness of our self-consistent bootstrapping method to extend the object annotations to high-quality referring-expression-bounding-box pairs.

Instruction Templates
Instruction templates of visual relation reasoning.
We list all the instruction templates of visual relation reasoning task below. For the Task2, we use different instruction templates to ask the model output different contents (coordinates, class name, or both coordinates and class name).
[title=Instruction templates of visual relation reasoning-Task1]
What is the relation between <subject> and <object>?
Describe the relation between <subject> and <object>?
Assist me in finding the relation between <subject> and <object> in the photo.
In the given image, could you find and tell me the relation between <subject> and <object>?
I need help identifying the relation between <subject> and <object>. Can you point it out in this image?
What is the relation between <subject> and <object> in this picture?
Could you describe the relation between <subject> and <object> in this image?
I'm having trouble identifying the relation between <subject> and <object>. Could you clarify it for me in this image?
Can you help me understand the relationship between <subject> and <object> in this image?
I'm trying to understand the relation between <subject> and <object>. Can you help me by describing it?
I need some assistance in identifying the relation between <subject> and <object> in this image.
In this image, can you find and describe the relation between <subject> and <object> for me?
Could you please explain the relation between <subject> and <object>.

[title=Instruction templates of visual relation reasoning-Task2#1]
Assist me in locating the position of all the objects <relation> the <subject>?
I want to know the coordinates of all the objects <relation> the <subject>?
Detect all the objects have a relationship <relation> with the <subject> and output there locations.
There are some objects that are <relation> the <subject>. Could you tell me there locations?
Identify all the objects that have a relationship <relation> with the <subject>. Where are they located?
Please locate all the objects that are <relation> the <subject> and provide their coordinates.
Find all the objects that have a relation of <relation> with the <subject>. Can you give me their positions?
Point out the objects that are <relation> the <subject>. Where can I find them?
I need to locate all the objects that are <relation> the <subject>. Can you assist me with this task?
Could you help me find all the objects that have a relation of <relation> with the <subject>? Please provide their locations.
Please detect all the objects that are <relation> the <subject>. Output their positions.
Identify and provide the coordinates of all objects that are <relation> the <subject>.
Find the objects that have a relation of <relation> with the <subject>. Where are they situated?
What objects have the relation of <relation> with the <subject>? Could you locate them for me?
Can you help me locate all the objects that are <relation> the <subject> and give me their positions?
Output the positions of all objects that have a relation of <relation> with the <subject>.
Identify the objects that are <relation> the <subject>. Where are they located?
Please locate all the objects that are <relation> the <subject> and provide their positions.

[title=Instruction templates of visual relation reasoning-Task2#2]
Assist me in identifying the categories of all the objects <relation> by the <subject>?
Detect all the objects <relation> by the <subject> and output there categories, respectively.
There are some objects that are <relation> the <subject>. Could you tell me there categories?
I want to know the categories of all the objects whose relation is <relation> with the <subject>?
Identify the object categories that are <relation> the <subject>.
Find all objects that are related to <subject> using the relationship <relation>, and categorize them.
Your task is to recognize and classify all objects that are <relation> by the <subject>.
Please determine the categories of all objects that are <relation> by the <subject>.
Can you identify the categories of objects <relation> the <subject>?
Your job is to identify all objects that <relation> the <subject> and list their categories.
Detect and categorize all objects that are <relation> the <subject>.
I need you to determine the categories of all objects that <relation> the <subject>.
Identify and classify all objects that are <relation> the <subject>.
Please identify the categories of all objects that are <relation> the <subject>.
Please help me identify the object categories whose relationship is <relation> with <subject>.
Recognize and categorize all objects that are related to <subject> using the relationship <relation>.
I need you to categorize all objects that is related to <subject> with relationship as <relation>.

[title=Instruction templates of visual relation reasoning-Task2#3]
Your task is to locate all objects that have a relation <relation> with <subject> and classify them.
I need you to categorize and locate all objects that is related to <subject> with relationship as <relation>.
Please locate and categorize all the objects that have a relation of <relation> with <subject>.
Assist me in locating and classifying all the objects <relation> the <subject>?
Find all the objects that have a relation of <relation> with the <subject>. Can you give me their positions and categories
Your task is to locate all objects that have a relation <relation> with <subject> and classify them.
I need you to categorize and locate all objects that is related to <subject> with relationship as <relation>.
Please locate and categorize all the objects that have a relation of <relation> with <subject>.
Assist me in locating and classifying the position of all the objects <relation> the <subject>?
Find all the objects that have a relation of <relation> with the <subject>. Can you give me their positions and categories?
Your task is to locate and classify all objects that are related to <subject> using the relationship <relation>.
I need you to locate and categorize all objects having a relationship <relation> with the given <subject>.
Find all objects related to <subject> with the relationship <relation>. Categorize and locate them for me.
Your objective is to locate and classify the objects that are related to <subject> through the relationship <relation>.
I require you to detect and categorize all objects that have a relationship <relation> with <subject>.
Please find and classify all objects that has a relationship <relation> with <subject>.
Assist me in locating and categorizing all objects that related to <subject> with the relationship <relation>.
Find all objects that are related to <subject> using the relationship <relation>. Categorize and locate their positions.
Your task is to identify and classify all objects related to <subject> through the relationship <relation>.
I need you to locate and categorize all objects that have a relationship <relation> with <subject>.
Assist me in locating and classifying all objects that are related to <subject> through the relationship <relation>.
Find all the objects that has a relationship <relation> with <subject>. Categorize and locate their positions for me.

Instruction templates of coarse visual spatial reasoning.
We list all the instruction templates of coarse visual spatial reasoning task below. Similar to visual relation reasoning, different instruction templates are used to ask the model output different contents.
[title=Instruction templates of coarse visual spatial reasoning#1]
Identify the objects located at <loc> of <object>. Please classify them by category and provide their locations.
I need to know what objects are present at <loc> of <object>. Can you help me locate and categorize them?
Find all the objects at <loc> of <object>. Please provide me with their categories and locations.
I want to know the categories and positions of the objects located at <loc> of <object>.
Locate and classify all the objects at <loc> of <object>.
Could you tell me the categories and positions of the objects present at <loc> of <object>?
Help me locate and categorize all the objects at <loc> of <object>.
I need to know the categories and locations of the objects at <loc> of <object>.
What are the categories and positions of the objects located at <loc> of <object>?
Identify and locate all the objects at <loc> of <object>. I need their categories and positions.
I want to know the categories and positions of the objects at <loc> of <object>.
Locate and classify all the objects at <loc> of <object>. Please provide me with their categories and positions.

[title=Instruction templates of coarse visual spatial reasoning#2]
What are the categories of the objects located at <loc> of <object>?
Detect and classify all the objects at <loc> of <object>. I need to know their categories.
Please find and categorize all the objects present at <loc> of <object>.
Give the categories of all the objects you can find at <loc> of <object>.
I need you to find and categorize all the objects that are at <loc> of <object>.
Please provide me with the categories of all the objects present at <loc> of <object>.
What types of objects are located at <loc> of <object>? Please list their categories.
Please find all the objects at <loc> of <object> and give me their categories.
What are the categories of the objects that are present at <loc> of <object>?
I need you to classify all the objects located at <loc> of <object>.
Please give me the categories of all the objects that are located at <loc> of <object>.

[title=Instruction templates of coarse visual spatial reasoning#3]
What are the coordinates of the objects located at <loc> of <object>?
Detect and give the coordinates of all the objects at <loc> of <object>.
Please find and locate all the objects present at <loc> of <object>.
Give the detail locations of all the objects you can find at of <loc> <object>.
Locate all the objects and give there coordinates found at <loc> of <object>.
What are the positions of all the objects at <loc> of <object>?
Can you find and list the positions of all the objects present at <loc> of <object>?
Provide the coordinates of objects located at <loc> of <object>.
List and indicate the positions of all objects at <loc> of <object>.
Enumerate and specify the positions of all objects found at <loc> of <object>.
What objects are situated at <loc> of <object> and where precisely are they located?
What are the coordinates of all objects found at <loc> of <object>?

Instruction templates of object counting.
We list all the instruction templates of object counting task below. <category> will be replaced by the category name.
[title=Instruction templates of object counting#1]
Can you tell me how many <category> are present in this picture?
I need to know the number of <category> in this image.
Count how many <category> are in this picture.
Please determine the quantity of <category> shown in this image.
How many instances of <category> can you find in this picture?
I would like to know how many <category> are visible in this image.
Count the number of <category> that you see in this picture.
Please provide me with the count of <category> in this image.
How many objects of <category> are in this image?
Can you count the items of <category> in this picture?
What is the total number of <category> in this image?
How many <category> can you spot in this image?
Please determine the quantity of <category> in this image.
Count the number of <category> that appear in this picture.
How many <category> are in the picture?
Counting the number of <category> appeared in the image.
Please give me the number of <category> appeared in the image.

[title=Instruction templates of object counting#2]
How many objects in the image are of the same category as <object>?
Count the number of objects in the image that are similar to <object> in category.
What is the total count of objects that share the same category as <object> in the image?
How many objects in the image have the same category as <object>?
Count all the objects in the image that fall under the same category as <object>.
What is the number of objects that share the same category as <object> in the image?
Count the objects that belong to the same category as <object> in the image.
How many objects of the same category as the object represented by <object> appear in the image?
Count all the instances whose category is the same as <object> present in the image.

Instruction templates of object detection.
We list all the instruction templates of object detection task below. <category> will be replaced by the category name.
[title=Instruction templates of object detection#1]
Locate and mark the positions of all <category> in the image.
Find all the instances of <category> in the image and indicate their respective locations.
Spot and record the coordinates of every <category> present in the image.
Identify the <category> in the image and provide their precise locations.
Can you determine the positions of all the <category> in the image and list them?
Pinpoint the <category> in the image and give me their exact coordinates.
Locate all the <category> in the image and provide their locations in detail.
Detect and report the locations of all the <category> present in the image.
Find and list the locations of every <category> in the image.
Please identify the <category> in the image and give me their locations.
Provide me with the precise locations of all the <category> in the image.
Detect and record the positions of the <category> in the image.
Spot all the instances of <category> in the image and give me their coordinates.
Detect all the <category> in the image, and output there location.
There are some <category> in the image, could you help me to locate them and give me their coordinates.
What are the coordinates of the <category> in the image.
Give the detail locations of all the <category> you can find in the image.

[title=Instruction templates of object detection#2]
Locate all the items in the picture that share the same category as <object> and provide their coordinates.
Spot every object that belongs to the same category as <object> and indicate their positions.
Identify all the objects that fit the same category as <object> and display their coordinates.
Find all the objects that have a similar classification as <object> and output their locations.
Locate and report the coordinates of all the objects that share the category with <object>.
Detect all the objects in the image that have the same classification as <object> and provide their positions.
Spot all the objects that belong to the same category as <object> and show their coordinates.
Identify every instance that falls under the same category as <object> and report their locations.
Find and output the coordinates of all the objects that have the same category as <object>.
Locate all the objects in the picture that have a similar classification as <object> and display their positions.
Detect and report the positions of all the objects that share the category with <object>.
Spot every instance that has a similar classification as <object> and indicate its coordinates.
Identify all the objects that have the same classification as <object> and output their positions.
Find all the objects that belong to the same category as <object> and report their locations.
Locate and output the coordinates of all the items that have a similar category as <object>.
Detect all the instances in the image which have the same category with <object>, and output there location.
Detect and report the locations of all the instances present in the image, these instances should have similar category with <object>.
Given an <object>, please help me to find all the instances with the same category. The output should be the coordinates of detected instances.

Instruction templates of multi-choices VQA.
Our instruction tuning dataset also includes the multi-choices VQA, e.g., A-OKVQA. Therefore, we also construct some instruction templates for this task and list them below. Placeholder <options> will be replaced by the options.
[title=Instruction templates of multi-choices VQA]
Please take a look at the image and select the correct answer for <question> from the options given below \n<options>.
Examine the image and select the best matched answer to the question: <question> from the options given below\n<options>.
There are some options\n<options>. I have a question for you: <question> Can you select the best matched answers from the given options based on the image?
Regarding the image, you need to identify the correct answer to the question <question> from the given options\n<options>.
Analyzing the image, can you identify the best matched answer to <question> from the given options\n<options>.
Looking at the image, can you quickly answer my question: <question>. Some potential answers are given in the following options\n<options>.
Referring to the image, please select the answer for this question: <question> from the options\n<options>.
Could you please check the image and select the answer for my question: <question> from the options\n<options>.
Here is an image and a question: <question> for you. Please select an option that can answer the question from the given options\n<options>.
For this image, I want to know which option can answer my question: <question> correctly. The options are\n<options>.
Take a look at the image, can you select the best matched answer to the following question: <question> from following options\n<options>.
Considering these options\n<options>. I need a correct selection from these options that can answer this question: <question> in regards to the image.


Visual Spatial Reasoning

Abstract: Visual Question Answering (VQA) in its ideal form lets us study reasoning in the joint space of vision and language and serves as a proxy for the AI task of scene understanding. However, most VQA benchmarks to date are focused on questions such as simple counting, visual attributes, and object detection that do not require reasoning or knowledge beyond what is in the image. In this paper, we address the task of knowledge-based visual question answering and provide a benchmark, called OK-VQA, where the image content is not sufficient to answer the questions, encouraging methods that rely on external knowledge resources. Our new dataset includes more than 14,000 questions that require external knowledge to answer. We show that the performance of the state-of-the-art VQA models degrades drastically in this new setting. Our analysis shows that our knowledge-based VQA task is diverse, difficult, and large compared to previous knowledge-based VQA datasets. We hope that this dataset enables researchers to open up new avenues for research in this domain. See http://okvqa.allenai.org to download and browse the dataset.

Abstract: We propose a method that can generate an unambiguous description (known as a referring expression) of a specific object or region in an image, and which can also comprehend or interpret such an expression to infer which object is being described. We show that our method outperforms previous methods that generate descriptions of objects without taking into account other potentially ambiguous objects in the scene. Our model is inspired by recent successes of deep learning methods for image captioning, but while image captioning is difficult to evaluate, our task allows for easy objective evaluation. We also present a new large-scale dataset for referring expressions, based on MS-COCO. We have released the dataset and a toolbox for visualization and evaluation, see https://github.com/mjhucla/Google_Refexp_toolbox

Abstract:
Visual Question Answering (VQA) has become one of the key benchmarks of visual recognition progress. Multiple VQA extensions have been explored to better simulate real-world settings: different question formulations, changing training and test distributions, conversational consistency in dialogues, and explanation-based answering. In this work, we further expand this space by considering visual questions that include a spatial point of reference. Pointing is a nearly universal gesture among humans, and real-world VQA is likely to involve a gesture towards the target region.
Concretely, we (1) introduce and motivate point-input questions as an extension of VQA, (2) define three novel classes of questions within this space, and (3) for each class, introduce both a benchmark dataset and a series of baseline models to handle its unique challenges. There are two key distinctions from prior work. First, we explicitly design the benchmarks to require the point input, i.e., we ensure that the visual question cannot be answered accurately without the spatial reference. Second, we explicitly explore the more realistic point spatial input rather than the standard but unnatural bounding box input. Through our exploration we uncover and address several visual recognition challenges, including the ability to infer human intent, reason both locally and globally about the image, and effectively combine visual, language and spatial inputs. Code is available at: this https URL .

Summary of evaluation datasets.

Statistics of Training and Evaluation Datasets

The statistics for the training datasets and evaluation datasets are summarized in Table {{table:1881d4a7-d5bb-4f74-bb6d-abc83fd134c7}}  and Table {{table:5c87bb1e-80c9-43b5-b734-307a91476dc0}} , respectively. We also provide an overview of the generated referring-expression-bounding-box pairs in Object365 in Table {{table:67de358d-9674-4efe-afb3-1f394b8db489}} .
To ensure the quality of data, we apply certain filters during preprocessing. Firstly, we exclude images containing more than 15 objects. Moreover, for the purpose of bounding box description bootstrapping, we only consider objects that cover an area of more than 2,000 pixels. As a result, our dataset comprises 1,063,034 images, with a total of 4,961,822 generated referring-expression-bounding-box pairs. To further enhance the reliability of our dataset, we perform a self-consistent method that filters out 2,528,619 low-quality referring-expression-bounding-box pairs.


When the device leaves or is removed from the system

Device self-protection and self-defence:- Hardware self-protection - Software self-protection - Moving Target Defence

System availability and resilience: - Diversity - Exploiting data correlation - Exploiting data overhearing - Network Coding (NC)

Hardware Security in IoT Devices with Emphasis on Hardware Trojans

Abstract: Security of IoT devices is getting a lot of attention from researchers as they are becoming prevalent everywhere. However, implementation of hardware security in these devices has been overlooked, and many researches have mainly focused on software, network, and cloud security. A deeper understanding of hardware Trojans (HTs) and protection against them is of utmost importance right now as they are the prime threat to the hardware. This paper emphasizes the need for a secure hardware-level foundation for security of these devices, as depending on software security alone is not adequate enough. These devices must be protected against sophisticated attacks, especially if the groundwork for the attacks is already laid in devices during design or manufacturing process, such as with HTs. This paper will discuss the stealthy nature of these HT, highlight HT taxonomy and insertion methods, and provide countermeasures.

Synergistic Security for the Industrial Internet of Things: Integrating Redundancy, Diversity, and Hardening

Abstract: As the Industrial Internet of Things (IIot) becomes more prevalent in critical application domains, ensuring security and resilience in the face of cyber-attacks is becoming an issue of paramount importance. Cyber-attacks against critical infrastructures, for example, against smart water-distribution and transportation systems, pose serious threats to public health and safety. Owing to the severity of these threats, a variety of security techniques are available. However, no single technique can address the whole spectrum of cyber-attacks that may be launched by a determined and resourceful attacker. In light of this, we consider a multi-pronged approach for designing secure and resilient IIoT systems, which integrates redundancy, diversity, and hardening techniques. We introduce a framework for quantifying cyber-security risks and optimizing IIoT design by determining security investments in redundancy, diversity, and hardening. To demonstrate the applicability of our framework, we present two case studies in water distribution and transportation a case study in water-distribution systems. Our numerical evaluation shows that integrating redundancy, diversity, and hardening can lead to reduced security risk at the same cost.

A taxonomy for mitigation techniques throughout the life-cycle of an IoT device

To answer these questions, this study explores below additional security measures that can be applied across different IoT application contexts, and elaborates on how to apply each mitigation technique, its benefits and limitations and the extent to which progress has been reported in the literature. By contrast to the vast majority of the research directions presented in literature which is related to this work, this survey does not discuss the mitigation techniques in the IoT domain from the layered network architecture of an IoT system, such as IoT-Cloud/Edge, IoT-IoT, and IoT-gateway. In another words, the discussions in this work is not structured based on each network layer of the IoT device or its associated protocols. This is because discussing the topic from the layering architecture perspective does not help to answer the two questions presented above: the layering discussion does not account for the dynamicity of the IoT environment, one of which is the mobility of the IoT device itself when joining or leaving a network, which is one of the main reasons for the fast propagation of a cyber attack in the IoT environment. Rather, this survey provides answers to the questions of how IoT device(s) connect and communicate with new devices and systems safely, from the perspective of discussing the life cycle of the device(s), starting from the moment when the IoT device(s) join the new environment, whilst operating in it, when an attack occurs, and until the device is removed/decommissioned or leaves the environment, and hence the discussed measures address the dynamicity of the IoT environment. Those steps are depicted in Fig {{figure:77316450-fec7-48b4-9e22-a497d0b045a1}} , which shows the mitigation steps as a continuous process run in a “defence in depth” approach starting when the device tries to join a system until it dies or leaves. Those mitigation techniques are summarised in Table {{table:2c73a3c6-74ef-46a1-8f1a-a1a010174389}}  and are discussed in detail in the following Sections.

This survey has collated current research into risk and threat mitigations in the IoT environment, providing an overview and presenting these uniquely throughout the life cycle of the IoT device, starting from its design, to the moment when a device joins a network, while the IoT device operates in the network, and until it leaves (or is removed) from the network and is eventually decommissioned, hence adopting a “defence-in-depth” approach. A summary of the discussed mitigation techniques is presented in Table {{table:2c73a3c6-74ef-46a1-8f1a-a1a010174389}} . The survey has discussed threat mitigation techniques applied across different IoT application contexts, and has elaborated on how to apply each mitigation technique, its benefits and limitations and the extent to which progress has been reported in the literature.


Few-shot Object Detection on Remote Sensing Images

Abstract: In this paper, we deal with the problem of object detection on remote sensing images. Previous methods have developed numerous deep CNN-based methods for object detection on remote sensing images and the report remarkable achievements in detection performance and efficiency. However, current CNN-based methods mostly require a large number of annotated samples to train deep neural networks and tend to have limited generalization abilities for unseen object categories. In this paper, we introduce a few-shot learning-based method for object detection on remote sensing images where only a few annotated samples are provided for the unseen object categories. More specifically, our model contains three main components: a meta feature extractor that learns to extract feature representations from input images, a reweighting module that learn to adaptively assign different weights for each feature representation from the support images, and a bounding box prediction module that carries out object detection on the reweighted feature maps. We build our few-shot object detection model upon YOLOv3 architecture and develop a multi-scale object detection framework. Experiments on two benchmark datasets demonstrate that with only a few annotated samples our model can still achieve a satisfying detection performance on remote sensing images and the performance of our model is significantly better than the well-established baseline models.

A Fast and Accurate Few-Shot Detector for Objects with Fewer Pixels in Drone Image

Abstract: Unmanned aerial vehicles (UAVs) are important in modern war, and object detection performance influences the development of related intelligent drone application. At present, the target categories of UAV detection tasks are diversified. However, the lack of training samples of novel categories will have a bad impact on the task. At the same time, many state-of-the-arts are not suitable for drone images due to the particularity of perspective and large number of small targets. In this paper, we design a fast few-shot detector for drone targets. It adopts the idea of anchor-free in fully convolutional one-stage object detection (FCOS), which leads to a more reasonable definition of positive and negative samples and faster speed, and introduces Siamese framework with more discriminative target model and attention mechanism to integrate similarity measures, which enables our model to match the objects of the same categories and distinguish the different class objects and background. We propose a matching score map to utilize the similarity information of attention feature map. Finally, through soft-NMS, the predicted detection bounding boxes for support category objects are generated. We construct a DAN dataset as a collection of DOTA and NWPU VHR-10. Compared with many state-of-the-arts on the DAN dataset, our model is proved to outperform them for few-shot detection tasks of drone images.

Few-shot Object Detection with Self-adaptive Attention Network for Remote Sensing Images

Abstract: In remote sensing field, there are many applications of object detection in recent years, which demands a great number of labeled data. However, we may be faced with some cases where only limited data are available. In this paper, we proposed a few-shot object detector which is designed for detecting novel objects provided with only a few examples. Particularly, in order to fit the object detection settings, our proposed few-shot detector concentrates on the relations that lie in the level of objects instead of the full image with the assistance of Self-Adaptive Attention Network (SAAN). The SAAN can fully leverage the object-level relations through a relation GRU unit and simultaneously attach attention on object features in a self-adaptive way according to the object-level relations to avoid some situations where the additional attention is useless or even detrimental. Eventually, the detection results are produced from the features that are added with attention and thus are able to be detected simply. The experiments demonstrate the effectiveness of the proposed method in few-shot scenes.

PennSyn2Real: Training Object Recognition Models without Human Labeling

Abstract: Scalable training data generation is a critical problem in deep learning. We propose PennSyn2Real - a photo-realistic synthetic dataset consisting of more than 100,000 4K images of more than 20 types of micro aerial vehicles (MAVs). The dataset can be used to generate arbitrary numbers of training images for high-level computer vision tasks such as MAV detection and classification. Our data generation framework bootstraps chroma-keying, a mature cinematography technique with a motion tracking system, providing artifact-free and curated annotated images where object orientations and lighting are controlled. This framework is easy to set up and can be applied to a broad range of objects, reducing the gap between synthetic and real-world data. We show that synthetic data generated using this framework can be directly used to train CNN models for common object recognition tasks such as detection and segmentation. We demonstrate competitive performance in comparison with training using only real images. Furthermore, bootstrapping the generated synthetic data in few-shot learning can significantly improve the overall performance, reducing the number of required training data samples to achieve the desired accuracy.

Feature Enhancement Network for Object Detection in Optical Remote Sensing Images

Abstract: Automatic and robust object detection in remote sensing images is of vital significance in real-world applications such as land resource management and disaster rescue. However, poor performance arises when the state-of-the-art natural image detection algorithms are directly applied to remote sensing images, which largely results from the variations in object scale, aspect ratio, indistinguishable object appearances, and complex background scenario. In this paper, we propose a novel Feature Enhancement Network (FENet) for object detection in optical remote sensing images, which consists of a Dual Attention Feature Enhancement (DAFE) module and a Context Feature Enhancement (CFE) module. Specifically, the DAFE module is introduced to highlight the network to focus on the distinctive features of the objects of interest and suppress useless ones by jointly recalibrating the spatial and channel feature responses. The CFE module is designed to capture global context cues and selectively strengthen class-aware features by leveraging image-level contextual information that indicates the presence or absence of the object classes. To this end, we employ a context encoding loss to regularize the model training which promotes the object detector to understand the scene better and narrows the probable object categories in prediction. We achieve our proposed FENet by unifying DAFE and CFE into the framework of Faster R-CNN. In the experiments, we evaluate our proposed method on two large-scale remote sensing image object detection datasets including DIOR and DOTA and demonstrate its effectiveness compared with the baseline methods.

Rethinking Intersection Over Union for Small Object Detection in Few-Shot Regime

Abstract: In Few-Shot Object Detection (FSOD), detecting small objects is extremely difficult. The limited supervision cripples the localization capabilities of the models and a few pixels shift can dramatically reduce the Intersection over Union (IoU) between the ground truth and predicted boxes for small objects. To this end, we propose Scale-adaptive Intersection over Union (SIoU), a novel box similarity measure. SIoU changes with the objects' size, it is more lenient with small object shifts. We conducted a user study and SIoU better aligns than IoU with human judgment. Employing SIoU as an evaluation criterion helps to build more user-oriented models. SIoU can also be used as a loss function to prioritize small objects during training, outperforming existing loss functions. SIoU improves small object detection in the non-few-shot regime, but this setting is unrealistic in the industry as annotated detection datasets are often too expensive to acquire. Hence, our experiments mainly focus on the few-shot regime to demonstrate the superiority and versatility of SIoU loss. SIoU improves significantly FSOD performance on small objects in both natural (Pascal VOC and COCO datasets) and aerial images (DOTA and DIOR). In aerial imagery, small objects are critical and SIoU loss achieves new state-of-the-art FSOD on DOTA and DIOR.

Transformation-Invariant Network for Few-Shot Object Detection in Remote-Sensing Images

Abstract: Object detection in remote-sensing images (RSIs) relies on a large amount of labeled data for training. However, the increasing number of new categories and class imbalance make exhaustive annotation impractical. Few-shot object detection (FSOD) addresses this issue by leveraging meta-learning on seen base classes and fine-tuning on novel classes with limited labeled samples. Nonetheless, the substantial scale and orientation variations of objects in RSIs pose significant challenges to existing FSOD methods. To overcome these challenges, we propose integrating a feature pyramid network (FPN) and utilizing prototype features to enhance query features, thereby improving existing FSOD methods. We refer to this modified FSOD approach as a Strong Baseline, which has demonstrated significant performance improvements compared to the original baselines. Furthermore, we tackle the issue of spatial misalignment caused by orientation variations between the query and support images by introducing a transformation-invariant network (TINet). TINet ensures geometric invariance and explicitly aligns the features of the query and support branches, resulting in additional performance gains while maintaining the same inference speed as the Strong Baseline. Extensive experiments on three widely used remote-sensing object detection datasets, that is, NWPU VHR-10.v2, DIOR, and HRRSD demonstrated the effectiveness of the proposed method.

Overview of studies addressing challenges in few-shot object detection for aerial remote sensing images

Few-shot object detection in remote sensing

There are various methods used in few-shot object detection, including metric learning methods, meta-learning methods, and data augmentation methods {{cite:30272fc}}, {{cite:82c7238}}, {{cite:b0f9483}}. Metric learning methods aim to learn a distance function that can measure the similarity between objects. Meta-learning methods aim to learn a model that can quickly adapt to new tasks with a few training examples with the help other domain informations {{cite:209ff1a}}. Data augmentation methods aim to generate more training examples by applying transformations to the existing examples {{cite:ecd9284}}. Furthermore, a more comprehensive analysis of aerial image-based FSOD is available in the summarized Table {{table:5a89c0f5-cbf0-4b96-8c70-9c2bd3a4e6ef}} .


LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop

Abstract: While there has been remarkable progress in the performance of visual recognition algorithms, the state-of-the-art models tend to be exceptionally data-hungry. Large labeled training datasets, expensive and tedious to produce, are required to optimize millions of parameters in deep network models. Lagging behind the growth in model capacity, the available datasets are quickly becoming outdated in terms of size and density. To circumvent this bottleneck, we propose to amplify human effort through a partially automated labeling scheme, leveraging deep learning with humans in the loop. Starting from a large set of candidate images for each category, we iteratively sample a subset, ask people to label them, classify the others with a trained model, split the set into positives, negatives, and unlabeled based on the classification confidence, and then iterate with the unlabeled set. To assess the effectiveness of this cascading procedure and enable further progress in visual recognition research, we construct a new image dataset, LSUN. It contains around one million labeled images for each of 10 scene categories and 20 object categories. We experiment with training popular convolutional networks and find that they achieve substantial performance gains when trained on this dataset.

The TerraByte Client: providing access to terabytes of plant data

Abstract: —In this paper we demonstrate the TerraByte Client, a software to download user-deﬁned plant datasets from a data portal hosted at Compute Canada. To that end the client offers two key functionalities: (1) It allows the user to get an overview on what data is available and a quick way to visually check samples of that data. For this the client receives the results of queries to a database and displays the number of images that fulﬁll the search criteria. Furthermore, a sample can be downloaded within seconds to conﬁrm that the data suits the user’s needs. (2) The user can then download the speciﬁed data to their own drive. This data is prepared into chunks server-side and sent to the user’s end-system, where it is automatically extracted into individual ﬁles. The ﬁrst chunks of data are available for inspection after a brief waiting period of a minute or less depending on available bandwidth and type of data. The TerraByte Client has a full graphical user interface for easy usage and uses end-to-end encryption. The user interface is built on top of a low-level client. This architecture in combination of offering the client program open-source makes it possible for the user to develop their own user interface or use the client’s functionality directly. An example for direct usage could be to download speciﬁc data on demand within a larger application, such as training machine learning models.

Datasets used

Outside of simulated learning curves, we have also trained several models on datasets to validate our strategies in real scenarios. For this we have selected three datasets with different characteristics. ImageNet ({{cite:59001f8}}) represents a well studied large dataset that has many different classes. LSUN ({{cite:84781bd}}) has fewer classes and thus a higher image density per class. Finally, we also retrieved a plant dataset through a data portal ({{cite:b7d7fe5}}). The such constructed dataset is much smaller than ImageNet or LSUN, but offers different characteristics. Its images show different plants on front of a blue background and each individual plant is imaged from different angles. Thus, the images contain a lot of similar features and can be extremely similar to each other. Table {{table:b280b47f-c966-4123-a373-6246e8ca157c}}  gives an overview on the different dataset characteristics and the models applied to them.