From Recognition to Cognition: Visual Commonsense Reasoning

Abstract: Visual understanding goes well beyond object recognition. With one glance at an image, we can effortlessly imagine the world beyond the pixels: for instance, we can infer people's actions, goals, and mental states. While this task is easy for humans, it is tremendously difficult for today's vision systems, requiring higher-order cognition and commonsense reasoning about the world. We formalize this task as Visual Commonsense Reasoning. Given a challenging question about an image, a machine must answer correctly and then provide a rationale justifying its answer. Next, we introduce a new dataset, VCR, consisting of 290k multiple choice QA problems derived from 110k movie scenes. The key recipe for generating non-trivial and high-quality problems at scale is Adversarial Matching, a new approach to transform rich annotations into multiple choice questions with minimal bias. Experimental results show that while humans find VCR easy (over 90% accuracy), state-of-the-art vision models struggle (~45%). To move towards cognition-level understanding, we present a new reasoning engine, Recognition to Cognition Networks (R2C), that models the necessary layered inferences for grounding, contextualization, and reasoning. R2C helps narrow the gap between humans and machines (~65%); still, the challenge is far from solved, and we provide analysis that suggests avenues for future work.

Broaden the Vision: Geo-Diverse Visual Commonsense Reasoning

Abstract: Commonsense is defined as the knowledge on which everyone agrees. However, certain types of commonsense knowledge are correlated with culture and geographic locations and they are only shared locally. For example, the scenes of wedding ceremonies vary across regions due to different customs influenced by historical and religious factors. Such regional characteristics, however, are generally omitted in prior work. In this paper, we construct a Geo-Diverse Visual Commonsense Reasoning dataset (GD-VCR) to test vision-and-language models’ ability to understand cultural and geo-location-specific commonsense. In particular, we study two state-of-the-art Vision-and-Language models, VisualBERT and ViLBERT trained on VCR, a standard benchmark with images primarily from Western regions. We then evaluate how well the trained models can generalize to answering the questions in GD-VCR. We find that the performance of both models for non-Western regions including East Asia, South Asia, and Africa is significantly lower than that for Western region. We analyze the reasons behind the performance disparity and find that the performance gap is larger on QA pairs that: 1) are concerned with culture-related scenarios, e.g., weddings, religious activities, and festivals; 2) require high-level geo-diverse commonsense reasoning rather than low-order perception and recognition. Dataset and code are released at https://github.com/WadeYin9712/GD-VCR.

Breaking Common Sense: WHOOPS! A Vision-and-Language Benchmark of Synthetic and Compositional Images

Abstract: Weird, unusual, and uncanny images pique the curiosity of observers because they challenge commonsense. For example, an image released during the 2022 world cup depicts the famous soccer stars Lionel Messi and Cristiano Ronaldo playing chess, which playfully violates our expectation that their competition should occur on the football field.1 Humans can easily recognize and interpret these unconventional images, but can AI models do the same? We introduce WHOOPS!, a new dataset and benchmark for visual commonsense. The dataset is comprised of purposefully commonsense-defying images created by designers using publicly-available image generation tools like Midjourney. We consider several tasks posed over the dataset. In addition to image captioning, cross-modal matching, and visual question answering, we introduce a difficult explanation generation task, where models must identify and explain why a given image is unusual. Our results show that state-of-the-art models such as GPT3 and BLIP2 still lag behind human performance on WHOOPS!. We hope our dataset will inspire the development of AI models with stronger visual commonsense reasoning abilities. 2

Do Androids Laugh at Electric Sheep? Humor “Understanding” Benchmarks from The New Yorker Caption Contest

Abstract: Large neural networks can now generate jokes, but do they really “understand” humor? We challenge AI models with three tasks derived from the New Yorker Cartoon Caption Contest: matching a joke to a cartoon, identifying a winning caption, and explaining why a winning caption is funny. These tasks encapsulate progressively more sophisticated aspects of “understanding” a cartoon; key elements are the complex, often surprising relationships between images and captions and the frequent inclusion of indirect and playful allusions to human experience and culture. We investigate both multimodal and language-only models: the former are challenged with the cartoon images directly, while the latter are given multifaceted descriptions of the visual scene to simulate human-level visual understanding. We find that both types of models struggle at all three tasks. For example, our best multimodal models fall 30 accuracy points behind human performance on the matching task, and, even when provided ground-truth visual scene descriptors, human-authored explanations are preferred head-to-head over the best machine-authored ones (few-shot GPT-4) in more than 2/3 of cases. We release models, code, leaderboard, and corpus, which includes newly-gathered annotations describing the image’s locations/entities, what’s unusual in the scene, and an explanation of the joke.

CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning

Abstract: When building artificial intelligence systems that can reason and answer questions about visual data, we need diagnostic tests to analyze our progress and discover short-comings. Existing benchmarks for visual question answering can help, but have strong biases that models can exploit to correctly answer questions without reasoning. They also conflate multiple sources of error, making it hard to pinpoint model weaknesses. We present a diagnostic dataset that tests a range of visual reasoning abilities. It contains minimal biases and has detailed annotations describing the kind of reasoning each question requires. We use this dataset to analyze a variety of modern visual reasoning systems, providing novel insights into their abilities and limitations.

Abstract Visual Reasoning with Tangram Shapes

Abstract: We introduce KiloGram, a resource for studying abstract visual reasoning in humans and machines. Drawing on the history of tangram puzzles as stimuli in cognitive science, we build a richly annotated dataset that, with >1k distinct stimuli, is orders of magnitude larger and more diverse than prior resources. It is both visually and linguistically richer, moving beyond whole shape descriptions to include segmentation maps and part labels. We use this resource to evaluate the abstract visual reasoning capacities of recent multi-modal models. We observe that pre-trained weights demonstrate limited abstract reasoning, which dramatically improves with fine-tuning. We also observe that explicitly describing parts aids abstract reasoning for both humans and models, especially when jointly encoding the linguistic and visual inputs.

What do you MEME? Generating Explanations for Visual Semantic Role Labelling in Memes

Abstract: Memes are powerful means for effective communication on social media. Their effortless amalgamation of viral visuals and compelling messages can have far-reaching implications with proper marketing. Previous research on memes has primarily focused on characterizing their affective spectrum and detecting whether the meme's message insinuates any intended harm, such as hate, offense, racism, etc. However, memes often use abstraction, which can be elusive. Here, we introduce a novel task - EXCLAIM, generating explanations for visual semantic role labeling in memes. To this end, we curate ExHVV, a novel dataset that offers natural language explanations of connotative roles for three types of entities - heroes, villains, and victims, encompassing 4,680 entities present in 3K memes. We also benchmark ExHVV with several strong unimodal and multimodal baselines. Moreover, we posit LUMEN, a novel multimodal, multi-task learning framework that endeavors to address EXCLAIM optimally by jointly learning to predict the correct semantic roles and correspondingly to generate suitable natural language explanations. LUMEN distinctly outperforms the best baseline across 18 standard natural language generation evaluation metrics. Our systematic evaluation and analyses demonstrate that characteristic multimodal cues required for adjudicating semantic roles are also helpful for generating suitable explanations.

Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering

Abstract: When answering a question, humans utilize the information available across different modalities to synthesize a consistent and complete chain of thought (CoT). This process is normally a black box in the case of deep learning models like large-scale language models. Recently, science question benchmarks have been used to diagnose the multi-hop reasoning ability and interpretability of an AI system. However, existing datasets fail to provide annotations for the answers, or are restricted to the textual-only modality, small scales, and limited domain diversity. To this end, we present Science Question Answering (ScienceQA), a new benchmark that consists of ~21k multimodal multiple choice questions with a diverse set of science topics and annotations of their answers with corresponding lectures and explanations. We further design language models to learn to generate lectures and explanations as the chain of thought (CoT) to mimic the multi-hop reasoning process when answering ScienceQA questions. ScienceQA demonstrates the utility of CoT in language models, as CoT improves the question answering performance by 1.20% in few-shot GPT-3 and 3.99% in fine-tuned UnifiedQA. We also explore the upper bound for models to leverage explanations by feeding those in the input; we observe that it improves the few-shot performance of GPT-3 by 18.96%. Our analysis further shows that language models, similar to humans, benefit from explanations to learn from fewer data and achieve the same performance with just 40% of the data. The data and code are available at https://scienceqa.github.io.

OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge

Abstract: Visual Question Answering (VQA) in its ideal form lets us study reasoning in the joint space of vision and language and serves as a proxy for the AI task of scene understanding. However, most VQA benchmarks to date are focused on questions such as simple counting, visual attributes, and object detection that do not require reasoning or knowledge beyond what is in the image. In this paper, we address the task of knowledge-based visual question answering and provide a benchmark, called OK-VQA, where the image content is not sufficient to answer the questions, encouraging methods that rely on external knowledge resources. Our new dataset includes more than 14,000 questions that require external knowledge to answer. We show that the performance of the state-of-the-art VQA models degrades drastically in this new setting. Our analysis shows that our knowledge-based VQA task is diverse, difficult, and large compared to previous knowledge-based VQA datasets. We hope that this dataset enables researchers to open up new avenues for research in this domain.

A-OKVQA: A Benchmark for Visual Question Answering using World Knowledge

Abstract: The Visual Question Answering (VQA) task aspires to provide a meaningful testbed for the development of AI models that can jointly reason over visual and natural language inputs. Despite a proliferation of VQA datasets, this goal is hindered by a set of common limitations. These include a reliance on relatively simplistic questions that are repetitive in both concepts and linguistic structure, little world knowledge needed outside of the paired image, and limited reasoning required to arrive at the correct answer. We introduce A-OKVQA, a crowdsourced dataset composed of a diverse set of about 25K questions requiring a broad base of commonsense and world knowledge to answer. In contrast to the existing knowledge-based VQA datasets, the questions generally cannot be answered by simply querying a knowledge base, and instead require some form of commonsense reasoning about the scene depicted in the image. We demonstrate the potential of this new dataset through a detailed analysis of its contents and baseline performance measurements over a variety of state-of-the-art vision-language models. Project page: http://a-okvqa.allenai.org/

A Corpus for Reasoning about Natural Language Grounded in Photographs

Abstract: We introduce a new dataset for joint reasoning about natural language and images, with a focus on semantic diversity, compositionality, and visual reasoning challenges. The data contains 107,292 examples of English sentences paired with web photographs. The task is to determine whether a natural language caption is true about a pair of photographs. We crowdsource the data using sets of visually rich images and a compare-and-contrast task to elicit linguistically diverse language. Qualitative analysis shows the data requires compositional joint reasoning, including about quantities, comparisons, and relations. Evaluation using state-of-the-art visual reasoning methods shows the data presents a strong challenge.

Abstract: The ability to integrate context, including perceptual and temporal cues, plays a pivotal role in grounding the meaning of a linguistic utterance. In order to measure to what extent current vision-and-language models master this ability, we devise a new multimodal challenge, Image Retrieval from Contextual Descriptions (ImageCoDe). In particular, models are tasked with retrieving the correct image from a set of 10 minimally contrastive candidates based on a contextual description.As such, each description contains only the details that help distinguish between images.Because of this, descriptions tend to be complex in terms of syntax and discourse and require drawing pragmatic inferences. Images are sourced from both static pictures and video frames.We benchmark several state-of-the-art models, including both cross-encoders such as ViLBERT and bi-encoders such as CLIP, on ImageCoDe.Our results reveal that these models dramatically lag behind human performance: the best variant achieves an accuracy of 20.9 on video frames and 59.4 on static pictures, compared with 90.8 in humans.Furthermore, we experiment with new model variants that are better equipped to incorporate visual and temporal context into their representations, which achieve modest gains. Our hope is that ImageCoDE will foster progress in grounded language understanding by encouraging models to focus on fine-grained visual differences.

Learning to Describe Differences Between Pairs of Similar Images

Abstract: In this paper, we introduce the task of automatically generating text to describe the differences between two similar images. We collect a new dataset by crowd-sourcing difference descriptions for pairs of image frames extracted from video-surveillance footage. Annotators were asked to succinctly describe all the differences in a short paragraph. As a result, our novel dataset provides an opportunity to explore models that align language and vision, and capture visual salience. The dataset may also be a useful benchmark for coherent multi-sentence generation. We perform a first-pass visual analysis that exposes clusters of differing pixels as a proxy for object-level differences. We propose a model that captures visual salience by using a latent variable to align clusters of differing pixels with output sentences. We find that, for both single-sentence generation and as well as multi-sentence generation, the proposed model outperforms the models that use attention alone.

VASR: Visual Analogies of Situation Recognition

Abstract:
A core process in human cognition is analogical mapping: the ability to identify a similar relational structure between different situations.
We introduce a novel task, Visual Analogies of Situation Recognition, adapting the classical word-analogy task into the visual domain. Given a triplet of images, the task is to select an image candidate B' that completes the analogy (A to A' is like B to what?). Unlike previous work on visual analogy that focused on simple image transformations, we tackle complex analogies requiring understanding of scenes.

We leverage situation recognition annotations and the CLIP model to generate a large set of 500k candidate analogies. Crowdsourced annotations for a sample of the data indicate that humans agree with the dataset label ~80% of the time (chance level 25%). Furthermore, we use human annotations to create a gold-standard dataset of 3,820 validated analogies.
Our experiments demonstrate that state-of-the-art models do well when distractors are chosen randomly (~86%), but struggle with carefully chosen distractors (~53%, compared to 90% human accuracy). We hope our dataset will encourage the development of new analogy-making models. Website: https://vasr-dataset.github.io/

WinoGAViL: Gamified Association Benchmark to Challenge Vision-and-Language Models

Abstract: While vision-and-language models perform well on tasks such as visual question answering, they struggle when it comes to basic human commonsense reasoning skills. In this work, we introduce WinoGAViL: an online game of vision-and-language associations (e.g., between werewolves and a full moon), used as a dynamic evaluation benchmark. Inspired by the popular card game Codenames, a spymaster gives a textual cue related to several visual candidates, and another player tries to identify them. Human players are rewarded for creating associations that are challenging for a rival AI model but still solvable by other human players. We use the game to collect 3.5K instances, finding that they are intuitive for humans (>90% Jaccard index) but challenging for state-of-the-art AI models, where the best model (ViLT) achieves a score of 52%, succeeding mostly where the cue is visually salient. Our analysis as well as the feedback we collect from players indicate that the collected associations require diverse reasoning skills, including general knowledge, common sense, abstraction, and more. We release the dataset, the code and the interactive game, allowing future data collection that can be used to develop models with better association abilities.

IRFL: Image Recognition of Figurative Language

Abstract: Figures of speech such as metaphors, similes, and idioms are integral parts of human communication. They are ubiquitous in many forms of discourse, allowing people to convey complex, abstract ideas and evoke emotion. As figurative forms are often conveyed through multiple modalities (e.g., both text and images), understanding multimodal figurative language is an important AI challenge, weaving together profound vision, language, commonsense and cultural knowledge. In this work, we develop the Image Recognition of Figurative Language (IRFL) dataset. We leverage human annotation and an automatic pipeline we created to generate a multimodal dataset, and introduce two novel tasks as a benchmark for multimodal figurative language understanding. We experimented with state-of-the-art vision and language models and found that the best (22%) performed substantially worse than humans (97%). We release our dataset, benchmark, and code, in hopes of driving the development of models that can better understand figurative language.

IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoning

Abstract: Current visual question answering (VQA) tasks mainly consider answering human-annotated questions for natural images. However, aside from natural images, abstract diagrams with semantic richness are still understudied in visual understanding and reasoning research. In this work, we introduce a new challenge of Icon Question Answering (IconQA) with the goal of answering a question in an icon image context. We release IconQA, a large-scale dataset that consists of 107,439 questions and three sub-tasks: multi-image-choice, multi-text-choice, and filling-in-the-blank. The IconQA dataset is inspired by real-world diagram word problems that highlight the importance of abstract diagram understanding and comprehensive cognitive reasoning. Thus, IconQA requires not only perception skills like object recognition and text understanding, but also diverse cognitive reasoning skills, such as geometric reasoning, commonsense reasoning, and arithmetic reasoning. To facilitate potential IconQA models to learn semantic representations for icon images, we further release an icon dataset Icon645 which contains 645,687 colored icons on 377 classes. We conduct extensive user studies and blind experiments and reproduce a wide range of advanced VQA methods to benchmark the IconQA task. Also, we develop a strong IconQA baseline Patch-TRM that applies a pyramid cross-modal Transformer with input diagram embeddings pre-trained on the icon dataset. IconQA and Icon645 are available at https://iconqa.github.io.

Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation

Abstract: The ability to collect a large dataset of human preferences from text-to-image users is usually limited to companies, making such datasets inaccessible to the public. To address this issue, we create a web app that enables text-to-image users to generate images and specify their preferences. Using this web app we build Pick-a-Pic, a large, open dataset of text-to-image prompts and real users' preferences over generated images. We leverage this dataset to train a CLIP-based scoring function, PickScore, which exhibits superhuman performance on the task of predicting human preferences. Then, we test PickScore's ability to perform model evaluation and observe that it correlates better with human rankings than other automatic evaluation metrics. Therefore, we recommend using PickScore for evaluating future text-to-image generation models, and using Pick-a-Pic prompts as a more relevant dataset than MS-COCO. Finally, we demonstrate how PickScore can enhance existing text-to-image models via ranking.

We thank Pang Wei Koh, Ashima Suvarna, Nitzan Guetta and Roee Aharoni for their valuable feedback. Hritik Bansal is supported in part by AFOSR MURI grant FA9550-22-1-0380.
RT is supported by the NSF GRFP under Grant No. DGE 1656518.

Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan,
Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, et al.
A general language assistant as a laboratory for alignment.
arXiv preprint arXiv:2112.00861, 2021.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela
Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al.
Training language models to follow instructions with human feedback.
Advances in Neural Information Processing Systems,
35:27730–27744, 2022.
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian
Lester, Nan Du, Andrew M Dai, and Quoc V Le.
Finetuned language models are zero-shot learners.
arXiv preprint arXiv:2109.01652, 2021.
Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza
Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Anjana
Arunkumar, David Stap, et al.
Super-naturalinstructions: Generalization via declarative
instructions on 1600+ nlp tasks.
In Proceedings of the 2022 Conference on Empirical Methods in
Natural Language Processing, pages 5085–5109, 2022a.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos
Guestrin, Percy Liang, and Tatsunori B. Hashimoto.
Stanford alpaca: An instruction-following llama model.
https://github.com/tatsu-lab/stanford_alpaca, 2023.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin
Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and
Eric P. Xing.
Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt
quality, March 2023.
URL https://lmsys.org/blog/2023-03-30-vicuna/.
OpenAI.
Gpt-4 technical report.
arXiv, 2023.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla
Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell,
et al.
Language models are few-shot learners.
Advances in neural information processing systems,
33:1877–1901, 2020.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra,
Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian
Gehrmann, et al.
Palm: Scaling language modeling with pathways.
arXiv preprint arXiv:2204.02311, 2022.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne
Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric
Hambro, Faisal Azhar, et al.
Llama: Open and efficient foundation language models.
arXiv preprint arXiv:2302.13971, 2023.
Anas Awadalla, Irena Gao, Joshua Gardner, Jack Hessel, Yusuf Hanafy, Wanrong
Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Jenia Jitsev, Simon
Kornblith, Pang Wei Koh, Gabriel Ilharco, Mitchell Wortsman, and Ludwig
Schmidt.
Openflamingo, March 2023.
URL https://doi.org/10.5281/zenodo.7733589.
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr,
Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds,
Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina
Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andy Brock,
Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira,
Oriol Vinyals, Andrew Zisserman, and Karen Simonyan.
Flamingo: a visual language model for few-shot learning.
ArXiv, abs/2204.14198, 2022.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee.
Visual instruction tuning, 2023a.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao,
Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi.
Instructblip: Towards general-purpose vision-language models with
instruction tuning.
arXiv preprint arXiv:2305.06500, 2023.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi.
Blip-2: Bootstrapping language-image pre-training with frozen image
encoders and large language models.
arXiv preprint arXiv:2301.12597, 2023a.
Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, Wei
Zhang, Pan Lu, Conghui He, Xiangyu Yue, et al.
Llama-adapter v2: Parameter-efficient visual instruction model.
arXiv preprint arXiv:2304.15010, 2023.
Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang
Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al.
mplug-owl: Modularization empowers large language models with
multimodality.
arXiv preprint arXiv:2304.14178, 2023.
Yixuan Su, Tian Lan, Huayang Li, Jialu Xu, Yan Wang, and Deng Cai.
Pandagpt: One model to instruction-follow them all.
arXiv preprint arXiv:2305.16355, 2023.
Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh.
Making the V in vqa matter: Elevating the role of image
understanding in visual question answering.
In Proceedings of the IEEE conference on computer vision and
pattern recognition, pages 6904–6913, 2017.
Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva
Ramanan, Piotr Dollár, and C Lawrence Zitnick.
Microsoft coco: Common objects in context.
In Computer Vision–ECCV 2014: 13th European Conference,
Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages
740–755. Springer, 2014.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra,
C Lawrence Zitnick, and Devi Parikh.
Vqa: Visual question answering.
In Proceedings of the IEEE international conference on computer
vision, pages 2425–2433, 2015.
Dong Huk Park, Trevor Darrell, and Anna Rohrbach.
Robust change captioning.
In Proceedings of the IEEE/CVF International Conference on
Computer Vision, pages 4624–4633, 2019.
Jordi Pont-Tuset, Jasper Uijlings, Soravit Changpinyo, Radu Soricut, and
Vittorio Ferrari.
Connecting vision and language with localized narratives.
In Computer Vision–ECCV 2020: 16th European Conference,
Glasgow, UK, August 23–28, 2020, Proceedings, Part V 16, pages 647–664.
Springer, 2020.
Yushi Hu, Hang Hua, Zhengyuan Yang, Weijia Shi, Noah A Smith, and Jiebo Luo.
Promptcap: Prompt-guided task-aware image captioning.
arXiv preprint arXiv:2211.09699, 2022.
Arpad E Elo.
The proposed uscf rating system. its development, theory, and
applications.
Chess Life, 22(8):242–247, 1967.
Lianmin Zheng, Ying Sheng, Wei-Lin Chiang, Hao Zhang, Joseph E. Gonzalez, and
Ion Stoica.
Chatbot arena: Benchmarking llms in the wild with elo ratings.
2023.
URL https://lmsys.org/blog/2023-05-03-arena/.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu.
Bleu: a method for automatic evaluation of machine translation.
In ACL, 2002.
Chin-Yew Lin.
Rouge: A package for automatic evaluation of summaries.
Text Summarization Branches Out, 2004.
Satanjeev Banerjee and Alon Lavie.
METEOR: an automatic metric for mt evaluation with improved
correlation with human judgments.
In ACL workshop on Evaluation Measures for MT and
Summarization, 2005.
Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh.
Cider: Consensus-based image description evaluation.
In CVPR, 2015.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi.
BERTScore: Evaluating text generation with BERT.
In ICLR, 2020.
Zhiyang Xu, Ying Shen, and Lifu Huang.
Multiinstruct: Improving multi-modal zero-shot learning via
instruction tuning.
arXiv preprint arXiv:2212.10773, 2022.
Lei Li, Yuwei Yin, Shicheng Li, Liang Chen, Peiyi Wang, Shuhuai Ren, Mukai Li,
Yazheng Yang, Jingjing Xu, Xu Sun, et al.
M3it: A large-scale dataset towards multi-modal multilingual
instruction tuning.
arXiv preprint arXiv:2306.04387, 2023b.
Peng Xu, Wenqi Shao, Kaipeng Zhang, Peng Gao, Shuo Liu, Meng Lei, Fanqing Meng,
Siyuan Huang, Yu Qiao, and Ping Luo.
Lvlm-ehub: A comprehensive evaluation benchmark for large
vision-language models.
arXiv preprint arXiv:2306.09265, 2023.
Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang.
Aligning large multi-modal model with robust instruction tuning.
arXiv preprint arXiv:2306.14565, 2023b.
Alane Suhr, Stephanie Zhou, Ally Zhang, Iris Zhang, Huajun Bai, and Yoav Artzi.
A corpus for reasoning about natural language grounded in
photographs.
arXiv preprint arXiv:1811.00491, 2018.
Sarah Wiegreffe, Jack Hessel, Swabha Swayamdipta, Mark Riedl, and Yejin Choi.
Reframing human-ai collaboration for generating free-text
explanations.
arXiv preprint arXiv:2112.08674, 2021.
Andy Zeng, Maria Attarian, Brian Ichter, Krzysztof Choromanski, Adrian Wong,
Stefan Welker, Federico Tombari, Aveek Purohit, Michael Ryoo, Vikas
Sindhwani, et al.
Socratic models: Composing zero-shot multimodal reasoning with
language.
arXiv preprint arXiv:2204.00598, 2022.
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel
Khashabi, and Hannaneh Hajishirzi.
Self-instruct: Aligning language model with self generated
instructions.
arXiv preprint arXiv:2212.10560, 2022b.
Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi.
You only look once: Unified, real-time object detection.
In Proceedings of the IEEE conference on computer vision and
pattern recognition, pages 779–788, 2016.
Nitzan Bitton-Guetta, Yonatan Bitton, Jack Hessel, Ludwig Schmidt, Yuval
Elovici, Gabriel Stanovsky, and Roy Schwartz.
Breaking common sense: Whoops! a vision-and-language benchmark of
synthetic and compositional images.
arXiv preprint arXiv:2303.07274, 2023.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny.
Minigpt-4: Enhancing vision-language understanding with advanced
large language models.
arXiv preprint arXiv:2304.10592, 2023.
Chenfei Wu, Shengming Yin, Weizhen Qi, Xiaodong Wang, Zecheng Tang, and Nan
Duan.
Visual chatgpt: Talking, drawing and editing with visual foundation
models.
arXiv preprint arXiv:2303.04671, 2023.
Tao Gong, Chengqi Lyu, Shilong Zhang, Yudong Wang, Miao Zheng, Qian Zhao,
Kuikun Liu, Wenwei Zhang, Ping Luo, and Kai Chen.
Multimodal-gpt: A vision and language model for dialogue with humans.
arXiv preprint arXiv:2305.04790, 2023a.
Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Jingkang Yang, and Ziwei Liu.
Otter: A multi-modal model with in-context instruction tuning.
arXiv preprint arXiv:2305.03726, 2023c.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer
Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.
Roberta: A robustly optimized bert pretraining approach.
arXiv preprint arXiv:1907.11692, 2019.
Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba,
Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto.
Alpacafarm: A simulation framework for methods that learn from human
feedback, 2023.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and
Denny Zhou.
Chain-of-thought prompting elicits reasoning in large language
models.
In NeurIPS, 2022.
URL https://arxiv.org/abs/2201.11903.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke
Iwasawa.
Large language models are zero-shot reasoners.
In NeurIPS, 2022.
URL https://arxiv.org/abs/2205.11916.
Rohan Pandey, Rulin Shao, Paul Pu Liang, Ruslan Salakhutdinov, and
Louis-Philippe Morency.
Cross-modal attention congruence regularization for vision-language
relation alignment.
arXiv preprint arXiv:2212.10549, 2022.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross
Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell
Wortsman, et al.
Laion-5b: An open large-scale dataset for training next generation
image-text models.
arXiv preprint arXiv:2210.08402, 2022.
Xinyang Geng, Arnav Gudibande, Hao Liu, Eric Wallace, Pieter Abbeel, Sergey
Levine, and Dawn Song.
Koala: A dialogue model for academic research.
Blog post, April 2023.
URL https://bair.berkeley.edu/blog/2023/04/03/koala/.
Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao.
Instruction tuning with gpt-4.
arXiv preprint arXiv:2304.03277, 2023.
Da Yin, Xiao Liu, Fan Yin, Ming Zhong, Hritik Bansal, Jiawei Han, and Kai-Wei
Chang.
Dynosaur: A dynamic growth paradigm for instruction-tuning data
curation.
arXiv preprint arXiv:2305.14327, 2023a.
Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe
Ma, Avia Efrat, Ping Yu, Lili Yu, et al.
Lima: Less is more for alignment.
arXiv preprint arXiv:2305.11206, 2023.
Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li,
Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, et al.
Grounded language-image pre-training.
In Proceedings of the IEEE/CVF Conference on Computer Vision
and Pattern Recognition, pages 10965–10975, 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn
Ommer.
High-resolution image synthesis with latent diffusion models.
In Proceedings of the IEEE/CVF Conference on Computer Vision
and Pattern Recognition, pages 10684–10695, 2022.
Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Ehsan Azarnasab, Faisal
Ahmed, Zicheng Liu, Ce Liu, Michael Zeng, and Lijuan Wang.
Mm-react: Prompting chatgpt for multimodal reasoning and action.
arXiv preprint arXiv:2303.11381, 2023.
Dídac Surís, Sachit Menon, and Carl Vondrick.
Vipergpt: Visual inference via python execution for reasoning.
arXiv preprint arXiv:2303.08128, 2023.
Tanmay Gupta and Aniruddha Kembhavi.
Visual programming: Compositional visual reasoning without training.
In Proceedings of the IEEE/CVF Conference on Computer Vision
and Pattern Recognition, pages 14953–14962, 2023.
Pan Lu, Baolin Peng, Hao Cheng, Michel Galley, Kai-Wei Chang, Ying Nian Wu,
Song-Chun Zhu, and Jianfeng Gao.
Chameleon: Plug-and-play compositional reasoning with large language
models.
arXiv preprint arXiv:2304.09842, 2023.
Tao Gong, Chengqi Lyu, Shilong Zhang, Yudong Wang, Miao Zheng, Qian Zhao,
Kuikun Liu, Wenwei Zhang, Ping Luo, and Kai Chen.
Multimodal-gpt: A vision and language model for dialogue with humans,
2023b.
Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma,
Tengchao Lv, Lei Cui, Owais Khan Mohammed, Qiang Liu, et al.
Language is not all you need: Aligning perception with language
models.
arXiv preprint arXiv:2302.14045, 2023.
Drew A Hudson and Christopher D Manning.
Gqa: A new dataset for real-world visual reasoning and compositional
question answering.
In Proceedings of the IEEE/CVF conference on computer vision
and pattern recognition, pages 6700–6709, 2019.
Thomas Liao, Rohan Taori, Inioluwa Deborah Raji, and Ludwig Schmidt.
Are we learning yet? a meta review of evaluation failures across
machine learning.
In Thirty-fifth Conference on Neural Information Processing
Systems Datasets and Benchmarks Track (Round 2), 2021.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma,
Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al.
Imagenet large scale visual recognition challenge.
International journal of computer vision, 115:211–252, 2015.
Alex Krizhevsky, Geoffrey Hinton, et al.
Learning multiple layers of features from tiny images.
2009.
Pranav Rajpurkar, Robin Jia, and Percy Liang.
Know what you don't know: Unanswerable questions for squad.
arXiv preprint arXiv:1806.03822, 2018.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning,
Andrew Y Ng, and Christopher Potts.
Recursive deep models for semantic compositionality over a sentiment
treebank.
In Proceedings of the 2013 conference on empirical methods in
natural language processing, pages 1631–1642, 2013.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R
Bowman.
Glue: A multi-task benchmark and analysis platform for natural
language understanding.
arXiv preprint arXiv:1804.07461, 2018.
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael,
Felix Hill, Omer Levy, and Samuel Bowman.
Superglue: A stickier benchmark for general-purpose language
understanding systems.
Advances in neural information processing systems, 32, 2019.
Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles
Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff,
et al.
A framework for few-shot language model evaluation.
Version v0. 0.1. Sept, 2021.
Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu,
Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar,
et al.
Holistic evaluation of language models.
arXiv preprint arXiv:2211.09110, 2022.
John Miller, Karl Krauth, Benjamin Recht, and Ludwig Schmidt.
The effect of natural distribution shift on question answering
models.
In International Conference on Machine Learning, pages
6905–6916. PMLR, 2020.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar.
Do cifar-10 classifiers generalize to cifar-10?
arXiv preprint arXiv:1806.00451, 2018.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar.
Do imagenet classifiers generalize to imagenet?
In International conference on machine learning, pages
5389–5400. PMLR, 2019.
Rebecca Roelofs, Vaishaal Shankar, Benjamin Recht, Sara Fridovich-Keil, Moritz
Hardt, John Miller, and Ludwig Schmidt.
A meta-analysis of overfitting in machine learning.
Advances in Neural Information Processing Systems, 32, 2019.
Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen.
A survey on multimodal large language models.
arXiv preprint arXiv:2306.13549, 2023b.
Inbal Magar and Roy Schwartz.
Data contamination: From memorization to exploitation.
arXiv preprint arXiv:2203.08242, 2022.
Alon Jacovi, Avi Caciularu, Omer Goldman, and Yoav Goldberg.
Stop uploading test data in plain text: Practical strategies for
mitigating data contamination by evaluation benchmarks.
arXiv preprint arXiv:2305.10160, 2023.
Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi.
From recognition to cognition: Visual commonsense reasoning.
In Proceedings of the IEEE/CVF conference on computer vision
and pattern recognition, pages 6720–6731, 2019.
Da Yin, Liunian Harold Li, Ziniu Hu, Nanyun Peng, and Kai-Wei Chang.
Broaden the vision: Geo-diverse visual commonsense reasoning.
arXiv preprint arXiv:2109.06860, 2021.
Jack Hessel, Ana Marasović, Jena D Hwang, Lillian Lee, Jeff Da, Rowan
Zellers, Robert Mankoff, and Yejin Choi.
Do androids laugh at electric sheep? humor" understanding" benchmarks
from the new yorker caption contest.
arXiv preprint arXiv:2209.06293, 2022.
Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei,
C Lawrence Zitnick, and Ross Girshick.
Clevr: A diagnostic dataset for compositional language and elementary
visual reasoning.
In Proceedings of the IEEE conference on computer vision and
pattern recognition, pages 2901–2910, 2017.
Anya Ji, Noriyuki Kojima, Noah Rush, Alane Suhr, Wai Keen Vong, Robert D
Hawkins, and Yoav Artzi.
Abstract visual reasoning with tangram shapes.
arXiv preprint arXiv:2211.16492, 2022.
Shivam Sharma, Siddhant Agarwal, Tharun Suresh, Preslav Nakov, Md Shad Akhtar,
and Tanmoy Charkraborty.
What do you meme? generating explanations for visual semantic role
labelling in memes.
arXiv preprint arXiv:2212.00715, 2022.
Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu,
Oyvind Tafjord, Peter Clark, and Ashwin Kalyan.
Learn to explain: Multimodal reasoning via thought chains for science
question answering.
In The 36th Conference on Neural Information Processing Systems
(NeurIPS), 2022.
Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi.
Ok-vqa: A visual question answering benchmark requiring external
knowledge.
In Proceedings of the IEEE/cvf conference on computer vision
and pattern recognition, pages 3195–3204, 2019.
Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and
Roozbeh Mottaghi.
A-okvqa: A benchmark for visual question answering using world
knowledge.
arXiv, 2022.
Jeffrey P Bigham, Chandrika Jayant, Hanjie Ji, Greg Little, Andrew Miller,
Robert C Miller, Robin Miller, Aubrey Tatarowicz, Brandyn White, Samual
White, et al.
Vizwiz: nearly real-time answers to visual questions.
In Proceedings of the 23nd annual ACM symposium on User
interface software and technology, pages 333–342, 2010.
Oleksii Sidorov, Ronghang Hu, Marcus Rohrbach, and Amanpreet Singh.
Textcaps: a dataset for image captioning with reading comprehension.
In Computer Vision–ECCV 2020: 16th European Conference,
Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16, pages 742–758.
Springer, 2020.
Benno Krojer, Vaibhav Adlakha, Vibhav Vineet, Yash Goyal, Edoardo Ponti, and
Siva Reddy.
Image retrieval from contextual descriptions.
arXiv preprint arXiv:2203.15867, 2022.
Harsh Jhamtani and Taylor Berg-Kirkpatrick.
Learning to describe differences between pairs of similar images.
arXiv preprint arXiv:1808.10584, 2018.
Yonatan Bitton, Ron Yosef, Eli Strugo, Dafna Shahaf, Roy Schwartz, and Gabriel
Stanovsky.
Vasr: Visual analogies of situation recognition.
arXiv preprint arXiv:2212.04542, 2022a.
Yonatan Bitton, Nitzan Bitton Guetta, Ron Yosef, Yuval Elovici, Mohit Bansal,
Gabriel Stanovsky, and Roy Schwartz.
Winogavil: Gamified association benchmark to challenge
vision-and-language models.
Advances in Neural Information Processing Systems,
35:26549–26564, 2022b.
Ron Yosef, Yonatan Bitton, and Dafna Shahaf.
Irfl: Image recognition of figurative language.
arXiv preprint arXiv:2303.15445, 2023.
Pan Lu, Liang Qiu, Jiaqi Chen, Tony Xia, Yizhou Zhao, Wei Zhang, Zhou Yu,
Xiaodan Liang, and Song-Chun Zhu.
Iconqa: A new benchmark for abstract diagram understanding and visual
language reasoning.
arXiv preprint arXiv:2110.13214, 2021.
Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and
Omer Levy.
Pick-a-pic: An open dataset of user preferences for text-to-image
generation.
arXiv preprint arXiv:2305.01569, 2023.

Appendix

License and Intended Use
The VisIT-Bench dataset, along with its various contributions such as instructions, reference outputs, and model ranking annotations, is licensed under the Creative Commons Attribution 4.0 International License (CC BY 4.0). This license applies to all the images we have directly contributed, each of which carries a public license specification in the “public images metadata” field within the dataset sheets. However, the dataset also incorporates images sourced from pre-existing collections. For these images, the original licensing terms are respected and remain applicable.
VisIT-Bench's primary purpose is to function as a dynamic benchmark that continuously evolves and evaluates instruction-following vision-language models. In the current landscape, commercial chatbots are often trained on non-disclosed and non-public datasets, which raises concerns about potential data contamination and inadvertent training on our evaluation data {{cite:662035a}}. This risk is further highlighted by recent studies {{cite:391b41d}}, {{cite:a3b883b}}. To mitigate such concerns, we have chosen to withhold the complete VisIT-Bench test set from public disclosure, while still making the images and instructions available for direct download. Researchers, however, can utilize VisIT-Bench to its full potential as a dynamic benchmark by submitting their model predictions for evaluation. We will assess their models using the undisclosed test set, ensuring the ongoing evolution of the benchmark. Moreover, we are open to releasing the test data upon receiving reasonable and justified requests, particularly when additional analysis is necessary, provided that requesters agree to our non-contamination policy which prohibits the use of this data for training commercial chatbots. This approach strikes a balance between the need for robust model evaluation and the mitigation of potential data contamination.

Dataset Analysis
Count of the various COCO objects present in the VisIT-Bench images on a log-scale. The object detection was performed by Yolov5-Large {{cite:6b37420}}. We observe that the object `person' occurs most of the time in the dataset images i.e., 875 times.
{{figure:823bd59c-ddda-484a-bad7-fb62c0a97ec6}}
Interfaces for Collecting Human Annotations
In this section, we provide the templates we used to collect human annotations for the instruction generation (fig:templates-instruction-generation), the dense caption generation (fig:templates-dense-caption), the model verification (fig:templates-human-veri-pos and fig:templates-human-veri-neg), and the model rating (fig:templates-model-rating).
An interface that collects the image and corresponding instruction.
{{figure:30c6de22-136b-47ea-9e80-2ed907db6a1c}}An interface that collects the dense caption given an image-and-instruction pair.
{{figure:ac90365b-ce90-4ff8-87c2-04ff05d88128}}A sample of the model verification where the GPT4 follows the instruction correctly.
{{figure:39fcf508-9ca3-466e-8c13-0b6938fb9557}}A sample of the model verification where the GPT4 failed to follow the instruction due to its incorrect reasoning.
{{figure:854eae47-cfb0-43e5-a79a-a746a10c85d7}}An interface that collects the feedback of the model rating.
{{figure:316c43c3-1cd9-4811-aed5-8445118eb191}}
Existing Datasets incorporated in VisIT-Bench
In Table {{table:d26bbad7-391d-4572-987d-a234c68b9d55}} , we listed the existing datasets that are incoprated in our VisIT-Bench. Among these datasets, 15 contain a single image in each sample pair, and 10 require reasoning based on multiple images.
List of existing datasets in VisIT-Bench, categorized as single and multiple image datasets.

15*Single
Dataset
Topic

VQA {{cite:298e4c6}}
Visual Question Answering

VCR {{cite:e5abc64}}
Cognition-level Visual Understanding

GD-VCR {{cite:281ef47}}
Geo-Diverse Commonsense Reasoning

WHOOPS {{cite:d6a5d11}}
What Makes this Image Strange

Newyork Caption {{cite:9f8c4bd}}
Humor Understanding

CLEVR {{cite:46c06c0}}
Visual Question Answering

Kilogram {{cite:76a0b35}}
Tangrams Identification

Harmful Memes {{cite:d015189}}
Memes Understanding

ScienceQA {{cite:febdaeb}}
Science Question Answering

OK-VQA {{cite:1e102f4}}
Outside Knowledge Visual Question Answering

AOK-VQA {{cite:fa95ba5}}
Outside Knowledge Visual Question

AOK-VQA {{cite:fa95ba5}}
Question Generation

VizWiz {{cite:c33422e}}
Visual Question Answering

GQA {{cite:557312c}}
Visual Question Answering on Scene Graphs

TextCaps {{cite:2c33f96}}
Visual Question Answering on Texts
10*Multiple
Robust Change Captioning {{cite:e7768b7}}
Describing What has Change in a Scene

NLVR2 {{cite:5f9af87}}
Testing Visual Language Bias

ImageCoDE {{cite:fe0528e}}
Image Retrieval

Spot-the-Diff {{cite:c750624}}
Identifying Differences

VASR {{cite:17733c4}}
Visual Analogies

WinoGavil {{cite:6ca7a0a}}
Visual Associations

IRFL (Metaphor) {{cite:899424f}}
Figurative Speech Understanding

IRFL (Idioms) {{cite:899424f}}
Figurative Speech Understanding

IconQA {{cite:037c73a}}
Abstract Diagram Understanding

Pick-a-Pic{{cite:bfe051f}}
Text-to-Image User Preferences
{{table:015a22d5-f4ee-471f-ab87-2b2278b80997}}{{table:d26bbad7-391d-4572-987d-a234c68b9d55}}List of skills and existing datasets in VisIT-Bench
`scienceqa', `ocr math', `recognition', `okvqa', `house plan understanding', `nlvr2', `gardening tips', `textcaps', `architectural styles', `dressing sense', `winoground', `food recipe', `paper folding', `whoops', `spot the diff', `winogavil', `imagecode', `exercise', `art knowledge', `gqa', `physical knowledge', `contextual knowledge of events', `home renovation', `aokvqa', `animals', `vasr', `counting', `board games', `solving geometry problems', `who to call?', `clevr', `building materials', `hazard identification', `pickapick', `astronomy', `figurative speech explanation', `write a story', `gestures understanding', `newyork', `cultural knowledge', `aokvqg', `traffic sign identification', `pop culture', `fashion products', `harmful memes', `write a poem', `vizwiz', `guesstimate of capacity', `location understanding', `graph reasoning', `vqa', `game playing', `differently abled', `chemical identification', `history knowledge', `climate and weather understanding', `irfl metaphor', `human emotion recognition', `medical', `gd vcr', `vcr', `technical support', `catchy titles', `kilogram', `anagrams', `color', `tour guide', `directions', `irfl idiom', `rcc'
{{table:5245649c-5416-4fe2-9926-ae02c65f5d21}}{{table:dbeada08-9d9e-4903-bdd9-af050a72a900}}
Elo Rating
For many years, the Elo rating has been popular in ranking players in zero-sum games such as chess {{cite:d0604f9}}. Recently, it has been adopted to rate large language models (LLMs) against each other on the user instructions. In this work, we adopt the same strategy to rank a set of instruction-following vision-language models, that can grow dynamically with further advances in the field.
Given two multimodal chatbots {{formula:abddcc87-0d3f-49ef-8817-dab378e75207}}  and {{formula:beda3b7e-d62a-4069-a3e2-4040e6b558db}}  with their absolute Elo rating {{formula:bcee3201-0f3c-43e0-8476-00c63f312510}}  and {{formula:d3f87040-3d3e-43f7-9e6c-7f12e59336eb}} , respectively. Simply put, the probability of {{formula:ba0081b8-ffe2-413c-80b7-891956620dba}}  winning over {{formula:c7de8ffc-c86e-4f80-bda5-03bbb229daa2}}  in a head-to-head battle is given by:
{{formula:296ab989-c449-4023-bdf6-be2408827760}}
In practice, calculating the Elo rating requires us to set hyperparameters to decide the weightage for each win and loss in a head-to-head battle between two models. In our work, we use the open implementation of Elo for LLMs by FastChat at https://github.com/lm-sys/FastChat/blob/main/fastchat/serve/monitor/elo_analysis.py.

GPT-4 Pairwise Evaluation Prompts
The specific prompts we use to extract pairwise judgements from our language model are provided in Table {{figure:e157ec95-4b23-4acb-8351-7ed76bcdb6ea}}  (reference-free version) and Table {{figure:0b30f1a0-6bab-4f54-85ab-8b25fdb8ae19}}  (reference-backed version). When applied to GPT-4 {{cite:d746af7}}, these prompts usually solicit a definitive pairwise response by the model. But, in some cases, the model either produces a pairwise judgement in an unexpected format, or, refuses to issue a judgement at all. For cases like these, we issue an additional query to ChatGPT to extract an answer (or decide there is no answer) using an additional prompt, given in Table {{figure:d58cf090-de6f-409a-9fa0-6c955711e5e4}} . If after this step there is still no definitive pairwise judgment, we call the result a tie.
The prompt we use for reference-free evaluation, along with a sample instance/completion from GPT-4 {{cite:d746af7}}.
system prompt (human authored)

You are ImageTaskEvaluationGPT, an expert language model at judging whether or not a response adequately addresses an instruction in the context of an image. More specifically, you will be given the following:

1. An image context: This will describe the contents of an image with sufficient detail to address the instruction.
2. An instruction: This is a question, an imperative request, or something similar about the image which requires a response.
3. Two responses, response A and response B: These two responses attempt to address the instruction in the context of the image.
Your job is to judge whether response A or response B better. A and B are randomly ordered.

Some things to remember:

- Even though you are just a language model, the image description will be sufficiently detailed so that your judgements can be accurate.
- You are capable of judging response quality, accounting for important factors like correctness, relevance, fluency, specificity, etc.
- You think step-by-step, but ultimately respond with "Response A" or "Response B"
user (human authored)

I will describe an image to you, and provide an instruction. Then, I will give you two candidate responses that address the instruction in the context of the image: these will be labelled "Response A" and "Response B". Your job is to first reason step-by-step about which response is best in terms of accuracy, specificity, fluency, etc. After reasoning step-by-step and comparing the pros/cons of each response, in the end, respond with "Overall, Response X is better." where X is either A or B.

assistant (human authored)

Sure, please provide the image context, the instruction, and the two candidate responses, Response A and Response B. Then, I will think step-by-step and provide my ultimate judgement as to which response is better.

user (from human-authored template)

OK. Here is the image description, the instruction, and the two response options, Response A and Response B.
Image context: A skier is wearing a green jacket, burgundy pants, white shoes, white gloves, and a gray beanie. There are people sitting on the background with their snowboards.

Instruction:
Answer the following question based on the provided image: Are the trousers the same color as the shoes?

Response A: no

Response B: No, the trousers are not the same color as the shoes. The trousers are burgundy, while the shoes are white.

Think step-by-step and finish your response with "Overall, Response X is better." where X is either A or B.

assistant (machine generated; used to derive the prediction)

Step 1: Identify the key information from the image description.

The skier is wearing:
- Burgundy pants
- White shoes

Step 2: Assess the accuracy of each response in addressing the instruction.

Both Response A and Response B convey that the trousers and shoes are not of the same color.

Step 3: Compare specificity and detail of each response.

Response A is concise and provides a straightforward answer, but lacks detail.
Response B provides a clear, comprehensive answer that includes the color of both the trousers and shoes.

Step 4: Consider the fluency and correctness.

Both responses are fluent and grammatically correct.

Overall, Response B is better.
{{table:a286d5ab-1a10-4da3-a5a9-8353dfd0b946}}
{{figure:e157ec95-4b23-4acb-8351-7ed76bcdb6ea}}The prompt we use for reference-backed evaluation, along with a sample instance/completion from GPT-4 {{cite:d746af7}}.
system prompt (human authored)

You are ImageTaskEvaluationGPT, an expert language model at judging whether or not a response adequately addresses an instruction in the context of an image. More specifically, you will be given the following:

1. An image context: This will describe the contents of an image with sufficient detail to address the instruction.
2. An instruction: This is a question, an imparative request, or something similar about the image which requires a response.
3. A reference output: This is a high-quality example output that humans have judged to be an accurate response for the input instruction.
4. Two responses, response A and response B: These two responses attempt to address the instruction in the context of the image.

Your job is to judge whether response A or response B better. A and B are randomly ordered. Some things to remember:
- Even though you are just a language model, the image description will be sufficiently detailed so that your judgements can be accurate.
- Take the high-quality reference into account when making your judgements, but remember: some instructions are more open-ended than others, so for those cases, a high quality response can differ from the reference.
- You are capable of judging response quality, accounting for important factors like correctness, relevance, fluency, specificity, etc.
- You think step-by-step, but ultimately respond with "Response A" or "Response B"
user (human authored)

I will describe an image to you, and provide an instruction. Then, I will provide a reference output which is an example of a high quality output for that instruction in the context of the image. Then, I will give you two candidate responses that address the instruction in the context of the image: these will be labelled "Response A" and "Response B". Your job is to first reason step-by-step about which response is best in terms of accuracy, specificity, fluency, etc. After reasoning step-by-step and comparing the pros/cons of each response, in the end, respond with "Overall, Response X is better." where X is either A or B.

assistant (human authored)

Sure, please provide the image context, the instruction, the reference, and the two candidate responses, Response A and Response B. Then, I will think step-by-step and provide my ultimate judgement as to which response is better.

user (from human-authored template)

OK. Here is the image description, the instruction, the high-quality reference, and the two response options, Response A and Response B.
Image context: There is a white plate that has on it a small pizza with many topics, mostly unrecognizable, due to the pie being very well done, charred to the point of almost being burnt, and the crust blackened in many places.

Instruction:
Someone told the creator of this item that they are very talented. What is the implied meaning?
High quality reference: The implied meaning is that the person complimenting the creator is being sarcastic, as the pizza is overcooked and almost burnt, which is not a sign of talent.

Response A: When someone says that they are "very talented" in the context of the image, it implies that the person responsible for creating the pizza is skilled in the craft and able to create an appealing and tasty dish. The person praising the pizza's creator recognizes their culinary skills, which can be seen through the combination of ingredients and the overall look of the finished dish. The praise is a form of appreciation for their efforts and the enjoyable result they achieved.
Response B: It is difficult to determine the implied meaning without further context or information about the image. It appears to be a pizza that has been burnt or charred on the surface and has various toppings, including pepperoni, onion, and mushrooms. The image is shown on a white plate or surface. The implied meaning of this image could be that it is a dish that has been cooked badly or that it is a dish with a unique or unusual combination of ingredients.

Think step-by-step and finish your response with "Overall, Response X is better." where X is either A or B.

assistant (machine generated; used to derive the prediction)

Step 1: Comparing both responses with the high-quality reference, it's clear that Response A describes the pizza as well-made and appealing, while Response B is more focused on the fact that it is burnt or charred, somewhat aligning it with the reference.
Step 2: Evaluating specificity, Response A defines the creator as skilled, even though the image description contradicts this by stating that the pizza is overcooked. Response B, while not as specific, acknowledges the overcooked nature of the pizza.
Step 3: In terms of fluency, both responses are easily understood and written in coherent language.
Step 4: Considering accuracy, Response A is inaccurate in relation to the context and instruction because it implies a positive meaning when it should be sarcastic. Response B doesn't fully capture the sarcasm but is closer to the truth about the pizza being badly cooked.
Step 5: Understanding relevance, Response A is not entirely relevant due to the inaccuracy in the creator's talent, while Response B, although less direct, is more relevant by mentioning the burnt or charred nature of the pizza.
Overall, Response B is better.
{{table:88fed018-6925-4bcc-b08c-60c3749d528b}}
{{figure:0b30f1a0-6bab-4f54-85ab-8b25fdb8ae19}}The prompt we use to extract answers from ill-formatted GPT-4 {{cite:d746af7}} responses, along with two example outputs. In one case, GPT-4's pairwise judgement is given, but in a format different than requested. In the other case, GPT-4 “defies" the prompt, and refuses to issue a judgement because both options are bad.
system prompt (human authored)

You are FinalAnswerExtractionGPT, an expert language model at extracting multiple choice answers from written out from longer explanations. You will be given several sentences describing a thought process which should eventually contain a final answer, either A or B. Your job is to extract the final answer that the explanation arrives at.

Some things to remember:

- Keep your answers short: only output "Final Answer: X" where X is A or B
- If the explanation doesn't mention a final choice explicitly, you can output "Unknown"

...
(+ 2 in-context examples (human authored))
...
blueExample One: GPT-4 gives a definitive answer, but in the wrong format

user (GPT-4's raw pairwise judgement)

Step 1: Compare the two responses to the instruction
- The instruction asks if people are required to stop and requests an explanation.

Step 2: Assess Response A
- Response A indicates that people should stop due to the presence of a stop sign. However, it doesn't consider the fact that the sign is pressed against the bus, and the LED lights are off.

Step 3: Assess Response B
- Response B simply says "yes," acknowledging that people should stop, but it doesn't provide the explanation required by the instruction.

Step 4: Compare specificity, fluency, and other factors
- Response A is more specific and fluent but misconstrues the image description.
- Response B is accurate in that it acknowledges the stop sign but lacks an explanation.
 Overall, since neither response is fully correct, it is difficult to declare a better response. However, given the fact that Response B acknowledges the prominent stop sign, I would say that Response B is slightly better, but it should also explain how the situation indicates that people should stop.
Please extract the final answer from the above text.

assistant (machine generated; used to derive the prediction)

Final Answer: Response B is slightly better, but it should also explain how the situation indicates that people should stop.
blueExample Two: GPT-4 refuses to answer.

user (GPT-4's raw pairwise judgement)

Step 1: Consider the image context, which states that the original thirteen colonies are shown with New Hampshire specifically highlighted in a darker green.

Step 2: Assess the accuracy of Response A, which claims that the colony shown is "Virginia." This is not accurate since the image context indicates New Hampshire being highlighted.

Step 3: Assess the accuracy of Response B, which claims that the colony shown is the Massachusetts Bay Colony. This response is also not accurate, as the image context states New Hampshire is the highlighted colony.

Step 4: In terms of accuracy, both responses are incorrect. Neither response is consistent with the image context.

Step 5: Consider other factors like fluency and specificity. Both responses seem fluent and specific in nature.

After considering these factors, none of the responses are accurate given the information provided in the image context. However, I am unable to provide an "Overall, Response X is better." judgement in this case as both responses are incorrect and neither stands out as being better than the other.

Please extract the final answer from the above text.

assistant (machine generated; used to derive the prediction)

Final Answer: Unknown.

{{table:22fcb1fb-ac60-4ec9-b592-067f08c8e48b}}
{{figure:d58cf090-de6f-409a-9fa0-6c955711e5e4}}

{{cite:febdaeb}}

{{cite:febdaeb}}


MVP: Robust Multi-View Practice for Driving Action Localization

Abstract: Distracted driving causes thousands of deaths per year, and how to apply deep-learning methods to prevent these tragedies has become a crucial problem. In Track3 of the 6th AI City Challenge, researchers provide a high-quality video dataset with densely action annotations. Due to the small data scale and unclear action boundary, the dataset presents a unique challenge to precisely localize all the different actions and classify their categories. In this paper, we make good use of the multi-view synchronization among videos, and conduct robust Multi-View Practice (MVP) for driving action localization. To avoid overfitting, we fine-tune SlowFast with Kinetics-700 pre-training as the feature extractor. Then the features of different views are passed to ActionFormer to generate candidate action proposals. For precisely localizing all the actions, we design elaborate post-processing, including model voting, threshold filtering and duplication removal. The results show that our MVP is robust for driving action localization, which achieves 28.49% F1-score in the Track3 test set.

Density-Guided Label Smoothing for Temporal Localization of Driving Actions

Abstract: Temporal localization of driving actions plays a crucial role in advanced driver-assistance systems and naturalistic driving studies. However, this is a challenging task due to strict requirements for robustness, reliability and accurate localization. In this work, we focus on improving the overall performance by efficiently utilizing video action recognition networks and adapting these to the problem of action localization. To this end, we first develop a density-guided label smoothing technique based on label probability distributions to facilitate better learning from boundary video-segments that typically include multiple labels. Second, we design a post-processing step to efficiently fuse information from video-segments and multiple camera views into scene-level predictions, which facilitates elimination of false positives. Our methodology yields a competitive performance on the A2 test set of the naturalistic driving action recognition track of the 2022 NVIDIA AI City Challenge with an F1 score of 0.271.

Summary of the Track 3 leader board.

Tables {{table:ff4543a2-7097-4994-9eb8-ca8761bb313d}} , {{table:682382fa-93e1-451f-948a-2980b33c8dbc}} , {{table:e0e1ba8d-ce40-432c-b36c-052d4135dca3}} , and  {{table:3ad4ab5e-8074-40ea-b333-845b55d6c6d5}}  summarize the leader boards for Track 1 (city-scale MTMC vehicle tracking), Track 2 (NL based vehicle retrieval), Track 3 (natural driving action recognition), and Track 4 (multi-class product counting and recognition), respectively.


Progressive Growing of GANs for Improved Quality, Stability, and Variation

Abstract: We describe a new training methodology for generative adversarial networks. The key idea is to grow both the generator and discriminator progressively: starting from a low resolution, we add new layers that model increasingly fine details as training progresses. This both speeds the training up and greatly stabilizes it, allowing us to produce images of unprecedented quality, e.g., CelebA images at 1024^2. We also propose a simple way to increase the variation in generated images, and achieve a record inception score of 8.80 in unsupervised CIFAR10. Additionally, we describe several implementation details that are important for discouraging unhealthy competition between the generator and discriminator. Finally, we suggest a new metric for evaluating GAN results, both in terms of image quality and variation. As an additional contribution, we construct a higher-quality version of the CelebA dataset.

A Style-Based Generator Architecture for Generative Adversarial Networks

Abstract: We propose an alternative generator architecture for generative adversarial networks, borrowing from style transfer literature. The new architecture leads to an automatically learned, unsupervised separation of high-level attributes (e.g., pose and identity when trained on human faces) and stochastic variation in the generated images (e.g., freckles, hair), and it enables intuitive, scale-specific control of the synthesis. The new generator improves the state-of-the-art in terms of traditional distribution quality metrics, leads to demonstrably better interpolation properties, and also better disentangles the latent factors of variation. To quantify interpolation quality and disentanglement, we propose two new, automated methods that are applicable to any generator architecture. Finally, we introduce a new, highly varied and high-quality dataset of human faces.

Projected GANs Converge Faster

Abstract: Generative Adversarial Networks (GANs) produce high-quality images but are challenging to train. They need careful regularization, vast amounts of compute, and expensive hyper-parameter sweeps. We make significant headway on these issues by projecting generated and real samples into a fixed, pretrained feature space. Motivated by the finding that the discriminator cannot fully exploit features from deeper layers of the pretrained model, we propose a more effective strategy that mixes features across channels and resolutions. Our Projected GAN improves image quality, sample efficiency, and convergence speed. It is further compatible with resolutions of up to one Megapixel and advances the state-of-the-art Fr\'echet Inception Distance (FID) on twenty-two benchmark datasets. Importantly, Projected GANs match the previously lowest FIDs up to 40 times faster, cutting the wall-clock time from 5 days to less than 3 hours given the same computational resources.

Diffusion-GAN: Training GANs with Diffusion

Abstract: Generative adversarial networks (GANs) are challenging to train stably, and a promising remedy of injecting instance noise into the discriminator input has not been very effective in practice. In this paper, we propose Diffusion-GAN, a novel GAN framework that leverages a forward diffusion chain to generate Gaussian-mixture distributed instance noise. Diffusion-GAN consists of three components, including an adaptive diffusion process, a diffusion timestep-dependent discriminator, and a generator. Both the observed and generated data are diffused by the same adaptive diffusion process. At each diffusion timestep, there is a different noise-to-data ratio and the timestep-dependent discriminator learns to distinguish the diffused real data from the diffused generated data. The generator learns from the discriminator's feedback by backpropagating through the forward diffusion chain, whose length is adaptively adjusted to balance the noise and data levels. We theoretically show that the discriminator's timestep-dependent strategy gives consistent and helpful guidance to the generator, enabling it to match the true data distribution. We demonstrate the advantages of Diffusion-GAN over strong GAN baselines on various datasets, showing that it can produce more realistic images with higher stability and data efficiency than state-of-the-art GANs.

Abstract: We present high quality image synthesis results using diffusion probabilistic models, a class of latent variable models inspired by considerations from nonequilibrium thermodynamics. Our best results are obtained by training on a weighted variational bound designed according to a novel connection between diffusion probabilistic models and denoising score matching with Langevin dynamics, and our models naturally admit a progressive lossy decompression scheme that can be interpreted as a generalization of autoregressive decoding. On the unconditional CIFAR10 dataset, we obtain an Inception score of 9.46 and a state-of-the-art FID score of 3.17. On 256x256 LSUN, we obtain sample quality similar to ProgressiveGAN. Our implementation is available at this https URL

Improved Denoising Diffusion Probabilistic Models

Abstract: Denoising diffusion probabilistic models (DDPM) are a class of generative models which have recently been shown to produce excellent samples. We show that with a few simple modifications, DDPMs can also achieve competitive log-likelihoods while maintaining high sample quality. Additionally, we find that learning variances of the reverse diffusion process allows sampling with an order of magnitude fewer forward passes with a negligible difference in sample quality, which is important for the practical deployment of these models. We additionally use precision and recall to compare how well DDPMs and GANs cover the target distribution. Finally, we show that the sample quality and likelihood of these models scale smoothly with model capacity and training compute, making them easily scalable. We release our code at https://github.com/openai/improved-diffusion

Diffusion Models Beat GANs on Image Synthesis

Abstract: We show that diffusion models can achieve image sample quality superior to the current state-of-the-art generative models. We achieve this on unconditional image synthesis by finding a better architecture through a series of ablations. For conditional image synthesis, we further improve sample quality with classifier guidance: a simple, compute-efficient method for trading off diversity for fidelity using gradients from a classifier. We achieve an FID of 2.97 on ImageNet 128$\times$128, 4.59 on ImageNet 256$\times$256, and 7.72 on ImageNet 512$\times$512, and we match BigGAN-deep even with as few as 25 forward passes per sample, all while maintaining better coverage of the distribution. Finally, we find that classifier guidance combines well with upsampling diffusion models, further improving FID to 3.94 on ImageNet 256$\times$256 and 3.85 on ImageNet 512$\times$512. We release our code at https://github.com/openai/guided-diffusion

High-Resolution Image Synthesis with Latent Diffusion Models

Abstract: By decomposing the image formation process into a sequential application of denoising autoencoders, diffusion models (DMs) achieve state-of-the-art synthesis results on image data and beyond. Additionally, their formulation allows for a guiding mechanism to control the image generation process without retraining. However, since these models typically operate directly in pixel space, optimization of powerful DMs often consumes hundreds of GPU days and inference is expensive due to sequential evaluations. To enable DM training on limited computational resources while retaining their quality and flexibility, we apply them in the latent space of powerful pretrained autoencoders. In contrast to previous work, training diffusion models on such a representation allows for the first time to reach a near-optimal point between complexity reduction and detail preservation, greatly boosting visual fidelity. By introducing cross-attention layers into the model architecture, we turn diffusion models into powerful and flexible generators for general conditioning inputs such as text or bounding boxes and high-resolution synthesis becomes possible in a convolutional manner. Our latent diffusion models (LDMs) achieve new state of the art scores for image inpainting and class-conditional image synthesis and highly competitive performance on various tasks, including unconditional image generation, text-to-image synthesis, and super-resolution, while significantly reducing computational requirements compared to pixel-based DMs.

Models evaluated in this work. FID {{cite:730cfc7}} on LSUN Bedroom 256{{formula:0d0f7c9f-b517-4b3d-82e5-95f708a08849}} 256 are taken from the original publications and from {{cite:3573306}} in the case of IDDPM. A lower FID corresponds to higher image quality.

An overview of the models under evaluation is shown in Table {{table:95e04552-a683-4dda-bb8d-664a5635d394}} , and example images are given in Appendix .
We consider data from ten models in total, five GAN and five DM.
This includes the seminal models ProGAN and StyleGAN, as well as the more recent ProjectedGAN.
Note that the recently proposed methods, Diff(usion)-StyleGAN2 and Diff(usion)-ProjectedGAN (the current state of the art on LSUN Bedroom) use a forward diffusion process to optimize GAN training, but this does not change the GAN model architecture.
From the class of DM, we consider the original DDPM, its successors IDDPM, and ADM, the latter outperforming several GAN with an FID of 1.90 on LSUN Bedroom.
PNDM optimizes the sampling process by the factor of 20 using pseudo numerical methods, which can be applied to existing pre-trained DM.
Lastly, LDM uses an adversarially trained autoencoder that transforms an image from the pixel space to a latent space.
Training the DM in this more suitable latent space significantly reduces the computational complexity and therefore enables training on higher resolutions.
The success of this approach is underpinned by the recent publication of Stable Diffusionhttps://github.com/CompVis/stable-diffusion, a powerful and publicly available text-to-image model, which is based on LDM.
An interesting detail is the fact that the autoencoder used to transform images is very similar to a VQGAN {{cite:e5a48c6}}.

Can we use State-of-the-Art Detectors for DM-Generated Images?

Using the dataset introduced in Section , we analyze the performance of the three state-of-the-art detectors.
We use pre-trained models provided by the authors without any modifications and compute the evaluation metrics using 10k real and 10k generated images.
The results are given in Table {{table:11fee2bd-0d15-4d3f-8eea-a19523e3e584}} .
For GAN-generated images, both classifiers provided by {{cite:768a20f}} achieve almost perfect detection results across all five generators, even w.r.t. the challenging Pd@1% metric.
For images generated by ProjectedGAN and Diff-ProjectedGAN, the variant trained on ProGAN images performs slightly better.
The reduced scores for these two models could in part be explained by the high image quality indicated by low FID (see Table {{table:95e04552-a683-4dda-bb8d-664a5635d394}} ).
In comparison, the performance of the classifier provided by {{cite:77c4c5b}} is reduced (except for ProGAN, on which the model was trained), which confirms the findings of {{cite:768a20f}}.
Similar to {{cite:768a20f}}, ProjectedGAN and Diff-ProjectedGAN achieve the lowest scores, with Pd@1% dropping below 50%.
Except for Diff-StyleGAN2, {{cite:da7afce}} achieves significantly worse results than its contestants.
A possible explanation could be that StyleGAN2 images make up a large portion of the training data for this classifier.


BioCreative V CDR task corpus: a resource for chemical disease relation extraction

Abstract: Community-run, formal evaluations and manually annotated text corpora are critically important for advancing biomedical text-mining research. Recently in BioCreative V, a new challenge was organized for the tasks of disease named entity recognition (DNER) and chemical-induced disease (CID) relation extraction. Given the nature of both tasks, a test collection is required to contain both disease/chemical annotations and relation annotations in the same set of articles. Despite previous efforts in biomedical corpus construction, none was found to be sufficient for the task. Thus, we developed our own corpus called BC5CDR during the challenge by inviting a team of Medical Subject Headings (MeSH) indexers for disease/chemical entity annotation and Comparative Toxicogenomics Database (CTD) curators for CID relation annotation. To ensure high annotation quality and productivity, detailed annotation guidelines and automatic annotation tools were provided. The resulting BC5CDR corpus consists of 1500 PubMed articles with 4409 annotated chemicals, 5818 diseases and 3116 chemical-disease interactions. Each entity annotation includes both the mention text spans and normalized concept identifiers, using MeSH as the controlled vocabulary. To ensure accuracy, the entities were first captured independently by two annotators followed by a consensus annotation: The average inter-annotator agreement (IAA) scores were 87.49% and 96.05% for the disease and chemicals, respectively, in the test set according to the Jaccard similarity coefficient. Our corpus was successfully used for the BioCreative V challenge tasks and should serve as a valuable resource for the text-mining research community. Database URL: http://www.biocreative.org/tasks/biocreative-v/track-3-cdr/

Introduction to the CoNLL-2003 Shared Task: Language-Independent Named Entity Recognition

Abstract: We describe the CoNLL-2003 shared task: language-independent named entity recognition. We give background information on the data sets (English and German) and the evaluation method, present a general overview of the systems that have taken part in the task and discuss their performance.

Description of the seven ET datasets.

Table {{table:ab0af8f2-eab2-4136-8835-5f44dd19f6ec}}  describes the seven datasets used in this work. These datasets are diverse, as they span several domains, none of them have an identical label set and some datasets capture fine-grained labels while others only have coarse labels. Also, the Wiki {{cite:3a55a90}} dataset is automatically generated using distant supervision process {{cite:11c6782}} and has multiple labels per entity mention in its label set. The other remaining datasets have a single label per entity mention.

Figure {{figure:729a7d08-d176-4338-8efa-623aae8d78c4}}  illustrates the complete experimental setup along with the learning models compared. In this setup, the objective is to measure the learning model's generalizability for the ET task as a whole, rather than on any specific dataset. To achieve this, we merged the test instances from the seven datasets listed in Table {{table:ab0af8f2-eab2-4136-8835-5f44dd19f6ec}}  to form a combined test corpus. On this test set, we compared the performance of the baseline models with the learning model trained via our proposed framework. We compare these models performance using the following evaluation schemes.


SLAM-Supported Self-Training for 6D Object Pose Estimation

Abstract: Recent progress in object pose prediction provides a promising path for robots to build object-level scene representations during navigation. However, as we deploy a robot in novel environments, the out-of-distribution data can degrade the prediction performance. To mitigate the domain gap, we can potentially perform self-training in the target domain, using predictions on robot-captured images as pseudo labels to fine-tune the object pose estimator. Unfortunately, the pose predictions are typically outlier-corrupted, and it is hard to quantify their uncertainties, which can result in low-quality pseudo-labeled data. To address the problem, we propose a SLAM-supported self-training method, leveraging robot understanding of the 3D scene geometry to enhance the object pose inference performance. Combining the pose predictions with robot odometry, we formulate and solve pose graph optimization to refine the object pose estimates and make pseudo labels more consistent across frames. We incorporate the pose prediction covariances as variables into the optimization to automatically model their uncertainties. This automatic covariance tuning (ACT) process can fit 6D pose prediction noise at the component level, leading to higher-quality pseudo training data. We test our method with the deep object pose estimator (DOPE) on the YCB video dataset and in real robot experiments. It achieves respectively 34.3% and 17.8% accuracy enhancements in pose prediction on the two tests. Our code is available at https://github.com/520xyxyzq/slam-super-6d.

Robust Category-Level 6D Pose Estimation with Coarse-to-Fine Rendering of Neural Features

Abstract: We consider the problem of category-level 6D pose estimation from a single RGB image. Our approach represents an object category as a cuboid mesh and learns a generative model of the neural feature activations at each mesh vertex to perform pose estimation through differentiable rendering. A common problem of rendering-based approaches is that they rely on bounding box proposals, which do not convey information about the 3D rotation of the object and are not reliable when objects are partially occluded. Instead, we introduce a coarse-to-fine optimization strategy that utilizes the rendering process to estimate a sparse set of 6D object proposals, which are subsequently refined with gradient-based optimization. The key to enabling the convergence of our approach is a neural feature representation that is trained to be scale- and rotation-invariant using contrastive learning. Our experiments demonstrate an enhanced category-level 6D pose estimation performance compared to prior work, particularly under strong partial occlusion.

GDR-Net: Geometry-Guided Direct Regression Network for Monocular 6D Object Pose Estimation

Abstract: 6D pose estimation from a single RGB image is a fundamental task in computer vision. The current top-performing deep learning-based methods rely on an indirect strategy, i.e., first establishing 2D-3D correspondences between the coordinates in the image plane and object coordinate system, and then applying a variant of the PnP/RANSAC algorithm. However, this two-stage pipeline is not end-to-end trainable, thus is hard to be employed for many tasks requiring differentiable poses. On the other hand, methods based on direct regression are currently inferior to geometry-based methods. In this work, we perform an in-depth investigation on both direct and indirect methods, and propose a simple yet effective Geometry-guided Direct Regression Network (GDR-Net) to learn the 6D pose in an end-to-end manner from dense correspondence-based intermediate geometric representations. Extensive experiments show that our approach remarkably outperforms state-of-the-art methods on LM, LM-O and YCB-V datasets. Code is available at https://git.io/GDR-Net.

Neural Correspondence Field for Object Pose Estimation

Abstract: We propose a method for estimating the 6DoF pose of a rigid object with an available 3D model from a single RGB image. Unlike classical correspondence-based methods which predict 3D object coordinates at pixels of the input image, the proposed method predicts 3D object coordinates at 3D query points sampled in the camera frustum. The move from pixels to 3D points, which is inspired by recent PIFu-style methods for 3D reconstruction, enables reasoning about the whole object, including its (self-)occluded parts. For a 3D query point associated with a pixel-aligned image feature, we train a fully-connected neural network to predict: (i) the corresponding 3D object coordinates, and (ii) the signed distance to the object surface, with the first defined only for query points in the surface vicinity. We call the mapping realized by this network as Neural Correspondence Field. The object pose is then robustly estimated from the predicted 3D-3D correspondences by the Kabsch-RANSAC algorithm. The proposed method achieves state-of-the-art results on three BOP datasets and is shown superior especially in challenging cases with occlusion. The project website is at: linhuang17.github.io/NCF.

Object Level Depth Reconstruction for Category Level 6D Object Pose Estimation From Monocular RGB Image

Abstract: Recently, RGBD-based category-level 6D object pose estimation has achieved promising improvement in performance, however, the requirement of depth information prohibits broader applications. In order to relieve this problem, this paper proposes a novel approach named Object Level Depth reconstruction Network (OLD-Net) taking only RGB images as input for category-level 6D object pose estimation. We propose to directly predict object-level depth from a monocular RGB image by deforming the category-level shape prior into object-level depth and the canonical NOCS representation. Two novel modules named Normalized Global Position Hints (NGPH) and Shape-aware Decoupled Depth Reconstruction (SDDR) module are introduced to learn high fidelity object-level depth and delicate shape representations. At last, the 6D object pose is solved by aligning the predicted canonical representation with the back-projected object-level depth. Extensive experiments on the challenging CAMERA25 and REAL275 datasets indicate that our model, though simple, achieves state-of-the-art performance.

RePOSE: Fast 6D Object Pose Refinement via Deep Texture Rendering

Abstract: We present RePOSE, a fast iterative refinement method for 6D object pose estimation. Prior methods perform refinement by feeding zoomed-in input and rendered RGB images into a CNN and directly regressing an update of a refined pose. Their runtime is slow due to the computational cost of CNN, which is especially prominent in multiple-object pose refinement. To overcome this problem, RePOSE leverages image rendering for fast feature extraction using a 3D model with a learnable texture. We call this deep texture rendering, which uses a shallow multilayer perceptron to directly regress a view-invariant image representation of an object. Furthermore, we utilize differentiable Levenberg-Marquardt (LM) optimization to refine a pose fast and accurately by minimizing the distance between the input and rendered image representations without the need of zooming in. These image representations are trained such that differentiable LM optimization converges within few iterations. Consequently, RePOSE runs at 92 FPS and achieves state-of-the-art accuracy of 51.6% on the Occlusion LineMOD dataset - a 4.1% absolute improvement over the prior art, and comparable result on the YCB-Video dataset with a much faster runtime. The code is available at https://github.com/sh8/repose.

Gen6D: Generalizable Model-Free 6-DoF Object Pose Estimation from RGB Images

Abstract: In this paper, we present a generalizable model-free 6-DoF object pose estimator called Gen6D. Existing generalizable pose estimators either need high-quality object models or require additional depth maps or object masks in test time, which significantly limits their application scope. In contrast, our pose estimator only requires some posed images of the unseen object and is able to accurately predict the poses of the object in arbitrary environments. Gen6D consists of an object detector, a viewpoint selector and a pose refiner, all of which do not require the 3D object model and can generalize to unseen objects. Experiments show that Gen6D achieves state-of-the-art results on two model-free datasets: the MOPED dataset and a new GenMOP dataset collected by us. In addition, on the LINEMOD dataset, Gen6D achieves competitive results compared with instance-specific pose estimators. Project page: https://liuyuan-pal.github.io/Gen6D/.

Knowledge Distillation for 6D Pose Estimation by Aligning Distributions of Local Predictions

Abstract: Knowledge distillation facilitates the training of a compact student network by using a deep teacher one. While this has achieved great success in many tasks, it remains completely unstudied for image-based 6D object pose estimation. In this work, we introduce the first knowledge distillation method driven by the 6D pose estimation task. To this end, we observe that most modern 6D pose estimation frameworks output local predictions, such as sparse 2D keypoints or dense representations, and that the compact student network typically struggles to predict such local quantities precisely. Therefore, instead of imposing prediction-to-prediction supervision from the teacher to the student, we propose to distill the teacher's distribution of local predictions into the student network, facilitating its training. Our experiments on several benchmarks show that our distillation method yields state-of-the-art results with different compact student models and for both keypoint-based and dense prediction-based architectures.

Ongoing monocular object pose estimation research Frequencies of the motivating problems for papers published in the years 2021 and 2022 and 2023 (up to paper submission).

Table {{table:71342d30-9e15-441a-b598-8e5ad1a1d05c}}  presents the motivating problems for the respective scientific works, sorted by frequency. The total number of publications considered for identifying these active research problems is 50, of which 22 are published in the field of robotics and 28 in computer vision. Single works count toward multiple problems.
The following section presents datasets that are commonly used for benchmarking these single-shot approaches for {{formula:8d67b444-b35f-4644-9c63-f66e5891bbe2}}  object pose estimation.

This section presents the important current research problems listed in Table {{table:71342d30-9e15-441a-b598-8e5ad1a1d05c}}  in detail.
Recent works that fundamentally contribute to solving the respective problems are presented and evaluated for open research question.
Note that the order of presentation does not follow the order of frequencies of the respective motivating topics, in order to provide a more natural reasoning through the grouping of semantically close open problems.

Occlusion handling is an important challenge for object pose estimation and present in many real-world robotics scenarios. Dense clutter, a hand or gripper manipulating an object, or even an ill-selected viewpoint may result in large parts of the object of interest being occluded.
Table {{table:71342d30-9e15-441a-b598-8e5ad1a1d05c}}  shows that occlusion handling is one of the major challenges based on the frequency at which it is a motivating current research.
Figure {{figure:5661c924-d64d-47b1-b9e2-2ddf2e01f545}}  presents the performance of general purpose methods {{cite:dea6797}}, {{cite:b444d96}}, {{cite:e040e0d}}, {{cite:4bb34ff}}, {{cite:60e009f}}, {{cite:7964cd5}}, {{cite:0fa403b}} compared to those designed for occlusion handling {{cite:c1a03dd}}, {{cite:9f4864a}}, {{cite:49a95a6}}, {{cite:96d05e5}}, {{cite:72e5d2f}}, {{cite:972e32d}}, {{cite:144b333}}, {{cite:339502a}}.
Evaluations on the LM {{cite:d6c0487}} and the LMO {{cite:7ed83bb}} dataset are presented.
LM features individual test sets for each of the 13 dataset objects with no occlusion.
LMO provides annotations for all eight LM objects that appear in the second test set, under strong occlusion.
The left plot of Figure {{figure:5661c924-d64d-47b1-b9e2-2ddf2e01f545}}  presents the performance of diverse methods on the LMO {{cite:7ed83bb}} dataset, plotted against the year of publication.
The performance of general purpose methods and those motivated by occlusion improved similarly over the years.
The right plot shows the correlation between the performance on LMO and LM {{cite:d6c0487}}, for general purpose methods and those motivated by occlusion handling.
Both types of methods exhibit similar performance ratios.