Fine-Grained Visual Classification of Aircraft

Abstract: This paper introduces FGVC-Aircraft, a new dataset containing 10,000 images of aircraft spanning 100 aircraft models, organised in a three-level hierarchy. At the finer level, differences between models are often subtle but always visually measurable, making visual recognition challenging but possible. A benchmark is obtained by defining corresponding classification tasks and evaluation protocols, and baseline results are presented. The construction of this dataset was made possible by the work of aircraft enthusiasts, a strategy that can extend to the study of number of other object classes. Compared to the domains usually considered in fine-grained visual classification (FGVC), for example animals, aircraft are rigid and hence less deformable. They, however, present other interesting modes of variation, including purpose, size, designation, structure, historical style, and branding.

Describing Textures in the Wild

Abstract: Patterns and textures are key characteristics of many natural objects: a shirt can be striped, the wings of a butterfly can be veined, and the skin of an animal can be scaly. Aiming at supporting this dimension in image understanding, we address the problem of describing textures with semantic attributes. We identify a vocabulary of forty-seven texture terms and use them to describe a large dataset of patterns collected "in the wild". The resulting Describable Textures Dataset (DTD) is a basis to seek the best representation for recognizing describable texture attributes in images. We port from object recognition to texture recognition the Improved Fisher Vector (IFV) and Deep Convolutional-network Activation Features (DeCAF), and show that surprisingly, they both outperform specialized texture descriptors not only on our problem, but also in established material recognition datasets. We also show that our describable attributes are excellent texture descriptors, transferring between datasets and tasks, in particular, combined with IFV and DeCAF, they significantly outperform the state-of-the-art by more than 10% on both FMD and KTH-TIPS-2b benchmarks. We also demonstrate that they produce intuitive descriptions of materials and Internet images.

EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classification

Abstract: In this paper, we present a patch-based land use and land cover classification approach using Sentinel-2 satellite images. The Sentinel-2 satellite images are openly and freely accessible, and are provided in the earth observation program Copernicus. We present a novel dataset, based on these images that covers 13 spectral bands and is comprised of ten classes with a total of 27 000 labeled and geo-referenced images. Benchmarks are provided for this novel dataset with its spectral bands using state-of-the-art deep convolutional neural networks. An overall classification accuracy of 98.57% was achieved with the proposed novel dataset. The resulting classification system opens a gate toward a number of earth observation applications. We demonstrate how this classification system can be used for detecting land use and land cover changes, and how it can assist in improving geographical maps. The geo-referenced dataset EuroSAT is made publicly available at https://github.com/phelber/eurosat.

Remote Sensing Image Scene Classification: Benchmark and State of the Art

Abstract: Remote sensing image scene classification plays an important role in a wide range of applications and hence has been receiving remarkable attention. During the past years, significant efforts have been made to develop various data sets or present a variety of approaches for scene classification from remote sensing images. However, a systematic review of the literature concerning data sets and methods for scene classification is still lacking. In addition, almost all existing data sets have a number of limitations, including the small scale of scene classes and the image numbers, the lack of image variations and diversity, and the saturation of accuracy. These limitations severely limit the development of new approaches especially deep learning-based methods. This paper first provides a comprehensive review of the recent progress. Then, we propose a large-scale data set, termed “NWPU-RESISC45,” which is a publicly available benchmark for REmote Sensing Image Scene Classification (RESISC), created by Northwestern Polytechnical University (NWPU). This data set contains 31 500 images, covering 45 scene classes with 700 images in each class. The proposed NWPU-RESISC45 1) is large-scale on the scene classes and the total image number; 2) holds big variations in translation, spatial resolution, viewpoint, object pose, illumination, background, and occlusion; and 3) has high within-class diversity and between-class similarity. The creation of this data set will enable the community to develop and evaluate various data-driven algorithms. Finally, several representative methods are evaluated using the proposed data set, and the results are reported as a useful baseline for future research.

Details of the downstream classification datasets.

We conducted evaluations on our pre-trained model using both ImageNet {{cite:adfaf81}} and 15 widely-used downstream datasets. To prepare the downstream datasets, we utilized torchvision and VISSL {{cite:6e470b5}}. The detailed information about the downstream datasets can be found in Table {{table:9d9456a3-1f16-4be9-b514-cf1a842a2a1a}} .


Faculty of Engineering, Alexandria University, Egypt

Multi-Dialect Arabic BERT for Country-Level Dialect Identification

Abstract: Arabic dialect identification is a complex problem for a number of inherent properties of the language itself. In this paper, we present the experiments conducted, and the models developed by our competing team, Mawdoo3 AI, along the way to achieving our winning solution to subtask 1 of the Nuanced Arabic Dialect Identification (NADI) shared task. The dialect identification subtask provides 21,000 country-level labeled tweets covering all 21 Arab countries. An unlabeled corpus of 10M tweets from the same domain is also presented by the competition organizers for optional use. Our winning solution itself came in the form of an ensemble of different training iterations of our pre-trained BERT model, which achieved a micro-averaged F1-score of 26.78% on the subtask at hand. We publicly release the pre-trained language model component of our winning solution under the name of Multi-dialect-Arabic-BERT model, for any interested researcher out there.

Arabic Dialect Identification Using BERT-Based Domain Adaptation

Abstract: Arabic is one of the most important and growing languages in the world. With the rise of social media platforms such as Twitter, Arabic spoken dialects have become more in use. In this paper, we describe our approach on the NADI Shared Task 1 that requires us to build a system to differentiate between different 21 Arabic dialects, we introduce a deep learning semi-supervised fashion approach along with pre-processing that was reported on NADI shared Task 1 Corpus. Our system ranks 4th in NADI's shared task competition achieving a 23.09% F1 macro average score with a simple yet efficient approach to differentiating between 21 Arabic Dialects given tweets.

List of the 14 teams that participated in Subtasks 1 and 2 and submitted description papers.

We received a total of 61 unique team registrations, among which 7 teams registered to participate in Subtask 1 only, 1 team registered to participate in Subtask 2 only, and 53 teams registered to participate in both subtasks. After evaluation phase, we received 47 submissions for Subtask 1 from 18 teams and 9 submissions for subtask 2 from 4 teams. Of participating teams, a total of 15 teams submitted description papers all of which except one were accepted for publication. Table {{table:5f9d75da-732d-4e78-badd-51c0f19a4842}}  lists the 14 teams whose papers were accepted.


REALM: Retrieval-Augmented Language Model Pre-Training

Abstract:
Language model pre-training has been shown to capture a surprising amount of world knowledge, crucial for NLP tasks such as question answering. However, this knowledge is stored implicitly in the parameters of a neural network, requiring ever-larger networks to cover more facts.
To capture knowledge in a more modular and interpretable way, we augment language model pre-training with a latent knowledge retriever, which allows the model to retrieve and attend over documents from a large corpus such as Wikipedia, used during pre-training, fine-tuning and inference. For the first time, we show how to pre-train such a knowledge retriever in an unsupervised manner, using masked language modeling as the learning signal and backpropagating through a retrieval step that considers millions of documents.
We demonstrate the effectiveness of Retrieval-Augmented Language Model pre-training (REALM) by fine-tuning on the challenging task of Open-domain Question Answering (Open-QA). We compare against state-of-the-art models for both explicit and implicit knowledge storage on three popular Open-QA benchmarks, and find that we outperform all previous methods by a significant margin (4-16% absolute accuracy), while also providing qualitative benefits such as interpretability and modularity.

Improving language models by retrieving from trillions of tokens

Abstract: We enhance auto-regressive language models by conditioning on document chunks retrieved from a large corpus, based on local similarity with preceding tokens. With a $2$ trillion token database, our Retrieval-Enhanced Transformer (RETRO) obtains comparable performance to GPT-3 and Jurassic-1 on the Pile, despite using 25$\times$ fewer parameters. After fine-tuning, RETRO performance translates to downstream knowledge-intensive tasks such as question answering. RETRO combines a frozen Bert retriever, a differentiable encoder and a chunked cross-attention mechanism to predict tokens based on an order of magnitude more data than what is typically consumed during training. We typically train RETRO from scratch, yet can also rapidly RETROfit pre-trained transformers with retrieval and still achieve good performance. Our work opens up new avenues for improving language models through explicit memory at unprecedented scale.

Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks

Abstract: Large pre-trained language models have been shown to store factual knowledge in their parameters, and achieve state-of-the-art results when fine-tuned on downstream NLP tasks. However, their ability to access and precisely manipulate knowledge is still limited, and hence on knowledge-intensive tasks, their performance lags behind task-specific architectures. Additionally, providing provenance for their decisions and updating their world knowledge remain open research problems. Pre-trained models with a differentiable access mechanism to explicit non-parametric memory can overcome this issue, but have so far been only investigated for extractive downstream tasks. We explore a general-purpose fine-tuning recipe for retrieval-augmented generation (RAG) -- models which combine pre-trained parametric and non-parametric memory for language generation. We introduce RAG models where the parametric memory is a pre-trained seq2seq model and the non-parametric memory is a dense vector index of Wikipedia, accessed with a pre-trained neural retriever. We compare two RAG formulations, one which conditions on the same retrieved passages across the whole generated sequence, the other can use different passages per token. We fine-tune and evaluate our models on a wide range of knowledge-intensive NLP tasks and set the state-of-the-art on three open domain QA tasks, outperforming parametric seq2seq models and task-specific retrieve-and-extract architectures. For language generation tasks, we find that RAG models generate more specific, diverse and factual language than a state-of-the-art parametric-only seq2seq baseline.

Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering

Abstract: Generative models for open domain question answering have proven to be competitive, without resorting to external knowledge. While promising, this approach requires to use models with billions of parameters, which are expensive to train and query. In this paper, we investigate how much these models can benefit from retrieving text passages, potentially containing evidence. We obtain state-of-the-art results on the Natural Questions and TriviaQA open benchmarks. Interestingly, we observe that the performance of this method significantly improves when increasing the number of retrieved passages. This is evidence that sequence-to-sequence models offers a flexible framework to efficiently aggregate and combine evidence from multiple passages.

The sizes of SPA models with those of comparable or outperformed LLMs.

In recent years, we have seen the promise of retrieval-augmented language models
partially addressing the aforementioned shortcomings {{cite:2212a32}}, {{cite:1dcc2d7}}, {{cite:c70af3d}}, {{cite:8510015}}, {{cite:b065ca9}}. The architecture of such models is semi-parametric, where the model integrates model parameters and knowledge from external data sources to make its predictions.
The first step of performing a task in these architectures
is to retrieve relevant knowledge from the external sources, and then perform finer-grained reasoning. Some of the benefits these architectures offer are that the external sources can be verified and updated easily, thereby reducing hallucinations {{cite:875a9df}} and making it easy to incorporate new knowledge and correct existing knowledge without needing to retrain the entire model {{cite:1dcc2d7}}.
Models that follow semi-parametric architectures (SPA) are typically smaller than LLMs and they have been shown to outperform LLMs on several NLP tasks such as open domain question answering (see Table {{table:d557677c-ae6b-4eb4-8da6-1974f6dbfbb7}} ).
Recent work that extends LLMs with modular reasoning and knowledge retrieval {{cite:e50e490}}, {{cite:f7758df}} is also a type of SPA.


The Kinetics Human Action Video Dataset

Abstract: We describe the DeepMind Kinetics human action video dataset. The dataset contains 400 human action classes, with at least 400 video clips for each action. Each clip lasts around 10s and is taken from a different YouTube video. The actions are human focussed and cover a broad range of classes including human-object interactions such as playing instruments, as well as human-human interactions such as shaking hands. We describe the statistics of the dataset, how it was collected, and give some baseline performance figures for neural network architectures trained and tested for human action classification on this dataset. We also carry out a preliminary analysis of whether imbalance in the dataset leads to bias in the classifiers.

Moments in Time Dataset: One Million Videos for Event Understanding

Abstract: We present the Moments in Time Dataset, a large-scale human-annotated collection of one million short videos corresponding to dynamic events unfolding within three seconds. Modeling the spatial-audio-temporal dynamics even for actions occurring in 3 second videos poses many challenges: meaningful events do not include only people, but also objects, animals, and natural phenomena; visual and auditory events can be symmetrical in time (“opening” is “closing” in reverse), and either transient or sustained. We describe the annotation process of our dataset (each video is tagged with one action or activity label among 339 different classes), analyze its scale and diversity in comparison to other large-scale video datasets for action recognition, and report results of several baseline models addressing separately, and jointly, three modalities: spatial, temporal and auditory. The Moments in Time dataset, designed to have a large coverage and diversity of events in both visual and auditory modalities, can serve as a new challenge to develop models that scale to the level of complexity and abstract reasoning that a human processes on a daily basis.

Aligning Books and Movies: Towards Story-Like Visual Explanations by Watching Movies and Reading Books

Abstract: Books are a rich source of both fine-grained information, how a character, an object or a scene looks like, as well as high-level semantics, what someone is thinking, feeling and how these states evolve through a story. This paper aims to align books to their movie releases in order to provide rich descriptive explanations for visual content that go semantically far beyond the captions available in the current datasets. To align movies and books we propose a neural sentence embedding that is trained in an unsupervised way from a large corpus of books, as well as a video-text neural embedding for computing similarities between movie clips and sentences in the book. We propose a context-aware CNN to combine information from multiple sources. We demonstrate good quantitative performance for movie/book alignment and show several qualitative examples that showcase the diversity of tasks our model can be used for.

Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts

Abstract: The availability of large-scale image captioning and visual question answering datasets has contributed significantly to recent successes in vision-and-language pretraining. However, these datasets are often collected with overrestrictive requirements inherited from their original target tasks (e.g., image caption generation), which limit the resulting dataset scale and diversity. We take a step further in pushing the limits of vision-and-language pretraining data by relaxing the data collection pipeline used in Conceptual Captions 3M (CC3M) [54] and introduce the Conceptual 12M (CC12M), a dataset with 12 million image-text pairs specifically meant to be used for visionand-language pre-training. We perform an analysis of this dataset and benchmark its effectiveness against CC3M on multiple downstream tasks with an emphasis on long-tail visual recognition. Our results clearly illustrate the benefit of scaling up pre-training data for vision-and-language tasks, as indicated by the new state-of-the-art results on both the nocaps and Conceptual Captions benchmarks.1

Microsoft COCO Captions: Data Collection and Evaluation Server

Abstract: In this paper we describe the Microsoft COCO Caption dataset and evaluation server. When completed, the dataset will contain over one and a half million captions describing over 330,000 images. For the training and validation images, five independent human generated captions will be provided. To ensure consistency in evaluation of automatic caption generation algorithms, an evaluation server is used. The evaluation server receives candidate captions and scores them using several popular metrics, including BLEU, METEOR, ROUGE and CIDEr. Instructions for using the evaluation server are provided.

PAQ: 65 Million Probably-Asked Questions and What You Can Do With Them

Abstract: Abstract Open-domain Question Answering models that directly leverage question-answer (QA) pairs, such as closed-book QA (CBQA) models and QA-pair retrievers, show promise in terms of speed and memory compared with conventional models which retrieve and read from text corpora. QA-pair retrievers also offer interpretable answers, a high degree of control, and are trivial to update at test time with new knowledge. However, these models fall short of the accuracy of retrieve-and-read systems, as substantially less knowledge is covered by the available QA-pairs relative to text corpora like Wikipedia. To facilitate improved QA-pair models, we introduce Probably Asked Questions (PAQ), a very large resource of 65M automatically generated QA-pairs. We introduce a new QA-pair retriever, RePAQ, to complement PAQ. We find that PAQ preempts and caches test questions, enabling RePAQ to match the accuracy of recent retrieve-and-read models, whilst being significantly faster. Using PAQ, we train CBQA models which outperform comparable baselines by 5%, but trail RePAQ by over 15%, indicating the effectiveness of explicit retrieval. RePAQ can be configured for size (under 500MB) or speed (over 1K questions per second) while retaining high accuracy. Lastly, we demonstrate RePAQ’s strength at selective QA, abstaining from answering when it is likely to be incorrect. This enables RePAQ to “back-off” to a more expensive state-of-the-art model, leading to a combined system which is both more accurate and 2x faster than the state-of-the-art model alone.

The work is supported by the National Key R&D Program of China (2020AAA0105200), Beijing Academy of Artificial Intelligence.

Hassan Akbari, Li Yuan, Rui Qian, Wei-Hong Chuang, Shih-Fu Chang, Yin Cui, and
Boqing Gong.
Vatt: Transformers for multimodal self-supervised learning from raw
video, audio and text.
arXiv preprint arXiv:2104.11178, 2021.
Chris Alberti, Jeffrey Ling, Michael Collins, and David Reitter.
Fusion of detected objects in text for visual question answering.
arXiv preprint arXiv:1908.05054, 2019.
Anonymous.
Are BERT families zero-shot learners? a study on their potential
and limitations.
In Submitted to ICLR, 2022.
under review.
Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučić,
and Cordelia Schmid.
Vivit: A video vision transformer.
arXiv preprint arXiv:2103.15691, 2021.
Gedas Bertasius, Heng Wang, and Lorenzo Torresani.
Is space-time attention all you need for video understanding?
In ICML, July 2021.
Gedas Bertasius, Heng Wang, and Lorenzo Torresani.
Is space-time attention all you need for video understanding?
arXiv preprint arXiv:2102.05095, 2021.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan,
Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda
Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, T. J.
Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeff Wu, Clemens
Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott
Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec
Radford, Ilya Sutskever, and Dario Amodei.
Language models are few-shot learners.
arXiv preprint arXiv:2005.14165, 2020.
Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut.
Conceptual 12M: Pushing web-scale image-text pre-training to
recognize long-tail visual concepts.
In CVPR, 2021.
Chun-Fu Chen, Quanfu Fan, and Rameswar Panda.
Crossvit: Cross-attention multi-scale vision transformer for image
classification.
arXiv preprint arXiv:2103.14899, 2021.
David Chen and William B Dolan.
Collecting highly parallel data for paraphrase evaluation.
In ACL, pages 190–200, 2011.
Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr
Dollár, and C. Lawrence Zitnick.
Microsoft coco captions: Data collection and evaluation server.
arXiv preprint arXiv:1504.00325, 2015.
Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr
Dollár, and C Lawrence Zitnick.
Microsoft coco captions: Data collection and evaluation server.
arXiv preprint arXiv:1504.00325, 2015.
Xinyang Chen, Sinan Wang, Bo Fu, Mingsheng Long, and Jianmin Wang.
Catastrophic forgetting meets negative transfer: Batch spectral
shrinkage for safe transfer learning.
In NeurIPS, 2019.
Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu
Cheng, and Jingjing Liu.
Uniter: Universal image-text representation learning.
In ECCV, pages 104–120. Springer, 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.
Imagenet: A large-scale hierarchical image database.
In CVPR, pages 248–255. Ieee, 2009.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.
Bert: Pre-training of deep bidirectional transformers for language
understanding.
arXiv preprint arXiv:1810.04805, 2018.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn,
Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg
Heigold, Sylvain Gelly, et al.
An image is worth 16x16 words: Transformers for image recognition at
scale.
arXiv preprint arXiv:2010.11929, 2020.
Han Fang, Pengfei Xiong, Luhui Xu, and Yu Chen.
Clip2video: Mastering video-text retrieval via image clip.
arXiv preprint arXiv:2106.11097, 2021.
Tianyu Gao, Adam Fisch, and Danqi Chen.
Making pre-trained language models better few-shot learners.
arXiv preprint arXiv:2012.15723, 2020.
Tianyu Gao, Adam Fisch, and Danqi Chen.
Making pre-trained language models better few-shot learners.
In ACL, 2021.
Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh.
Making the V in VQA matter: Elevating the role of image
understanding in Visual Question Answering.
In CVPR, 2017.
Ronghang Hu and Amanpreet Singh.
Unit: Multimodal multitask learning with a unified transformer.
arXiv preprint arXiv:2102.10772, 2021.
Haoyang Huang, Yaobo Liang, Nan Duan, Ming Gong, Linjun Shou, Daxin Jiang, and
Ming Zhou.
Unicoder: A universal language encoder by pre-training with multiple
cross-lingual tasks.
arXiv preprint arXiv:1909.00964, 2019.
Zhicheng Huang, Zhaoyang Zeng, Bei Liu, Dongmei Fu, and Jianlong Fu.
Pixel-bert: Aligning image pixels with text by deep multi-modal
transformers.
arXiv preprint arXiv:2004.00849, 2020.
Zhengbao Jiang, Frank F. Xu, J. Araki, and Graham Neubig.
How can we know what language models know?
TACL, 8:423–438, 2020.
Sebastian Kalkowski, Christian Schulze, Andreas Dengel, and Damian Borth.
Real-time analysis and visualization of the yfcc100m dataset.
In Proceedings of the 2015 workshop on community-organized
multimodal mining: opportunities for novel solutions, pages 25–30, 2015.
Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra
Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al.
The kinetics human action video dataset.
arXiv preprint arXiv:1705.06950, 2017.
Wonjae Kim, Bokyung Son, and Ildoo Kim.
Vilt: Vision-and-language transformer without convolution or region
supervision.
arXiv preprint arXiv:2102.03334, 2021.
Diederik P Kingma and Jimmy Ba.
Adam: A method for stochastic optimization.
arXiv preprint arXiv:1412.6980, 2014.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua
Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al.
Visual genome: Connecting language and vision using crowdsourced
dense image annotations.
IJCV, 123(1):32–73, 2017.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and
Radu Soricut.
Albert: A lite bert for self-supervised learning of language
representations.
arXiv preprint arXiv:1909.11942, 2019.
Gustav Larsson, Michael Maire, and Gregory Shakhnarovich.
Fractalnet: Ultra-deep neural networks without residuals.
arXiv preprint arXiv:1605.07648, 2017.
Brian Lester, Rami Al-Rfou, and Noah Constant.
The power of scale for parameter-efficient prompt tuning.
In EMNLP, 2021.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed,
Omer Levy, Ves Stoyanov, and Luke Zettlemoyer.
Bart: Denoising sequence-to-sequence pre-training for natural
language generation, translation, and comprehension.
arXiv preprint arXiv:1910.13461, 2019.
Patrick Lewis, Yuxiang Wu, Linqing Liu, Pasquale Minervini, Heinrich Küttler,
Aleksandra Piktus, Pontus Stenetorp, and Sebastian Riedel.
Paq: 65 million probably-asked questions and what you can do with
them.
arXiv preprint arXiv:2102.07033, 2021.
Gen Li, Nan Duan, Yuejian Fang, Daxin Jiang, and Ming Zhou.
Unicoder-vl: A universal encoder for vision and language by
cross-modal pre-training.
In AAAI, 2020.
Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang.
Visualbert: A simple and performant baseline for vision and language.
arXiv preprint arXiv:1908.03557, 2019.
Xiang Lisa Li and Percy Liang.
Prefix-tuning: Optimizing continuous prompts for generation, 2021.
Zewen Li, Fan Liu, Wenjie Yang, Shouheng Peng, and Jun Zhou.
A survey of convolutional neural networks: Analysis, applications,
and prospects.
IEEE Transactions on Neural Networks and Learning Systems,
pages 1–21, 2021.
Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva
Ramanan, Piotr Dollár, and C Lawrence Zitnick.
Microsoft coco: Common objects in context.
In ECCV, 2014.
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and
Graham Neubig.
Pre-train, prompt, and predict: A systematic survey of prompting
methods in natural language processing.
arXiv preprint arXiv:2107.13586, 2021.
Xiao Liu, Kaixuan Ji, Yicheng Fu, Zhengxiao Du, Zhilin Yang, and Jie Tang.
P-tuning v2: Prompt tuning can be comparable to fine-tuning
universally across scales and tasks.
arXiv preprint arXiv:2110.07602, 2021.
Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and
Jie Tang.
Gpt understands, too.
arXiv preprint arXiv:2103.10385, 2021.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer
Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.
Roberta: A robustly optimized bert pretraining approach.
arXiv preprint arXiv:1907.11692, 2019.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and
Baining Guo.
Swin transformer: Hierarchical vision transformer using shifted
windows.
ICCV, 2021.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee.
Vilbert: Pretraining task-agnostic visiolinguistic representations
for vision-and-language tasks.
In NeurIPS, 2019.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee.
Vilbert: Pretraining task-agnostic visiolinguistic representations
for vision-and-language tasks.
arXiv preprint arXiv:1908.02265, 2019.
Huaishao Luo, Lei Ji, Ming Zhong, Yang Chen, Wen Lei, Nan Duan, and Tianrui Li.
Clip4clip: An empirical study of clip for end to end video clip
retrieval.
arXiv preprint arXiv:2104.08860, 2021.
Mathew Monfort, Alex Andonian, Bolei Zhou, Kandan Ramakrishnan, Sarah Adel
Bargal, Tom Yan, Lisa Brown, Quanfu Fan, Dan Gutfreund, Carl Vondrick, et al.
Moments in time dataset: one million videos for event understanding.
TPAMI, 42(2):502–508, 2019.
Vicente Ordonez, Girish Kulkarni, and Tamara Berg.
Im2text: Describing images using 1 million captioned photographs.
NeurIPS, 24:1143–1151, 2011.
Gao Peng, Geng Shijie, Zhang Renrui, Ma Teli, Fang Rongyao, Zhang Yongfeng, Li
Hongsheng, and Qiao Yu.
Clip-adapter: Better vision-language models with feature adapters.
arXiv preprint arXiv:2110.04544, 2021.
Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia
Hockenmaier, and Svetlana Lazebnik.
Flickr30k entities: Collecting region-to-phrase correspondences for
richer image-to-sentence models.
In ICCV, pages 2641–2649, 2015.
Di Qi, Lin Su, Jianwei Song, Edward Cui, Taroon Bharti, and Arun Sacheti.
Imagebert: Cross-modal pre-training with large-scale weak-supervised
image-text data.
arXiv preprint arXiv:2001.07966, 2020.
Xipeng Qiu, Tianxiang Sun, Yige Xu, Yunfan Shao, Ning Dai, and Xuanjing Huang.
Pre-trained models for natural language processing: A survey.
Science China Technological Sciences, pages 1–26, 2020.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh,
Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark,
et al.
Learning transferable visual models from natural language
supervision.
arXiv preprint arXiv:2103.00020, 2021.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh,
Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark,
Gretchen Krueger, and Ilya Sutskever.
Learning transferable visual models from natural language
supervision.
In ICML, 2021.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.
Faster r-cnn: Towards real-time object detection with region proposal
networks.
NeurIPS, 28:91–99, 2015.
Zhang Renrui, Fang Rongyao, Gao Peng, Zhang Wei, Li Kunchang, Dai Jifeng, Qiao
Yu, and Li Hongsheng.
Tip-adapter: Training-free clip-adapter for better vision-language
modeling.
arXiv preprint arXiv:2111.03930, 2021.
Rico Sennrich, Barry Haddow, and Alexandra Birch.
Neural machine translation of rare words with subword units.
In ACL, 2016.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut.
Conceptual captions: A cleaned, hypernymed, image alt-text dataset
for automatic image captioning.
In ACL, pages 2556–2565, 2018.
Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer
Singh.
AutoPrompt: Eliciting knowledge from language models with
automatically generated prompts.
In EMNLP, 2020.
Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai.
Vl-bert: Pre-training of generic visual-linguistic representations.
arXiv preprint arXiv:1908.08530, 2019.
Chen Sun, Fabien Baradel, Kevin Murphy, and Cordelia Schmid.
Learning video representations using contrastive bidirectional
transformer.
arXiv preprint arXiv:1906.05743, 2019.
Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid.
Videobert: A joint model for video and language representation
learning.
In CVPR, pages 7464–7473, 2019.
Hao Tan and Mohit Bansal.
Lxmert: Learning cross-modality encoder representations from
transformers.
arXiv preprint arXiv:1908.07490, 2019.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre
Sablayrolles, and Herve Jegou.
Training data-efficient image transformers & distillation through
attention.
In ICML, volume 139, pages 10347–10357, July 2021.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre
Sablayrolles, and Hervé Jégou.
Training data-efficient image transformers & distillation through
attention.
In ICML, pages 10347–10357. PMLR, 2021.
Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and
Hervé Jégou.
Going deeper with image transformers.
arXiv preprint arXiv:2103.17239, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones,
Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.
Attention is all you need.
In NeurIPS, pages 5998–6008, 2017.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R
Bowman.
Glue: A multi-task benchmark and analysis platform for natural
language understanding.
arXiv preprint arXiv:1804.07461, 2018.
Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong
Lu, Ping Luo, and Ling Shao.
Pyramid vision transformer: A versatile backbone for dense prediction
without convolutions.
In ICCV, 2021.
Han Xu, Zhang Zhengyan, Ding Ning, Gu Yuxian, Liu Xiao, Huo Yuqi, Qiu Jiezhong,
Zhang Liang, Han Wentao, Huang Minlie, et al.
Pre-trained models: Past, present and future.
arXiv preprint arXiv:2106.07139, 2021.
Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Zi-Hang Jiang,
Francis E.H. Tay, Jiashi Feng, and Shuicheng Yan.
Tokens-to-token vit: Training vision transformers from scratch on
imagenet.
In ICCV, pages 558–567, October 2021.
Yanyi Zhang, Xinyu Li, Chunhui Liu, Bing Shuai, Yi Zhu, Biagio Brattoli, Hao
Chen, Ivan Marsic, and Joseph Tighe.
Vidtr: Video transformer without convolutions.
In ICCV, pages 13577–13587, October 2021.
Ziqi Zhang, Yaya Shi, Chunfeng Yuan, Bing Li, Peijin Wang, Weiming Hu, and
Zheng-Jun Zha.
Object relational graph with teacher-recommended learning for video
captioning.
In CVPR, pages 13278–13288, 2020.
Daquan Zhou, Bingyi Kang, Xiaojie Jin, Linjie Yang, Xiaochen Lian, Qibin Hou,
and Jiashi Feng.
Deepvit: Towards deeper vision transformer.
arXiv preprint arXiv:2103.11886, 2021.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu.
Learning to prompt for vision-language models.
arXiv preprint arXiv:2109.01134, 2021.
Luowei Zhou, Hamid Palangi, Lei Zhang, Houdong Hu, Jason Corso, and Jianfeng
Gao.
Unified vision-language pre-training for image captioning and vqa.
In AAAI, 2020.
Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun,
Antonio Torralba, and Sanja Fidler.
Aligning books and movies: Towards story-like visual explanations by
watching movies and reading books.
In ICCV, pages 19–27, 2015.

Tokenizer
Given the raw inputs from text, image, and video modalities, modality-specific tokenizers are applied to generate the input token sequences for the Transformer encoder.
Here, we use the BPE tokenizer {{cite:cb60cc3}} for text modality, the image patch tokenizer {{cite:4686ad3}} for image modality, and the temporal frame patch tokenizer {{cite:43f84a9}} for video modality. These outputted tokens are attached with additional modality type embeddings to identify which modality the raw input belongs to. The tokenizers are illustrated in Fig. {{figure:68532472-c5c8-40a1-b316-6abf650405be}} .
Text Tokenizer.  The BPE Tokenizer {{cite:cb60cc3}} is employed for text modality. The text inputs are split into sub-words and projected by a linear embedding layer. A learnable 1D positional embedding for text with a max length of 256 is added to the word embeddings. To specify the input modality, an additional trainable textual modality embedding <T> is added to each token embedding. Note that all the text inputs of our model share the same vocabulary.
Image Tokenizer.  The image patch tokenizer {{cite:4686ad3}} is utilized as the image tokenizer. The input images are resized to 224{{formula:54ed1466-1655-4540-8603-edbc6123017c}} 224, and flattened to a sequence of image patches with shapes 16{{formula:2a386aa6-3847-4f45-9e5a-50cfd70f4b78}} 16, which are further mapped by a linear projection. A sequence of learnable image positional embeddings with a fixed length {{formula:1a5cce4b-d382-4144-92db-4b2f12292a39}} , and an additional visual modality embedding <V> are also added.
Video Tokenizer.  The temporal frame patch tokenizer {{cite:43f84a9}} is used for video tokenization. Each frame of the input video is flattened to image patches with shape 16{{formula:1309f6e6-20e0-479b-822b-7a27a1fe808f}} 16. For a video with {{formula:6b4cff14-24f8-4c76-a356-4412c227d75c}}  frames ({{formula:7801023c-8e89-4ccd-bb00-340f54fd13af}}  by default), the number of tokens would be {{formula:3088a92f-5fb5-4dbd-a6b8-8f5296043028}} . The spatial positional embeddings for images as well as a 1D temporal position embedding with max length {{formula:8672346a-57cb-4744-8f12-b8b422fb624b}}  are added to the video embeddings. Besides, the additional visual modality embedding <V> is added to each token embedding.

Implementation Details for Auto-regressive Language Modeling
The attention mask of autoregressive language modeling.
{{figure:bbd90e25-cdeb-4566-b3e3-5a292e09ab83}}In the formulation of autoregressive tasks from previous works, each input token attends to all previous tokens, including itself, to predict the next word.
Unlike previous works, we use <SPE> to predict the current word.
Fig. {{figure:bbd90e25-cdeb-4566-b3e3-5a292e09ab83}}  shows how we achieve this. In the inference stage, <SPE> is appended to the end of the predicted tokens as the input. The corresponding output is used for prediction.
In the training stage, we append several <SPE> tokens after the input sequence. Each <SPE> token is trained to predict a word in the input sequence.
The attention mask is designed to make sure that the word tokens do not attend to <SPE> tokens, and each <SPE> token only attends to itself and the previous word tokens. In this way, the training stage and the inference stage are aligned.

Prompt Tuning
Four groups of parameters are learnable in prompt-tuning. They are <SPE>, layer normalization parameters, prompt tokens, and linear heads. This section introduces the usage and the number of parameters of each parameter group.
Details Similar to the pre-training stage, <SPE> token is shared for both inputs and targets.
Layer normalization weights and biases in each Transformer layer and tokenizer are tuned.
Learnable prompts are added to each layer of the Transformer encoder.
Specifically, the input prompts of each layer do not come from the output of the previous layer.
They are random initialized learnable parameters.
For all prompt-tuning experiments, we use 10 learnable prompts for each layer on both inputs.
The linear heads only apply for classification tasks.
It takes the feature that is to be classified as input, and returns a classification logit.
The output probability is a linear combination of the probability from the similarity score and the linear head:
{{formula:df5032db-4cea-417c-8376-4c1f402f4ae0}}
where {{formula:6d55ea80-3599-400f-b48a-7eaae4eb0c17}}  and {{formula:966df45a-7210-48a7-98ea-df9e0389789b}}  are the weights and bias of the linear, and {{formula:f4061f95-ecda-49ab-bdd0-d4bd5d839fd8}}  is a learnable scalar.
{{formula:1c1ce2fd-382c-437e-aa14-07a263318ebc}}  and {{formula:2071c51d-2d18-4d1d-b6b3-1ecf5da6607e}}  are initialized with 0 and {{formula:cf3fa00b-4f29-4af1-9ede-51c8cd6b8c5b}}  is initialized with 1.
Number of Parameters
The number of learnable parameters in the prompt-tuning stage. Note that the linear head only applies for classification tasks.

Number of Parameters
<SPE>
768
Layer Norm
41,472
Prompt
184,320
Linear head
768 * num_classes
{{table:1ee13f88-ca20-4e3a-96d6-be2c8b499aca}}{{table:a701aca3-dfc9-4663-a3f0-94f2ce43b611}}Tab. {{table:a701aca3-dfc9-4663-a3f0-94f2ce43b611}}  shows the number of parameters of each learnable component in prompt-tuning.
For tasks other than classification, the number of parameters is 227K in total.
When the linear head is added, , in ImageNet-1k classification task, the number is 995K, which is still less than 1% of all the parameters in the pre-training stage.

Formulation of Novel Tasks
Input and target formats of our novel tasks. For each task, the left column represents the format of input sequence {{formula:dbfedc60-b05c-4014-96c2-15747ce895dd}} , and the right column represents the format of the target sequence {{formula:dfd265d5-37ec-4736-9720-2ab8c0663c25}} . {{formula:45662b62-b708-42e1-99ff-d8e645230bd2}}  and {{formula:85a10dcb-64bc-45a9-b386-376013a240a5}}  are used to calculate the joint probability distribution. Here, we have omitted the tokenizer and encoder for concision.
{{figure:b550f2e0-c7f6-4760-a11d-15d3acfad034}}The generic perceptual modeling makes it easy to convert existing tasks into the unified task formulation of our Uni-Perceiver.
Fig. {{figure:b550f2e0-c7f6-4760-a11d-15d3acfad034}}  illustrates the input and the output formulations of our novel tasks, which we will describe in details.
Video Caption.
Similar to image caption, video caption is modeled as the autoregressive language modeling task with video clues.
In this task, the model predicts each word based on the video and its previous words.
The input set {{formula:c36f762b-45a0-45dc-9bb4-8328ad31714e}}  consists of the sequence concatenation of video and the words that have been predicted, followed with a <SPE> token.
The output features of the <SPE> token is used to calculate the joint probability with each words in the the vocabulary set {{formula:cdcfb2cc-7103-4f49-9a3b-25795fbc8778}} .
Then the word with the highest probability is the predicted word at the current location.
Additionally, video caption follows the efficient implementation of autoregressive training introduced in Sec. .
Video-Text Retrieval.
The Video-Text retrieval follows the formulation of Image-Text retrieval task, except that the image sequence is replaced by the video sequence.
Specifically, the input sets {{formula:c70f331e-7b58-40d3-a511-465c2626f3d4}}  and {{formula:83f9abfa-0489-4741-9d02-09d1530abd9d}}  are composed of video and text sequences respectively.
Each sequence in {{formula:3166974f-dc87-4738-b4fc-d6b9926999f2}}  and {{formula:d462d739-79f3-4b70-9207-1c9ffcbec378}}  also has a <SPE> token at the beginning.
We use the output feature at the <SPE> token as the final representation of the input video or the text to calculate the joint probability distribution.
Visual Question Answering.
We formulate the VQA task as a special case of masked language modeling with image clues.
The input {{formula:0f576191-cc9d-4f51-b2f4-e0a62f32f46e}}  is the combination of image and question sequences. It should be noted that each question ends with a “?” mark and a <SPE> token is appended to the end of the question sequence.
The {{formula:eaeb4ab9-346e-420c-9e97-e727062fd827}}  set consists of candidate answer sequences, in which each begins with a <SPE> token too.
The joint probability distribution between {{formula:4664d4cf-417d-4e18-86e9-a5079cbdbff0}}  and {{formula:4e854d2b-71bb-4e47-b544-3c36db06e5cc}}  can be calculated by using the output feature from <SPE> tokens.
Natural Language Understanding.
The formulation of natural language understanding task is similar to that of image classification task.
{{formula:b7a15bd7-cbbf-4c19-bb8b-bada07158f6b}}  denotes the set of the input single sentence or the sentence-pair, and {{formula:86b23fd5-26c3-4d10-8d0c-cabce49152c2}}  is the set contains the textual class label.
For example, in SST-2 SST2, {{formula:cb5960e8-f4e0-4ab5-8905-6c400f6e3cdb}}  denotes the sentence sequence of movie review and {{formula:a113cb73-a43a-484e-b079-8309cdb68187}}  is the sentiment label.
In MRPC mrpc, {{formula:c6773771-85fd-4bb2-a747-ce177c9a0ea8}}  instead is the sequence combination of sentence pairs extracted from news, and {{formula:e65baf05-df04-4f2f-b22f-d642f50cbbd7}}  is the label to indicate whether the sentence pair are semantically equivalent.
We also add <SPE> tokens at the beginning of the sequences {{formula:8ebc21d8-5925-4c76-9280-66553559a021}}  and {{formula:6daeafbc-e7d1-424b-9f0a-2d4e3ea8bd35}} , of which output features are used to computed joint probability.

Extra pre-training details
Ingredients and hyper-parameters for our pre-training.

Task
Dataset
Batch Size
Sampling Weight
1* Image Classification
ImageNet-21k {{cite:c175469}}
64
0.333
2* Video Classification
Kinetics-700 {{cite:e7e2f79}}
4
0.0925

Moments in Time {{cite:39e113a}}
24
0.0185
8* Auto-encoding LM
Books&Wiki {{cite:d18c245}}
64
0.07775

YFCC {{cite:f30e7c2}}
64
0.02778

CC12M {{cite:940e960}}
64
0.02778

CC3M {{cite:477fe9e}}
64
0.01389

Visual Genome {{cite:3ff07fb}}
64
0.01389

COCO Caption {{cite:8f5f427}}
64
0.01389

SBU {{cite:30a3919}}
64
0.01389

PAQ {{cite:eec85c7}}
512
0.0222
8* Auto-regressive LM
Books&Wiki {{cite:d18c245}}
64
0.07775

YFCC {{cite:f30e7c2}}
56
0.02778

CC12M {{cite:940e960}}
56
0.02778

CC3M {{cite:477fe9e}}
56
0.01389

Visual Genome {{cite:3ff07fb}}
56
0.01389

COCO Caption {{cite:8f5f427}}
56
0.01389

SBU {{cite:30a3919}}
56
0.01389

PAQ {{cite:eec85c7}}
400
0.0222
7* Retrieval
YFCC {{cite:f30e7c2}}
128
0.02778

CC12M {{cite:940e960}}
128
0.02778

CC3M {{cite:477fe9e}}
128
0.01389

Visual Genome {{cite:3ff07fb}}
128
0.01389

COCO Caption {{cite:8f5f427}}
128
0.01389

SBU {{cite:30a3919}}
128
0.01389

PAQ {{cite:eec85c7}}
512
0.0222

{{table:a0323251-8890-414a-8e0a-25262b781fba}}{{table:608db33c-5a35-4c02-a6ae-a469d57c1f7b}}Sampling Weight & Batch Size & Loss   Tab. {{table:608db33c-5a35-4c02-a6ae-a469d57c1f7b}}  lists the batch size and sampling weight of each task and dataset in the pre-training stage.
We use cross-entropy loss for language modeling tasks. The other tasks are trained with cross-entropy loss with 0.1 label smoothing.
The loss weight of video classification is 0.05, which helps stabilize training in our experiments. The other loss weights are 1.0 by default.
For retrieval tasks like image-text retrieval, we use training samples in the same batch as negative samples, whose typical size is 127 except the PAQ dataset.
Note that we do not use memory bank and do not gather feature across GPU devices to provide more negative samples, which may further promote the performance of retrieval tasks.
Data Augmentation
We apply augmentation techniques to image and video modalities to avoid overfitting.
For images in ImageNet-21k dataset, we apply augmentation same as {{cite:8fe387c}}.
Rand-Augcubuk2020randaugment, random erasing zhong2020random, mixup zhang2017mixup and cutmix yun2019cutmix are used simultaneously.
For images in other datasets, we resize the images to the short edge size of 256, and then a {{formula:6bb66995-9079-4242-ad01-c553506a7388}}  region is cropped randomly from the resized images during training.
During inference, the random crop is replaced with center crop operation.
For all Video inputs, we apply the same augmentations used in {{cite:43f84a9}}. We use clips of size {{formula:3a494947-8525-44ad-8b53-61a7b0db659d}}  for Kinetics-700 and Kinetics-400, and {{formula:a05a9d19-3d34-4485-af8d-1a08e65746c0}}  for Moments in Time. The temporal sample rate is 32. We use a single temporal clip. During training, the start frame is randomly picked if the video is longer than the clip.
In the training stage, we first resize the shorter side of the video to a random value in {{formula:7643e67f-12dc-463c-beb3-77e56c0019e5}} , then we randomly sample a {{formula:03251993-82a7-4a47-a384-f42aba1b730c}}  crop. In the test stage, the short side of the video is resized to 224. For classification tasks, We use 3 spatial crops with size {{formula:5c032f2f-d870-42a8-9b00-57b71768e77d}}  to cover a larger range of content and average their logits for evaluation. For video captioning task, we use center crop after resizing.
Data Parallel for Vocabulary and Class Labels
Naive implementation of language modeling and ImageNet-21k classification is impractical due to memory limitation. In language modeling, we need to compare the feature of a <SPE> token in a sequence to the feature of each token in our tokenizer. Since the vocabulary size is large, we apply data parallel for the vocabulary set. A similar method is used for class labels of ImageNet-21k.
Removing Overlap
For the training set of K700 participating in pre-training, we remove those videos overlapping with validation set of K400.

Licences of Datasets
ImageNet-21K {{cite:c175469}} is subject to the ImageNet terms of use imagenetterms.
Kinetics-700 Smaira2020ASN & Kinetics-400 {{cite:e7e2f79}} The kinetics dataset is licensed by Google Inc. under a Creative Commons Attribution 4.0 International License.
BooksCorpus {{cite:d18c245}} Replicate Toronto BookCorpus is open-source and licensed under GNU GPL, Version 3.
Wikipedia Most of Wikipedia's text is co-licensed under the Creative Commons Attribution-ShareAlike 3.0 Unported License (CC BY-SA) and the GNU Free Documentation License (GFDL) (unversioned, with no invariant sections, front-cover texts, or back-cover texts). Some text has been imported only under CC BY-SA and CC BY-SA-compatible license and cannot be reused under GFDL.
YFCC {{cite:f30e7c2}} All the photos and videos provided in YFCC dataset are licensed under one of the Creative Commons copyright licenses.
CC12M {{cite:940e960}} is licensed under the Terms of Use of Conceptual 12M cc12mlicense.
CC3M {{cite:477fe9e}} is licensed under the Conceptual Captions Terms of Use cc3mlicense.
Visual Genome {{cite:3ff07fb}} is licensed under a Creative Commons Attribution 4.0 International License vgterms.
COCO Caption {{cite:8f5f427}} The images are subject to the Flickr terms of use flickr2020terms.
SBU Caption {{cite:30a3919}} The images are subject to the Flickr terms of use flickr2020terms.
PAQ {{cite:eec85c7}} is licensed under the Attribution-NonCommercial 4.0 International License.

ieeefullname
egbib


Towards Causal VQA: Revealing and Reducing Spurious Correlations by Invariant and Covariant Semantic Editing

Abstract: Despite significant success in Visual Question Answering (VQA), VQA models have been shown to be notoriously brittle to linguistic variations in the questions. Due to deficiencies in models and datasets, today’s models often rely on correlations rather than predictions that are causal w.r.t. data. In this paper, we propose a novel way to analyze and measure the robustness of the state of the art models w.r.t semantic visual variations as well as propose ways to make models more robust against spurious correlations. Our method performs automated semantic image manipulations and tests for consistency in model predictions to quantify the model robustness as well as generate synthetic data to counter these problems. We perform our analysis on three diverse, state of the art VQA models and diverse question types with a particular focus on challenging counting questions. In addition, we show that models can be made significantly more robust against inconsistent predictions using our edited data. Finally, we show that results also translate to real-world error cases of state of the art models, which results in improved overall performance

Adversarial VQA: A New Benchmark for Evaluating the Robustness of VQA Models

Abstract: Benefiting from large-scale pre-training, we have witnessed significant performance boost on the popular Visual Question Answering (VQA) task. Despite rapid progress, it remains unclear whether these state-of-the-art (SOTA) models are robust when encountering examples in the wild. To study this, we introduce Adversarial VQA, a new large-scale VQA benchmark, collected iteratively via an adversarial human-and-model-in-the-loop procedure. Through this new benchmark, we discover several interesting findings. (i) Surprisingly, we find that during dataset collection, non-expert annotators can easily attack SOTA VQA models successfully. (ii) Both large-scale pre-trained models and adversarial training methods achieve far worse performance on the new benchmark than over standard VQA v2 dataset, revealing the fragility of these models while demonstrating the effectiveness of our adversarial dataset. (iii) When used for data augmentation, our dataset can effectively boost model performance on other robust VQA benchmarks. We hope our Adversarial VQA dataset can shed new light on robustness study in the community and serve as a valuable benchmark for future work.

Current causal image question answering datasets.

Image question answering benchmarks evaluate models' capability of answering natural language questions based on a corresponded image. Recent image question answering benchmarks try to collect or generate balanced QA pairs to make the dataset distribution more balanced in question distribution. VQA v2.0 {{cite:119de8a}} collects complementary QA pairs by replacing the image and the answer in a QA pairs. VQA-CP {{cite:9b576be}} resplited the VQA v1 dataset and VQA v2 dataset to construct two new dataset VQA-CP v1 and VQA-CP v2. As Fig. {{figure:340be4f5-7114-4316-abfe-f9f123b568b7}}  shows, {{cite:2a7e2de}} constructed IV-VQA and CV-VQA datasets using semantic editing to generate image and reexamine the image by human. {{cite:04a31ab}} proposed a human-machine adversarial to collect robust QA pairs. Fig. {{figure:9b56a195-c0c5-480f-8af9-60f46c24e3a6}}  illustrates the adversarial data collecting procedure. In Table {{table:6877649f-acd5-4618-b77e-09b97191b80b}} , we summarize these datasets in terms of image source, split numbers, collected or not, and rebalanced or not.


Simple online and realtime tracking

Abstract: This paper explores a pragmatic approach to multiple object tracking where the main focus is to associate objects efficiently for online and realtime applications. To this end, detection quality is identified as a key factor influencing tracking performance, where changing the detector can improve tracking by up to 18.9%. Despite only using a rudimentary combination of familiar techniques such as the Kalman Filter and Hungarian algorithm for the tracking components, this approach achieves an accuracy comparable to state-of-the-art online trackers. Furthermore, due to the simplicity of our tracking method, the tracker updates at a rate of 260 Hz which is over 20x faster than other state-of-the-art trackers.

Simple online and realtime tracking with a deep association metric

Abstract: Simple Online and Realtime Tracking (SORT) is a pragmatic approach to multiple object tracking with a focus on simple, effective algorithms. In this paper, we integrate appearance information to improve the performance of SORT. Due to this extension we are able to track objects through longer periods of occlusions, effectively reducing the number of identity switches. In spirit of the original framework we place much of the computational complexity into an offline pre-training stage where we learn a deep association metric on a largescale person re-identification dataset. During online application, we establish measurement-to-track associations using nearest neighbor queries in visual appearance space. Experimental evaluation shows that our extensions reduce the number of identity switches by 45%, achieving overall competitive performance at high frame rates.

Performance comparison on DanceTrack dataset between Z-GMOT and fully supervised MOT methods.

Table {{table:848ea779-b687-44c4-ab0e-b643ff12d05e}}  presents the performance evaluation of Z-GMOT on the DanceTrack dataset {{cite:8767554}} in comparison with other fully-supervised MOT methods. Despite not utilizing any training data, our proposed Z-GMOT achieves impressive results. For example, when using the SORT association, Z-GMOT achieves a HOTA of 45.28%, a MOTA of 82.95%, and an IDF1 of 45.65%, which is highly compatible with the performance of the fully-supervised MOT that employs YOLOX {{cite:8b36722}} as the object detector.


Public Kaggle Competition “IceCube - Neutrinos in Deep ice”

Abstract: The reconstruction of neutrino events in the IceCube experiment is crucial for many scientific analyses, including searches for cosmic neutrino sources. The Kaggle competition"IceCube -- Neutrinos in Deep ice"was a public machine learning challenge designed to encourage the development of innovative solutions to improve the accuracy and efficiency of neutrino event reconstruction. Participants worked with a dataset of simulated neutrino events and were tasked with creating a suitable model to predict the direction vector of incoming neutrinos. From January to April 2023, hundreds of teams competed for a total of $50k prize money, which was awarded to the best performing few out of the many thousand submissions. In this contribution I will present some insights into the organization of this large outreach project, and summarize some of the main findings, results and takeaways.

The HAM10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions

Abstract: Training of neural networks for automated diagnosis of pigmented skin lesions is hampered by the small size and lack of diversity of available datasets of dermatoscopic images. We tackle this problem by releasing the HAM10000 ("Human Against Machine with 10000 training images") dataset. We collected dermatoscopic images from different populations acquired and stored by different modalities. Given this diversity we had to apply different acquisition and cleaning methods and developed semi-automatic workflows utilizing specifically trained neural networks. The final dataset consists of 10015 dermatoscopic images which are released as a training set for academic machine learning purposes and are publicly available through the ISIC archive. This benchmark dataset can be used for machine learning and for comparisons with human experts. Cases include a representative collection of all important diagnostic categories in the realm of pigmented lesions. More than 50% of lesions have been confirmed by pathology, while the ground truth for the rest of the cases was either follow-up, expert consensus, or confirmation by in-vivo confocal microscopy.

Skin Lesion Analysis toward Melanoma Detection: A Challenge at the International Symposium on Biomedical Imaging (ISBI) 2016, hosted by the International Skin Imaging Collaboration (ISIC)

Abstract: In this article, we describe the design and implementation of a publicly accessible dermatology image analysis benchmark challenge. The goal of the challenge is to sup- port research and development of algorithms for automated diagnosis of melanoma, a lethal form of skin cancer, from dermoscopic images. The challenge was divided into sub-challenges for each task involved in image analysis, including lesion segmentation, dermoscopic feature detection within a lesion, and classification of melanoma. Training data included 900 images. A separate test dataset of 379 images was provided to measure resultant performance of systems developed with the training data. Ground truth for both training and test sets was generated by a panel of dermoscopic experts. In total, there were 79 submissions from a group of 38 participants, making this the largest standardized and comparative study for melanoma diagnosis in dermoscopic images to date. While the official challenge duration and ranking of participants has concluded, the datasets remain available for further research and development.

A Public Image Database for Benchmark of Plant Seedling Classification Algorithms

Abstract: A database of images of approximately 960 unique plants belonging to 12 species at several growth stages is made publicly available. It comprises annotated RGB images with a physical resolution of roughly 10 pixels per mm. To standardise the evaluation of classification results obtained with the database, a benchmark based on $f_{1}$ scores is proposed. The dataset is available at this https URL

Description of the biomedical datasets employed in our experiments.

In this work, we propose a benchmark of 10 partially annotated biomedical datasets, described in Table {{table:1d90fe69-4013-479b-9b80-0e642e4b3a8e}} , and evaluate the performance of deep learning models and semi-supervised methods using such a benchmark.


Pitfalls of Graph Neural Network Evaluation

Abstract: Semi-supervised node classification in graphs is a fundamental problem in graph mining, and the recently proposed graph neural networks (GNNs) have achieved unparalleled results on this task. Due to their massive success, GNNs have attracted a lot of attention, and many novel architectures have been put forward. In this paper we show that existing evaluation strategies for GNN models have serious shortcomings. We show that using the same train/validation/test splits of the same datasets, as well as making significant changes to the training procedure (e.g. early stopping criteria) precludes a fair comparison of different architectures. We perform a thorough empirical evaluation of four prominent GNN models and show that considering different splits of the data leads to dramatically different rankings of models. Even more importantly, our findings suggest that simpler GNN architectures are able to outperform the more sophisticated ones if the hyperparameters and the training procedure are tuned fairly for all models.

Predicting multicellular function through multi-layer tissue networks

Abstract: Motivation: Understanding functions of proteins in specific human tissues is essential for insights into disease diagnostics and therapeutics, yet prediction of tissue-specific cellular function remains a critical challenge for biomedicine. Results: Here we present OhmNet, a hierarchy-aware unsupervised node feature learning approach for multi-layer networks. We build a multi-layer network, where each layer represents molecular interactions in a different human tissue. OhmNet then automatically learns a mapping of proteins, represented as nodes, to a neural embedding based low-dimensional space of features. OhmNet encourages sharing of similar features among proteins with similar network neighborhoods and among proteins activated in similar tissues. The algorithm generalizes prior work, which generally ignores relationships between tissues, by modeling tissue organization with a rich multiscale tissue hierarchy. We use OhmNet to study multicellular function in a multi-layer protein interaction network of 107 human tissues. In 48 tissues with known tissue-specific cellular functions, OhmNet provides more accurate predictions of cellular function than alternative approaches, and also generates more accurate hypotheses about tissue-specific protein actions. We show that taking into account the tissue hierarchy leads to improved predictive power. Remarkably, we also demonstrate that it is possible to leverage the tissue hierarchy in order to effectively transfer cellular functions to a functionally uncharacterized tissue. Overall, OhmNet moves from flat networks to multiscale models able to predict a range of phenotypes spanning cellular subsystems

Image-based Recommendations on Styles and Substitutes

Abstract: Humans inevitably develop a sense of the relationships between objects, some of which are based on their appearance. Some pairs of objects might be seen as being alternatives to each other (such as two pairs of jeans), while others may be seen as being complementary (such as a pair of jeans and a matching shirt). This information guides many of the choices that people make, from buying clothes to their interactions with each other. We seek here to model this human sense of the relationships between objects based on their appearance. Our approach is not based on fine-grained modeling of user annotations but rather on capturing the largest dataset possible and developing a scalable method for uncovering human notions of the visual relationships within. We cast this as a network inference problem defined on graphs of related images, and provide a large-scale dataset for the training and evaluation of the same. The system we develop is capable of recommending which clothes and accessories will go well together (and which will not), amongst a host of other applications.

Statistics of experimental datasets.

For link prediction tasks, we evaluate our proposed model and baselines on four types of real-world datasets, i.e., the co-authorship graph (CS), the protein-protein interactions graph (PPI), co-purchase graphs (Computers and Photo), and citation network datasets (Cora, CiteSeer and PubMed). Details of these datasets are summarised in Table {{table:37ebaa42-c9d0-4222-b443-6e29453d79ae}} .


Dynamic scene (multiple sources) {{cite:04209dc}}

L1-norm content loss, Adversarial loss, Perceptual loss

Content loss, Adversarial loss, Perceptual loss

L2-norm content loss, Perceptual loss, Adversarial loss

Perceptual loss, L2-norm content loss, Adversarial loss, Relativistic loss

L1-norm content loss, L1-norm Frequency reconstruction loss

Learning to Deblur

Abstract: We describe a learning-based approach to blind image deconvolution. It uses a deep layered architecture, parts of which are borrowed from recent work on neural network learning, and parts of which incorporate computations that are specific to image deconvolution. The system is trained end-to-end on a set of artificially generated training examples, enabling competitive performance in blind deconvolution, both with respect to quality and runtime.

A Neural Approach to Blind Motion Deblurring

Abstract: We present a new method for blind motion deblurring that uses a neural network trained to compute estimates of sharp image patches from observations that are blurred by an unknown motion kernel. Instead of regressing directly to patch intensities, this network learns to predict the complex Fourier coefficients of a deconvolution filter to be applied to the input patch for restoration. For inference, we apply the network independently to all overlapping patches in the observed image, and average its outputs to form an initial estimate of the sharp image. We then explicitly estimate a single global blur kernel by relating this estimate to the observed image, and finally perform non-blind deconvolution with this kernel. Our method exhibits accuracy and robustness close to state-of-the-art iterative methods, while being much faster when parallelized on GPU hardware.

From Motion Blur to Motion Flow: a Deep Learning Solution for Removing Heterogeneous Motion Blur

Abstract: Removing pixel-wise heterogeneous motion blur is challenging due to the ill-posed nature of the problem. The predominant solution is to estimate the blur kernel by adding a prior, but the extensive literature on the subject indicates the difficulty in identifying a prior which is suitably informative, and general. Rather than imposing a prior based on theory, we propose instead to learn one from the data. Learning a prior over the latent image would require modeling all possible image content. The critical observation underpinning our approach is thus that learning the motion flow instead allows the model to focus on the cause of the blur, irrespective of the image content. This is a much easier learning task, but it also avoids the iterative process through which latent image priors are typically applied. Our approach directly estimates the motion flow from the blurred image through a fully-convolutional deep neural network (FCN) and recovers the unblurred image from the estimated motion flow. Our FCN is the first universal end-to-end mapping from the blurred image to the dense motion flow. To train the FCN, we simulate motion flows to generate synthetic blurred-image-motion-flow pairs thus avoiding the need for human labeling. Extensive experiments on challenging realistic blurred images demonstrate that the proposed method outperforms the state-of-the-art.

Deep Generative Filter for Motion Deblurring

Abstract: Removing blur caused by camera shake in images has always been a challenging problem in computer vision literature due to its ill-posed nature. Motion blur caused due to the relative motion between the camera and the object in 3D space induces a spatially varying blurring effect over the entire image. In this paper, we propose a novel deep filter based on Generative Adversarial Network (GAN) architecture integrated with global skip connection and dense architecture in order to tackle this problem. Our model, while bypassing the process of blur kernel estimation, significantly reduces the test time which is necessary for practical applications. The experiments on the benchmark datasets prove the effectiveness of the proposed method which outperforms the state-of-the-art blind deblurring algorithms both quantitatively and qualitatively.

Deep Multi-scale Convolutional Neural Network for Dynamic Scene Deblurring

Abstract: Non-uniform blind deblurring for general dynamic scenes is a challenging computer vision problem as blurs arise not only from multiple object motions but also from camera shake, scene depth variation. To remove these complicated motion blurs, conventional energy optimization based methods rely on simple assumptions such that blur kernel is partially uniform or locally linear. Moreover, recent machine learning based methods also depend on synthetic blur datasets generated under these assumptions. This makes conventional deblurring methods fail to remove blurs where blur kernel is difficult to approximate or parameterize (e.g. object motion boundaries). In this work, we propose a multi-scale convolutional neural network that restores sharp images in an end-to-end manner where blur is caused by various sources. Together, we present multi-scale loss function that mimics conventional coarse-to-fine approaches. Furthermore, we propose a new large-scale dataset that provides pairs of realistic blurry image and the corresponding ground truth sharp image that are obtained by a high-speed camera. With the proposed model trained on this dataset, we demonstrate empirically that our method achieves the state-of-the-art performance in dynamic scene deblurring not only qualitatively, but also quantitatively.

Learning a Discriminative Prior for Blind Image Deblurring

Abstract: We present an effective blind image deblurring method based on a data-driven discriminative prior.Our work is motivated by the fact that a good image prior should favor clear images over blurred images.In this work, we formulate the image prior as a binary classifier which can be achieved by a deep convolutional neural network (CNN).The learned prior is able to distinguish whether an input image is clear or not.Embedded into the maximum a posterior (MAP) framework, it helps blind deblurring in various scenarios, including natural, face, text, and low-illumination images.However, it is difficult to optimize the deblurring method with the learned image prior as it involves a non-linear CNN.Therefore, we develop an efficient numerical approach based on the half-quadratic splitting method and gradient decent algorithm to solve the proposed model.Furthermore, the proposed model can be easily extended to non-uniform deblurring.Both qualitative and quantitative experimental results show that our method performs favorably against state-of-the-art algorithms as well as domain-specific image deblurring approaches.

DeblurGAN: Blind Motion Deblurring Using Conditional Adversarial Networks

Abstract: We present DeblurGAN, an end-to-end learned method for motion deblurring. The learning is based on a conditional GAN and the content loss . DeblurGAN achieves state-of-the art performance both in the structural similarity measure and visual appearance. The quality of the deblurring model is also evaluated in a novel way on a real-world problem -- object detection on (de-)blurred images. The method is 5 times faster than the closest competitor -- DeepDeblur. We also introduce a novel method for generating synthetic motion blurred images from sharp ones, allowing realistic dataset augmentation. The model, code and the dataset are available at https://github.com/KupynOrest/DeblurGAN

Scale-recurrent Network for Deep Image Deblurring

Abstract: In single image deblurring, the "coarse-to-fine" scheme, i.e. gradually restoring the sharp image on different resolutions in a pyramid, is very successful in both traditional optimization-based methods and recent neural-network-based approaches. In this paper, we investigate this strategy and propose a Scale-recurrent Network (SRN-DeblurNet) for this deblurring task. Compared with the many recent learning-based approaches in [25], it has a simpler network structure, a smaller number of parameters and is easier to train. We evaluate our method on large-scale deblurring datasets with complex motion. Results show that our method can produce better quality results than state-of-the-arts, both quantitatively and qualitatively.

Deep Semantic Face Deblurring

Abstract: In this paper, we present an effective and efficient face deblurring algorithm by exploiting semantic cues via deep convolutional neural networks (CNNs). As face images are highly structured and share several key semantic components (e.g., eyes and mouths), the semantic information of a face provides a strong prior for restoration. As such, we propose to incorporate global semantic priors as input and impose local structure losses to regularize the output within a multi-scale deep CNN. We train the network with perceptual and adversarial losses to generate photo-realistic results and develop an incremental training strategy to handle random blur kernels in the wild. Quantitative and qualitative evaluations demonstrate that the proposed face deblurring algorithm restores sharp images with more facial details and performs favorably against state-of-the-art methods in terms of restoration quality, face recognition and execution speed.

Deep Stacked Hierarchical Multi-patch Network for Image Deblurring

Abstract: Despite deep end-to-end learning methods have shown their superiority in removing non-uniform motion blur, there still exist major challenges with the current multi-scale and scale-recurrent models: 1) Deconvolution/upsampling operations in the coarse-to-fine scheme result in expensive runtime; 2) Simply increasing the model depth with finer-scale levels cannot improve the quality of deblurring. To tackle the above problems, we present a deep hierarchical multi-patch network inspired by Spatial Pyramid Matching to deal with blurry images via a fine-to-coarse hierarchical representation. To deal with the performance saturation w.r.t. depth, we propose a stacked version of our multi-patch model. Our proposed basic multi-patch model achieves the state-of-the-art performance on the GoPro dataset while enjoying a 40x faster runtime compared to current multi-scale methods. With 30ms to process an image at 1280x720 resolution, it is the first real-time deep motion deblurring model for 720p images at 30fps. For stacked networks, significant improvements (over 1.2dB) are achieved on the GoPro dataset by increasing the network depth. Moreover, by varying the depth of the stacked model, one can adapt the performance and runtime of the same network for different application scenarios.

DeblurGAN-v2: Deblurring (Orders-of-Magnitude) Faster and Better

Abstract: We present a new end-to-end generative adversarial network (GAN) for single image motion deblurring, named DeblurGAN-v2, which considerably boosts state-of-the-art deblurring efficiency, quality, and flexibility. DeblurGAN-v2 is based on a relativistic conditional GAN with a double-scale discriminator. For the first time, we introduce the Feature Pyramid Network into deblurring, as a core building block in the generator of DeblurGAN-v2. It can flexibly work with a wide range of backbones, to navigate the balance between performance and efficiency. The plug-in of sophisticated backbones (e.g., Inception-ResNet-v2) can lead to solid state-of-the-art deblurring. Meanwhile, with light-weight backbones (e.g., MobileNet and its variants), DeblurGAN-v2 reaches 10-100 times faster than the nearest competitors, while maintaining close to state-of-the-art results, implying the option of real-time video deblurring. We demonstrate that DeblurGAN-v2 obtains very competitive performance on several popular benchmarks, in terms of deblurring quality (both objective and subjective), as well as efficiency. Besides, we show the architecture to be effective for general image restoration tasks too. Our codes, models and data are available at: https://github.com/KupynOrest/DeblurGANv2

Human-Aware Motion Deblurring

Abstract: This paper proposes a human-aware deblurring model that disentangles the motion blur between foreground (FG) humans and background (BG). The proposed model is based on a triple-branch encoder-decoder architecture. The first two branches are learned for sharpening FG humans and BG details, respectively; while the third one produces global, harmonious results by comprehensively fusing multi-scale deblurring information from the two domains. The proposed model is further endowed with a supervised, human-aware attention mechanism in an end-to-end fashion. It learns a soft mask that encodes FG human information and explicitly drives the FG/BG decoder-branches to focus on their specific domains. To further benefit the research towards Human-aware Image Deblurring, we introduce a large-scale dataset, named HIDE, which consists of 8,422 blurry and sharp image pairs with 65,784 densely annotated FG human bounding boxes. HIDE is specifically built to span a broad range of scenes, human object sizes, motion patterns, and background complexities. Extensive experiments on public benchmarks and our dataset demonstrate that our model performs favorably against the state-of-the-art motion deblurring methods, especially in capturing semantic details.

Deblurring by Realistic Blurring

Abstract: Existing deep learning methods for image deblurring typically train models using pairs of sharp images and their blurred counterparts. However, synthetically blurring images does not necessarily model the blurring process in real-world scenarios with sufficient accuracy. To address this problem, we propose a new method which combines two GAN models, i.e., a learning-to-Blur GAN (BGAN) and learning-to-DeBlur GAN (DBGAN), in order to learn a better model for image deblurring by primarily learning how to blur images. The first model, BGAN, learns how to blur sharp images with unpaired sharp and blurry image sets, and then guides the second model, DBGAN, to learn how to correctly deblur such images. In order to reduce the discrepancy between real blur and synthesized blur, a relativistic blur loss is leveraged. As an additional contribution, this paper also introduces a Real-World Blurred Image (RWBI) dataset including diverse blurry images. Our experiments show that the proposed method achieves consistently superior quantitative performance as well as higher perceptual quality on both the newly proposed dataset and the public GOPRO dataset.

Blind Image Deconvolution using Deep Generative Priors

Abstract: This paper proposes a novel approach to regularize the \textit{ill-posed} and \textit{non-linear} blind image deconvolution (blind deblurring) using deep generative networks as priors. We employ two separate generative models --- one trained to produce sharp images while the other trained to generate blur kernels from lower-dimensional parameters. To deblur, we propose an alternating gradient descent scheme operating in the latent lower-dimensional space of each of the pretrained generative models. Our experiments show promising deblurring results on images even under large blurs, and heavy noise. To address the shortcomings of generative models such as mode collapse, we augment our generative priors with classical image priors and report improved performance on complex image datasets. The deblurring performance depends on how well the range of the generator spans the image class. Interestingly, our experiments show that even an untrained structured (convolutional) generative networks acts as an image prior in the image deblurring context allowing us to extend our results to more diverse natural image datasets.

Rethinking Coarse-to-Fine Approach in Single Image Deblurring

Abstract: Coarse-to-fine strategies have been extensively used for the architecture design of single image deblurring networks. Conventional methods typically stack sub-networks with multi-scale input images and gradually improve sharpness of images from the bottom sub-network to the top sub-network, yielding inevitably high computational costs. Toward a fast and accurate deblurring network design, we revisit the coarse-to-fine strategy and present a multi-input multi-output U-net (MIMO-UNet). The MIMO-UNet has three distinct features. First, the single encoder of the MIMO-UNet takes multi-scale input images to ease the difficulty of training. Second, the single decoder of the MIMO-UNet outputs multiple deblurred images with different scales to mimic multi-cascaded U-nets using a single U-shaped network. Last, asymmetric feature fusion is introduced to merge multi-scale features in an efficient manner. Extensive experiments on the GoPro and RealBlur datasets demonstrate that the proposed network outperforms the state-of-the-art methods in terms of both accuracy and computational complexity. Source code is available for research purposes at https://github.com/chosj95/MIMO-UNet.

Gaussian Kernel Mixture Network for Single Image Defocus Deblurring

Abstract: Defocus blur is one kind of blur effects often seen in images, which is challenging to remove due to its spatially variant amount. This paper presents an end-to-end deep learning approach for removing defocus blur from a single image, so as to have an all-in-focus image for consequent vision tasks. First, a pixel-wise Gaussian kernel mixture (GKM) model is proposed for representing spatially variant defocus blur kernels in an efficient linear parametric form, with higher accuracy than existing models. Then, a deep neural network called GKMNet is developed by unrolling a fixed-point iteration of the GKM-based deblurring. The GKMNet is built on a lightweight scale-recurrent architecture, with a scale-recurrent attention module for estimating the mixing coefficients in GKM for defocus deblurring. Extensive experiments show that the GKMNet not only noticeably outperforms existing defocus deblurring methods, but also has its advantages in terms of model complexity and computational efficiency.

BANet: Blur-aware Attention Networks for Dynamic Scene Deblurring

Abstract: Image motion blur results from a combination of object motions and camera shakes, and such blurring effect is generally directional and non-uniform. Previous research attempted to solve non-uniform blurs using self-recurrent multiscale, multi-patch, or multi-temporal architectures with self-attention to obtain decent results. However, using self-recurrent frameworks typically lead to a longer inference time, while inter-pixel or inter-channel self-attention may cause excessive memory usage. This paper proposes a Blur-aware Attention Network (BANet), that accomplishes accurate and efficient deblurring via a single forward pass. Our BANet utilizes region-based self-attention with multi-kernel strip pooling to disentangle blur patterns of different magnitudes and orientations and cascaded parallel dilated convolution to aggregate multi-scale content features. Extensive experimental results on the GoPro and RealBlur benchmarks demonstrate that the proposed BANet performs favorably against the state-of-the-arts in blurred image restoration and can provide deblurred results in real-time.

The applications and loss functions of blind deblurring papers in chronological order

In image restoration tasks, including image deblurring, training loss functions play a significant role in restoring clearer images with more texture details {{cite:f626506}}. Among various choices available, the content loss that measures the difference between the restored outcome and the original target image is the most widely used, and this loss can further improve the quality of the restored image when combined with auxiliary terms {{cite:f42495f}}. Likewise, many studies in the literature combine multiple loss functions in the form of a weighted sum to take advantage of the benefits of various loss functions and enhance the quality of recovered image. In this section, we list several well-known loss functions and discuss their impacts on the restored outcome.
For an easier overview, Table {{table:6ffebe34-6eb0-401d-b1af-7c50b6ed4212}}  summarizes training loss functions as well as application types considered in the studies reviewed in this paper.


Trajectories of 20 buses in Athens, Georgia

Taxi and bus trajectories in San Francisco, USA

Timetables of 167 bus lines in Chengdu, China

Chemical Reaction Optimization (meta-heuristic)

Sensor deployment for air pollution monitoring using public transportation system

Abstract: Air pollution monitoring is a very popular research topic and many monitoring systems have been developed. In this paper, we formulate the Bus Sensor Deployment Problem (BSDP) to select the bus routes on which sensors are deployed, and we use Chemical Reaction Optimization (CRO) to solve BSDP. CRO is a recently proposed metaheuristic designed to solve a wide range of optimization problems. Using the real world data, namely Hong Kong Island bus route data, we perform a series of simulations and the results show that CRO is capable of solving this optimization problem efficiently.

Spatio-Temporal Coverage Enhancement in Drive-By Sensing Through Utility-Aware Mobile Agent Selection

Abstract: In recent years, the drive-by sensing paradigm has become increasingly popular for cost-effective monitoring of urban areas. Drive-by sensing is a form of crowdsensing wherein sensor-equipped vehicles (aka, mobile agents) are the primary data gathering agents. Enhancing the efficacy of drive-by sensing poses many challenges, an important one of which is to select non-dedicated mobile agents on which a limited number of sensors are to be mounted. This problem, which we refer to as the mobile-agent selection problem, has a significant impact on the spatio-temporal coverage of the drive-by sensing platforms and the resultant datasets. The challenge here is to achieve maximum spatiotemporal coverage while taking the relative importance levels of geographical areas into account. In this paper, we address this problem in the context of the SCOUTS project, the goal of which is to map and analyze the urban heat island phenomenon accurately. Our work makes several major technical contributions. First, we delineate a model for representing the mobile agents selection problem. This model takes into account the trajectories of the vehicles (public transportation buses in our case) and the relative importance of the urban regions, and formulates it as an optimization problem. Second, we provide two algorithms that are based upon the utility (coverage) values of mobile agents, namely, a hotspot-based algorithm that limits the search space to important sub-regions and a utility-aware genetic algorithm that enables the latter algorithm to make unbiased selections. Third, we design a highly efficient coverage redundancy minimization algorithm that, at each step, chooses the mobile agent, which provides maximal improvement to the spatio-temporal coverage. This paper reports a series of experiments on a real-world dataset from Athens, GA, USA, to demonstrate the effectiveness of the proposed approaches.

Modulo: Drive-by Sensing at City-scale on the Cheap

Abstract: Ambient air pollution in urban areas is a significant health hazard, with over 4.2 million deaths annually attributed to it. A crucial step in tackling these challenge is to measure air quality at a fine spatiotemporal granularity. A promising approach for several smart city projects, called drive-by sensing, is to leverage vehicles retrofitted with different sensors (pollution monitors, etc.) that can provide the desired spatiotemporal coverage at a fraction of the cost. However, deploying a drive-by sensing network at a city-scale to optimally select vehicles from a large fleet is still unexplored. In this paper, we propose Modulo -- a system to bootstrap drive-by sensing deployment by taking into consideration a variety of aspects such as spatiotemporal coverage, budget constraints. Modulo is well-suited to satisfy unique deployment constraints such as colocations with other sensors (needed for gas and PM sensor calibration), etc. We compare Modulo with two baseline algorithms on real-world taxi and bus datasets. Modulo significantly outperforms the baselines when a fleet comprises of both taxis and fixed-route vehicles such as public transport buses. Finally, we present a real-world case study that uses Modulo to select vehicles for an air pollution sensing application.

Trip-based mobile sensor deployment for drive-by sensing with bus fleets

Abstract: Drive-by sensing (i.e. vehicle-based mobile sensing) is an emerging data collection paradigm that leverages vehicle mobilities to scan a city at low costs. It represents a positive social externality of urban transport activities. Bus transit systems are widely considered in drive-by sensing due to extensive spatial coverage, reliable operations, and low maintenance costs. It is critical for the underlying monitoring scenario (e.g. air quality, traffic state, and road roughness) to assign a limited number of sensors to a bus fleet to ensure their optimal spatial-temporal distribution. In this paper we present a trip-based sensor deployment problem, which explicitly considers timetabled trips that must be executed by the fleet while a portion of them perform sensing tasks. To address the computational challenge in large-scale instances, we design a multi-stage solution framework that decouples the spatial-temporal structures of the sensing task through line pre-selection and bi-level optimization. As a result, the computational complexity is reduced to be sub-linear w.r.t. the number of lines, rather than combinatorial w.r.t. the number of buses in existing vehicle-based approaches. A real-world case study covering 400 km$^2$ in central Chengdu demonstrates the effectiveness of the model in solving large-scale problems. It is found that coordinating bus scheduling and sensing tasks can substantially increase the spatial-temporal sensing coverage. We also provide a few model extensions and recommendation for practice regarding the application of this method.

Summary of sensor deployment problems.

Table {{table:dd6e9b63-be22-4c6d-92c2-554c49a621d3}}  summarizes relevant literature on the sensor location problem for various vehicle types. They all focused on allocating sensors to vehicles or bus/tram lines based on actual or predicted vehicle trajectories and employ heuristic, meta-heuristic, or approximate algorithms to solve the problem. It is noted that all these studies fall within the category of opportunistic sensing {{cite:a9de2da}}, {{cite:8dccb0d}}.