mGPT: Few-Shot Learners Go Multilingual

Abstract: Abstract This paper introduces mGPT, a multilingual variant of GPT-3, pretrained on 61 languages from 25 linguistically diverse language families using Wikipedia and the C4 Corpus. We detail the design and pretraining procedure. The models undergo an intrinsic and extrinsic evaluation: language modeling in all languages, downstream evaluation on cross-lingual NLU datasets and benchmarks in 33 languages, and world knowledge probing in 23 languages. The in-context learning abilities are on par with the contemporaneous language models while covering a larger number of languages, including underrepresented and low-resource languages of the Commonwealth of Independent States and the indigenous peoples in Russia. The source code and the language models are publicly available under the MIT license.

mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer

Abstract: The recent “Text-to-Text Transfer Transformer” (T5) leveraged a unified text-to-text format and scale to attain state-of-the-art results on a wide variety of English-language NLP tasks. In this paper, we introduce mT5, a multilingual variant of T5 that was pre-trained on a new Common Crawl-based dataset covering 101 languages. We detail the design and modified training of mT5 and demonstrate its state-of-the-art performance on many multilingual benchmarks. We also describe a simple technique to prevent “accidental translation” in the zero-shot setting, where a generative model chooses to (partially) translate its prediction into the wrong language. All of the code and model checkpoints used in this work are publicly available.

Canine: Pre-training an Efficient Tokenization-Free Encoder for Language Representation

Abstract: Pipelined NLP systems have largely been superseded by end-to-end neural modeling, yet nearly all commonly used models still require an explicit tokenization step. While recent tokenization approaches based on data-derived subword lexicons are less brittle than manually engineered tokenizers, these techniques are not equally suited to all languages, and the use of any fixed vocabulary may limit a model’s ability to adapt. In this paper, we present Canine, a neural encoder that operates directly on character sequences—without explicit tokenization or vocabulary—and a pre-training strategy that operates either directly on characters or optionally uses subwords as a soft inductive bias. To use its finer-grained input effectively and efficiently, Canine combines downsampling, which reduces the input sequence length, with a deep transformer stack, which encodes context. Canine outperforms a comparable mBert model by 5.7 F1 on TyDi QA, a challenging multilingual benchmark, despite having fewer model parameters.

Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignment

Abstract: The cross-lingual language models are typically pretrained with masked language modeling on multilingual text or parallel sentences. In this paper, we introduce denoising word alignment as a new cross-lingual pre-training task. Specifically, the model first self-label word alignments for parallel sentences. Then we randomly mask tokens in a bitext pair. Given a masked token, the model uses a pointer network to predict the aligned token in the other language. We alternately perform the above two steps in an expectation-maximization manner. Experimental results show that our method improves cross-lingual transferability on various datasets, especially on the token-level tasks, such as question answering, and structured prediction. Moreover, the model can serve as a pretrained word aligner, which achieves reasonably low error rate on the alignment benchmarks. The code and pretrained parameters are available at github.com/CZWin32768/XLM-Align.

XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale

Abstract: This paper presents XLS-R, a large-scale model for cross-lingual speech representation learning based on wav2vec 2.0. We train models with up to 2B parameters on nearly half a million hours of publicly available speech audio in 128 languages, an order of magnitude more public data than the largest known prior work. Our evaluation covers a wide range of tasks, domains, data regimes and languages, both high and low-resource. On the CoVoST-2 speech translation benchmark, we improve the previous state of the art by an average of 7.4 BLEU over 21 translation directions into English. For speech recognition, XLS-R improves over the best known prior work on BABEL, MLS, CommonVoice as well as VoxPopuli, lowering error rates by 14-34% relative on average. XLS-R also sets a new state of the art on VoxLingua107 language identification. Moreover, we show that with sufficient model size, cross-lingual pretraining can outperform English-only pretraining when translating English speech into other languages, a setting which favors monolingual pretraining. We hope XLS-R can help to improve speech processing tasks for many more languages of the world.

11 mPLMs considered in the paper. Layer denotes the number of layers used for measuring similarity. Both the static embedding layer and all layers of the transformer are considered. For encoder-decoder architectures, we only consider the encoder. |Lang|: the number of languages covered. Arch.: Architecture. Enc: Encoder. Dec: Decoder. MLM: Masked Language Modeling. CLM: Causal Language Modeling. TLM: Translation Language Modeling. NSP: Next Sentence Prediction. DWA: Denoising Word Alignment. MT: Machine Translation. MSP: Masked Speech Prediction. CC: CommonCrawl.

We consider a varied set of 11 mPLMs for our investigation, differing in model size, number of covered languages, architecture, modality, and data used for pretraining. Full list and detailed information of the selected mPLMs are shown in Tab. {{table:87cbeed0-a546-47cc-8640-5144aa28d18c}} .


PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space

Abstract: Few prior works study deep learning on point sets. PointNet by Qi et al. is a pioneer in this direction. However, by design PointNet does not capture local structures induced by the metric space points live in, limiting its ability to recognize fine-grained patterns and generalizability to complex scenes. In this work, we introduce a hierarchical neural network that applies PointNet recursively on a nested partitioning of the input point set. By exploiting metric space distances, our network is able to learn local features with increasing contextual scales. With further observation that point sets are usually sampled with varying densities, which results in greatly decreased performance for networks trained on uniform densities, we propose novel set learning layers to adaptively combine features from multiple scales. Experiments show that our network called PointNet++ is able to learn deep point set features efficiently and robustly. In particular, results significantly better than state-of-the-art have been obtained on challenging benchmarks of 3D point clouds.

PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation

Abstract: Point cloud is an important type of geometric data structure. Due to its irregular format, most researchers transform such data to regular 3D voxel grids or collections of images. This, however, renders data unnecessarily voluminous and causes issues. In this paper, we design a novel type of neural network that directly consumes point clouds, which well respects the permutation invariance of points in the input. Our network, named PointNet, provides a unified architecture for applications ranging from object classification, part segmentation, to scene semantic parsing. Though simple, PointNet is highly efficient and effective. Empirically, it shows strong performance on par or even better than state of the art. Theoretically, we provide analysis towards understanding of what the network has learnt and why the network is robust with respect to input perturbation and corruption.

Comparing RSMix and PointMixup {{cite:4fa5d4d}} on ModelNet40 {{cite:7db17a6}}.

Comparison against PointMixup {{cite:4fa5d4d}}. We demonstrate results of RSMix for two evaluation methods against PointMixup {{cite:4fa5d4d}}, the closest work to us, in  {{table:3e34e2c3-5096-4197-aa01-7345546f48d5}} . We compared pre-aligned and unaligned settings to for single and multi-view accuracies, respectively, since PointMixup {{cite:4fa5d4d}} do not specify their evaluation method but each are similar. They follow the PointCNN {{cite:d369df4}} setting discriminating pre-aligned and unaligned with horizontal rotation on point cloud samples. They randomly rotate the training point cloud along the y-axis for unaligned settings. For natural evaluation, we do not preprocess the dataset as pre-aligned or unaligned (denoted as Raw in  {{table:3e34e2c3-5096-4197-aa01-7345546f48d5}} ). RSMix achieves more competitive performance than PointMixup {{cite:4fa5d4d}} for networks that use local information  {{cite:86de3b1}}, {{cite:f3f7adc}}, and further enhances the network’s ability to recognize local information.


Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks

Abstract: State-of-the-art object detection networks depend on region proposal algorithms to hypothesize object locations. Advances like SPPnet and Fast R-CNN have reduced the running time of these detection networks, exposing region proposal computation as a bottleneck. In this work, we introduce a Region Proposal Network (RPN) that shares full-image convolutional features with the detection network, thus enabling nearly cost-free region proposals. An RPN is a fully convolutional network that simultaneously predicts object bounds and objectness scores at each position. The RPN is trained end-to-end to generate high-quality region proposals, which are used by Fast R-CNN for detection. We further merge RPN and Fast R-CNN into a single network by sharing their convolutional features---using the recently popular terminology of neural networks with 'attention' mechanisms, the RPN component tells the unified network where to look. For the very deep VGG-16 model, our detection system has a frame rate of 5fps (including all steps) on a GPU, while achieving state-of-the-art object detection accuracy on PASCAL VOC 2007, 2012, and MS COCO datasets with only 300 proposals per image. In ILSVRC and COCO 2015 competitions, Faster R-CNN and RPN are the foundations of the 1st-place winning entries in several tracks. Code has been made publicly available

R-FCN: Object Detection via Region-based Fully Convolutional Networks

Abstract: We present region-based, fully convolutional networks for accurate and efficient object detection. In contrast to previous region-based detectors such as Fast/Faster R-CNN that apply a costly per-region subnetwork hundreds of times, our region-based detector is fully convolutional with almost all computation shared on the entire image. To achieve this goal, we propose position-sensitive score maps to address a dilemma between translation-invariance in image classification and translation-variance in object detection. Our method can thus naturally adopt fully convolutional image classifier backbones, such as the latest Residual Networks (ResNets), for object detection. We show competitive results on the PASCAL VOC datasets (e.g., 83.6% mAP on the 2007 set) with the 101-layer ResNet. Meanwhile, our result is achieved at a test-time speed of 170ms per image, 2.5-20 times faster than the Faster R-CNN counterpart. Code is made publicly available at: https://github.com/daijifeng001/r-fcn.

Searching for MobileNetV3

Abstract: We present the next generation of MobileNets based on a combination of complementary search techniques as well as a novel architecture design. MobileNetV3 is tuned to mobile phone CPUs through a combination of hardware-aware network architecture search (NAS) complemented by the NetAdapt algorithm and then subsequently improved through novel architecture advances. This paper starts the exploration of how automated search algorithms and network design can work together to harness complementary approaches improving the overall state of the art. Through this process we create two new MobileNet models for release: MobileNetV3-Large and MobileNetV3-Small which are targeted for high and low resource use cases. These models are then adapted and applied to the tasks of object detection and semantic segmentation. For the task of semantic segmentation (or any dense pixel prediction), we propose a new efficient segmentation decoder Lite Reduced Atrous Spatial Pyramid Pooling (LR-ASPP). We achieve new state of the art results for mobile classification, detection and segmentation. MobileNetV3-Large is 3.2% more accurate on ImageNet classification while reducing latency by 20% compared to MobileNetV2. MobileNetV3-Small is 6.6% more accurate compared to a MobileNetV2 model with comparable latency. MobileNetV3-Large detection is over 25% faster at roughly the same accuracy as MobileNetV2 on COCO detection. MobileNetV3-Large LR-ASPP is 34% faster than MobileNetV2 R-ASPP at similar accuracy for Cityscapes segmentation.

Objects as Points

Abstract: Detection identifies objects as axis-aligned boxes in an image. Most successful object detectors enumerate a nearly exhaustive list of potential object locations and classify each. This is wasteful, inefficient, and requires additional post-processing. In this paper, we take a different approach. We model an object as a single point --- the center point of its bounding box. Our detector uses keypoint estimation to find center points and regresses to all other object properties, such as size, 3D location, orientation, and even pose. Our center point based approach, CenterNet, is end-to-end differentiable, simpler, faster, and more accurate than corresponding bounding box based detectors. CenterNet achieves the best speed-accuracy trade-off on the MS COCO dataset, with 28.1% AP at 142 FPS, 37.4% AP at 52 FPS, and 45.1% AP with multi-scale testing at 1.4 FPS. We use the same approach to estimate 3D bounding box in the KITTI benchmark and human pose on the COCO keypoint dataset. Our method performs competitively with sophisticated multi-stage methods and runs in real-time.

Overview of Pedestrian Detection Models.

Performance characterization was carried out on seven popular pedestrian detection models. All models use publicly available pretrained weights from two popular model zoos {{cite:eade310}}{{cite:844be2c}} and are trained using the COCO “train 2017” dataset {{cite:8598539}}. An overview of the pedestrian detection models can be seen in Table {{table:ef235581-6dd8-4d63-a7c3-a941dd5c68dd}} .
The pedestrian detection models chosen for characterization can be divided into 3 categories: Two-Stage Frameworks, One-Stage Frameworks and Keypoint Estimation.
Two-stage frameworks such as FasterRCNN{{cite:50b1a39}}, MaskRCNN{{cite:18ad208}} and R-FCN{{cite:9b83251}} apply two separate networks to perform classification. One network is used to propose regions of interest and a dedicated second network performs object detection {{cite:8a9df60}}.
One-stage frameworks such as RetinaNet{{cite:a83187e}}, SSD{{cite:8b7a7a6}} and SSDLite{{cite:031669b}}{{cite:61c7139}} attempt to reduce computation and increase speed by performing object detection using a single feed forward convolutional network that does not interact with a region proposal module. RetinaNet also implements a novel method of “focal loss” which is used to reduce the imbalance between foreground and background classes during training with a view to increasing detection precision.
CenterNet{{cite:50c2fc6}} takes an alternative approach based on keypoint estimation. Objects are represented as a single point at their bounding box center identified by a heat map generated using a fully convolutional network. Other object features such as object size, orientation and pose are then regressed directly from the image features at the center location.


Wide Residual Networks

Abstract: Deep residual networks were shown to be able to scale up to thousands of layers and still have improving performance. However, each fraction of a percent of improved accuracy costs nearly doubling the number of layers, and so training very deep residual networks has a problem of diminishing feature reuse, which makes these networks very slow to train. To tackle these problems, in this paper we conduct a detailed experimental study on the architecture of ResNet blocks, based on which we propose a novel architecture where we decrease depth and increase width of residual networks. We call the resulting network structures wide residual networks (WRNs) and show that these are far superior over their commonly used thin and very deep counterparts. For example, we demonstrate that even a simple 16-layer-deep wide residual network outperforms in accuracy and efficiency all previous deep residual networks, including thousand-layer-deep networks, achieving new state-of-the-art results on CIFAR, SVHN, COCO, and significant improvements on ImageNet. Our code and models are available at this https URL

Very Deep Convolutional Networks for Large-Scale Image Recognition

Abstract: In this work we investigate the effect of the convolutional network depth on its accuracy in the large-scale image recognition setting. Our main contribution is a thorough evaluation of networks of increasing depth using an architecture with very small (3x3) convolution filters, which shows that a significant improvement on the prior-art configurations can be achieved by pushing the depth to 16-19 weight layers. These findings were the basis of our ImageNet Challenge 2014 submission, where our team secured the first and the second places in the localisation and classification tracks respectively. We also show that our representations generalise well to other datasets, where they achieve state-of-the-art results. We have made our two best-performing ConvNet models publicly available to facilitate further research on the use of deep visual representations in computer vision.

List of neural networks used in this study.For each model, we list the minimum and maximum number of GPUs used in ourstrong scaling runs. We choose the minimum and maximum GPU counts such thatthe ratio of batch size to number of GPUs is 4 and 1 respectively.

Description of Neural Networks and Hyperparameters

Table {{table:ef790d82-b61f-4de4-8039-4086b77be7e2}}  lists the set of neural networks and the
corresponding hyperparameters used in this study. The VGG-19 {{cite:0ca5111}} and
WideResnet-101 {{cite:ea8fbdb}} are two convolutional neural network (CNN) architectures widely used
in computer vision. The GPT-3 {{cite:1ca04a6}}, a variant of the transformer architecture {{cite:d25276b}},
is extremely popular in natural language processing for causal lanuage modelling. For every model, we use the same
hyperparameters (batch size, sequence length, learning rate schedules, gradient clipping, l2
regularization and optimizer hyperparameters) as used by the authors. We use the SGD (with momentum {{cite:b9eb597}}) and the
AdamW{{cite:69afff9}} optimizer for training the CNN and GPT-3 models respectively. We use MegatronLM's highly optimized kernels
to implement the GPT-3 models {{cite:7748792}}. For the convolution neural networks, we use implementations
provided by the torchvision libraryhttps://pytorch.org/vision/stable/index.html.

We profile the neural networks listed in Table {{table:ef790d82-b61f-4de4-8039-4086b77be7e2}}  under a
strong scaling setup to demonstrate the efficacy of our optimizations. For every
model, we choose the minimum and maximum GPU counts such that the ratio of batch
size to number of GPUs is 4 and 1 respectively. For a given model, we
fix the batch size irrespective of the GPU count. This is because while
increasing the batch size leads to better performance, it also degrades the
quality of convergence{{cite:c933e99}}. Under a strict definition of strong scaling,
the final answer should be the same irrespective of the number of GPUs. Therefore,
it is important to keep the global batch size fixed. For our approach, we prune the networks to
a sparsity of 90% using You et al.'s “Early Bird Ticket” algorithm{{cite:2b92b3b}}.

We integrate our optimizations in AxoNN  {{cite:87cd80a}} and refer to it
as “AxoNN+SAMO”. We use AxoNN and DeepSpeed-3D {{cite:4acc7bb}}, {{cite:0c78d3b}} as dense
baselines. DeepSpeed-3D implements a hybrid of data, inter-layer and intra-layer
parallelism for parallel model training. Their data parallel implementation uses
the ZeRO optimizer to shard optimizer state memory across data parallel
ranks{{cite:0c78d3b}}. They use MegatronLM's implementation of intra-layer
parallelism of transformers {{cite:7748792}}. DeepSpeed-3D has been used to
train some of the largest neural networks till date like
Bloom-176B{{cite:ad8de0d}} and Megatron-Turing NLG
530B{{cite:407a549}}. Finally, we integrate Sputnik
{{cite:c95b006}} in AxoNN to create a sparse matrix multiplication baseline.
Note that Sputnik does not support sparse convolutions, so we do not implement
the convolution architectures in Table {{table:ef790d82-b61f-4de4-8039-4086b77be7e2}}  using Sputnik.
We build our baselines using CUDA 11.0, PyTorch 1.12.0, NCCL 2.8.4, GCC 9.1.0
and Spectrum-MPI 10.4.0.3.

The CNNs used in this study are nearly 10–100{{formula:3029051b-f6df-4ba3-b219-71f2fad45c0d}}  smaller than the GPT-3 based models (see Table {{table:ef790d82-b61f-4de4-8039-4086b77be7e2}} ).
Hence, all of AxoNN, DeepSpeed-3D and AxoNN+SAMO are able to run these architectures in a pure data parallel configuration,
with a full copy of the network on each GPU. Thus the only communication here is the all-reduce on the network
gradients. We illustrate these results in Figure {{figure:0d9b3f8c-8d49-4684-8117-351dcf33199c}} . We observe similar batch times for
both AxoNN and DeepSpeed-3D. This is explained by the fact that both these frameworks have very similar NCCL based implementations
of data parallelism. Our approach yields speedups of 7–12 % over WideResnet-101 and 18–44 % over VGG-19.
While both these speedups are significant, SAMO seems to benefit the latter architecture much more than the former.
This is because the WideResnet-101 architecture spends nearly 1.5 {{formula:51f445ca-dbda-4feb-acea-a77cd4ab2925}}  more time in the computation phase as compared to
VGG-19. Also, both these models have similar number of parameters and thus similar communication costs in the data-parallel
all-reduce. Thus the proportion of the batch time spent by the WideResnet-101 architecture in communication is significantly smaller
than VGG-19. Since our approach optimizes communication, the benefits for WideResnet-101 are smaller than that of VGG-19.


A Real-Time Wideband Neural Vocoder at 1.6 kb/s Using LPCNet

Abstract: Neural speech synthesis algorithms are a promising new approach for coding speech at very low bitrate. They have so far demonstrated quality that far exceeds traditional vocoders, at the cost of very high complexity. In this work, we present a low-bitrate neural vocoder based on the LPCNet model. The use of linear prediction and sparse recurrent networks makes it possible to achieve real-time operation on general-purpose hardware. We demonstrate that LPCNet operating at 1.6 kb/s achieves significantly higher quality than MELP and that uncompressed LPCNet can exceed the quality of a waveform codec operating at low bitrate. This opens the way for new codec designs based on neural synthesis models.

Improving Opus Low Bit Rate Quality with Neural Speech Synthesis

Abstract: The voice mode of the Opus audio coder can compress wideband speech at bit rates ranging from 6 kb/s to 40 kb/s. However, Opus is at its core a waveform matching coder, and as the rate drops below 10 kb/s, quality degrades quickly. As the rate reduces even further, parametric coders tend to perform better than waveform coders. In this paper we propose a backward-compatible way of improving low bit rate Opus quality by re-synthesizing speech from the decoded parameters. We compare two different neural generative models, WaveNet and LPCNet. WaveNet is a powerful, high-complexity, and high-latency architecture that is not feasible for a practical system, yet provides a best known achievable quality with generative models. LPCNet is a low-complexity, low-latency RNN-based generative model, and practically implementable on mobile phones. We apply these systems with parameters from Opus coded at 6 kb/s as conditioning features for the generative models. A listening test shows that for the same 6 kb/s Opus bit stream, synthesized speech using LPCNet clearly outperforms the output of the standard Opus decoder. This opens up ways to improve the decoding quality of existing speech and audio waveform coders without breaking compatibility.

Coders adopted in this study.

Speech Quality Datasets for Coded Speech

Table {{table:4186f186-94cd-4d7c-8d22-4b4b5d16f1fe}}  lists the speech coders that were employed for performance analysis in this study. The coders are sorted in descending order according to their MUSHRA scores. Fig. {{figure:aa1723ff-2ba2-4234-ac8d-091ece3ff699}}  shows these scores as a function of coder type. It can be seen that the best results were obtained by the LPCNet coder operating on unquantized features (LPCNetUnquan). The 9 kb/s Opus (Opus9.0) and 6 kb/s WaveNet (WaveNet6.0) coders achieved comparable results. The RNN-based coder at 1.6 kb/s (LPCNet1.6) outperformed the same coder operating on Opus quantized features at 6 kb/s (LPCNet6.0). Both WaveNet6.0 and LPCNet6.0 at provided higher quality results than that of the standard Opus decoder for the same 6 kb/s bit stream. The quality of MELP coder (source-filter coder) at 2.4 kb/s obtained better results than that of the Speex coder at 4 kb/s.


The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes

Abstract: This work proposes a new challenge set for multimodal classification, focusing on detecting hate speech in multimodal memes. It is constructed such that unimodal models struggle and only multimodal models can succeed: difficult examples ("benign confounders") are added to the dataset to make it hard to rely on unimodal signals. The task requires subtle reasoning, yet is straightforward to evaluate as a binary classification problem. We provide baseline performance numbers for unimodal models, as well as for multimodal models with various degrees of sophistication. We find that state-of-the-art methods perform poorly compared to humans (64.73% vs. 84.7% accuracy), illustrating the difficulty of the task and highlighting the challenge that this important problem poses to the community.

EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classification

Abstract: In this paper, we present a patch-based land use and land cover classification approach using Sentinel-2 satellite images. The Sentinel-2 satellite images are openly and freely accessible, and are provided in the earth observation program Copernicus. We present a novel dataset, based on these images that covers 13 spectral bands and is comprised of ten classes with a total of 27 000 labeled and geo-referenced images. Benchmarks are provided for this novel dataset with its spectral bands using state-of-the-art deep convolutional neural networks. An overall classification accuracy of 98.57% was achieved with the proposed novel dataset. The resulting classification system opens a gate toward a number of earth observation applications. We demonstrate how this classification system can be used for detecting land use and land cover changes, and how it can assist in improving geographical maps. The geo-referenced dataset EuroSAT is made publicly available at https://github.com/phelber/eurosat.

Remote Sensing Image Scene Classification: Benchmark and State of the Art

Abstract: Remote sensing image scene classification plays an important role in a wide range of applications and hence has been receiving remarkable attention. During the past years, significant efforts have been made to develop various datasets or present a variety of approaches for scene classification from remote sensing images. However, a systematic review of the literature concerning datasets and methods for scene classification is still lacking. In addition, almost all existing datasets have a number of limitations, including the small scale of scene classes and the image numbers, the lack of image variations and diversity, and the saturation of accuracy. These limitations severely limit the development of new approaches especially deep learning-based methods. This paper first provides a comprehensive review of the recent progress. Then, we propose a large-scale dataset, termed "NWPU-RESISC45", which is a publicly available benchmark for REmote Sensing Image Scene Classification (RESISC), created by Northwestern Polytechnical University (NWPU). This dataset contains 31,500 images, covering 45 scene classes with 700 images in each class. The proposed NWPU-RESISC45 (i) is large-scale on the scene classes and the total image number, (ii) holds big variations in translation, spatial resolution, viewpoint, object pose, illumination, background, and occlusion, and (iii) has high within-class diversity and between-class similarity. The creation of this dataset will enable the community to develop and evaluate various data-driven algorithms. Finally, several representative methods are evaluated using the proposed dataset and the results are reported as a useful baseline for future research.

Describing Textures in the Wild

Abstract: Patterns and textures are defining characteristics of many natural objects: a shirt can be striped, the wings of a butterfly can be veined, and the skin of an animal can be scaly. Aiming at supporting this analytical dimension in image understanding, we address the challenging problem of describing textures with semantic attributes. We identify a rich vocabulary of forty-seven texture terms and use them to describe a large dataset of patterns collected in the wild.The resulting Describable Textures Dataset (DTD) is the basis to seek for the best texture representation for recognizing describable texture attributes in images. We port from object recognition to texture recognition the Improved Fisher Vector (IFV) and show that, surprisingly, it outperforms specialized texture descriptors not only on our problem, but also in established material recognition datasets. We also show that the describable attributes are excellent texture descriptors, transferring between datasets and tasks; in particular, combined with IFV, they significantly outperform the state-of-the-art by more than 8 percent on both FMD and KTHTIPS-2b benchmarks. We also demonstrate that they produce intuitive descriptions of materials and Internet images.

Fine-Grained Visual Classification of Aircraft

Abstract: This paper introduces FGVC-Aircraft, a new dataset containing 10,000 images of aircraft spanning 100 aircraft models, organised in a three-level hierarchy. At the finer level, differences between models are often subtle but always visually measurable, making visual recognition challenging but possible. A benchmark is obtained by defining corresponding classification tasks and evaluation protocols, and baseline results are presented. The construction of this dataset was made possible by the work of aircraft enthusiasts, a strategy that can extend to the study of number of other object classes. Compared to the domains usually considered in fine-grained visual classification (FGVC), for example animals, aircraft are rigid and hence less deformable. They, however, present other interesting modes of variation, including purpose, size, designation, structure, historical style, and branding.

The authors gratefully acknowledge Chenguang Zhu, Wenhao Yu, Yuwei Fang for the early insightful discussions on the use of dictionary for rare concepts in NLP task, Hao Cheng for the inspirations of external knowledge in open-domain QA. The project is partly done in the MSR-Berkeley collaboration programhttps://www.microsoft.com/en-us/research/collaboration/bair. The work depends on publicly available knowledge databases; we acknowledge all the original authors and contributors who made their “knowledge” public.
Sheng Shen and Kurt Keutzer are supported by Samsung SAIT, Intel corporation, Intel VLAB team, Intel One-API center of excellence, as well as funding through
BDD and BAIR.
The work of Sheng Shen, Anna Rohrbach and Trevor Darrell was supported in part by DoD including DARPA’s LwLL, PTG and/or SemaFor programs, as well as BAIR’s industrial alliance programs.
FER 2013: Kaggle challenges in representation learning facial expression
recognition.
https://www.kaggle.com/.
Harsh Agrawal, Karan Desai, Yufei Wang, Xinlei Chen, Rishabh Jain, Mark
Johnson, Dhruv Batra, Devi Parikh, Stefan Lee, and Peter Anderson.
nocaps: novel object captioning at scale.
In ICCV, 2019.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra,
C Lawrence Zitnick, and Devi Parikh.
Vqa: Visual question answering.
In ICCV, 2015.
Sören Auer, Christian Bizer, Georgi Kobilarov, Jens Lehmann, Richard
Cyganiak, and Zachary Ives.
Dbpedia: A nucleus for a web of open data.
In The semantic web, pages 722–735. Springer, 2007.
Sven Bambach, Stefan Lee, David Crandall, and Chen Yu.
Lending a hand: Detecting hands and recognizing activities in complex
egocentric interactions.
In ICCV, 2015.
Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor.
Freebase: a collaboratively created graph database for structuring
human knowledge.
In ACM SIGMOD, 2008.
Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza
Rutherford, Katie Millican, George van den Driessche, Jean-Baptiste Lespiau,
Bogdan Damoc, Aidan Clark, et al.
Improving language models by retrieving from trillions of tokens.
arXiv preprint arXiv:2112.04426, 2021.
Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool.
Food-101–mining discriminative components with random forests.
In ECCV, 2014.
Yingshan Chang, Mridu Narang, Hisami Suzuki, Guihong Cao, Jianfeng Gao, and
Yonatan Bisk.
Webqa: Multihop and multimodal qa.
arXiv preprint arXiv:2109.00590, 2021.
Soravit Changpinyo, Wei-Lun Chao, and Fei Sha.
Predicting visual exemplars of unseen classes for zero-shot learning.
In ICCV, 2017.
Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut.
Conceptual 12m: Pushing web-scale image-text pre-training to
recognize long-tail visual concepts.
In CVPR, 2021.
Jiaoyan Chen, Yuxia Geng, Zhuo Chen, Ian Horrocks, Jeff Z Pan, and Huajun Chen.
Knowledge-aware zero-shot learning: Survey and perspective.
IJCAI, 2021.
Gong Cheng, Junwei Han, and Xiaoqiang Lu.
Remote sensing image scene classification: Benchmark and state of the
art.
Proceedings of the IEEE, 2017.
Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea
Vedaldi.
Describing textures in the wild.
In CVPR, 2014.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.
Imagenet: A large-scale hierarchical image database.
In CVPR, 2009.
Li Deng.
The MNIST database of handwritten digit images for machine learning
research.
IEEE signal processing magazine, 2012.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.
Bert: Pre-training of deep bidirectional transformers for language
understanding.
arXiv preprint arXiv:1810.04805, 2018.
Mohamed Elhoseiny, Babak Saleh, and Ahmed Elgammal.
Write a classifier: Zero-shot learning using purely textual
descriptions.
In ICCV, 2013.
Mohamed Elhoseiny, Yizhe Zhu, Han Zhang, and Ahmed Elgammal.
Link the head to the “beak”: Zero shot learning from noisy text
description at part precision.
In CVPR, 2017.
Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew
Zisserman.
The pascal visual object classes (VOC) challenge.
IJCV, 2010.
Ali Farhadi, Ian Endres, Derek Hoiem, and David Forsyth.
Describing objects by their attributes.
In CVPR, 2009.
Li Fei-Fei, Rob Fergus, and Pietro Perona.
Learning generative visual models from few training examples: An
incremental Bayesian approach tested on 101 object categories.
In CVPR workshop, 2004.
Rob Fergus, Hector Bernal, Yair Weiss, and Antonio Torralba.
Semantic label sharing for learning with many categories.
In ECCV, 2010.
Jannik Fritsch, Tobias Kuehnl, and Andreas Geiger.
A new performance measure and evaluation benchmark for road detection
algorithms.
In ITSC. IEEE, 2013.
Andrea Frome, Greg S Corrado, Jon Shlens, Samy Bengio, Jeff Dean, Marc'Aurelio
Ranzato, and Tomas Mikolov.
Devise: A deep visual-semantic embedding model.
NeurIPS, 2013.
Yanwei Fu and Leonid Sigal.
Semi-supervised vocabulary-informed learning.
In CVPR, 2016.
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A
Wichmann, and Wieland Brendel.
Imagenet-trained cnns are biased towards texture; increasing shape
bias improves accuracy and robustness.
arXiv preprint arXiv:1811.12231, 2018.
Agrim Gupta, Piotr Dollar, and Ross Girshick.
Lvis: A dataset for large vocabulary instance segmentation.
In CVPR, 2019.
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang.
Realm: Retrieval-augmented language model pre-training.
arXiv preprint arXiv:2002.08909, 2020.
Weituo Hao, Chunyuan Li, Xiujun Li, Lawrence Carin, and Jianfeng Gao.
Towards learning a generic agent for vision-and-language navigation
via pre-training.
In CVPR, 2020.
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth.
EuroSat: A novel dataset and deep learning benchmark for land use
and land cover classification.
IEEE Journal of Selected Topics in Applied Earth Observations
and Remote Sensing, 2019.
Matthew Honnibal and Ines Montani.
spaCy 2: Natural language understanding with Bloom embeddings,
convolutional neural networks and incremental parsing.
To appear, 2017.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin
De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly.
Parameter-efficient transfer learning for nlp.
In ICML, 2019.
Drew A Hudson and Christopher D Manning.
Gqa: A new dataset for real-world visual reasoning and compositional
question answering.
In CVPR, 2019.
Dinesh Jayaraman and Kristen Grauman.
Zero shot recognition with unreliable attributes.
arXiv preprint arXiv:1409.4327, 2014.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V
Le, Yunhsuan Sung, Zhen Li, and Tom Duerig.
Scaling up visual and vision-language representation learning with
noisy text supervision.
arXiv preprint arXiv:2102.05918, 2021.
Douwe Kiela, Hamed Firooz, Aravind Mohan, Vedanuj Goswami, Amanpreet Singh,
Pratik Ringshia, and Davide Testuggine.
The hateful memes challenge: Detecting hate speech in multimodal
memes.
NeurIPS, 2020.
Wonjae Kim, Bokyung Son, and Ildoo Kim.
Vilt: Vision-and-language transformer without convolution or region
supervision.
arXiv preprint arXiv:2102.03334, 2021.
Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung,
Sylvain Gelly, and Neil Houlsby.
Big transfer (bit): General visual representation learning.
In ECCV, 2020.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei.
3d object representations for fine-grained categorization.
In ICCV workshops, 2013.
Alex Krizhevsky, Geoffrey Hinton, et al.
Learning multiple layers of features from tiny images.
2009.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur
Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin,
Kenton Lee, et al.
Natural questions: a benchmark for question answering research.
TACL, 2019.
Christoph H Lampert, Hannes Nickisch, and Stefan Harmeling.
Learning to detect unseen object classes by between-class attribute
transfer.
In CVPR, 2009.
Christoph H Lampert, Hannes Nickisch, and Stefan Harmeling.
Attribute-based classification for zero-shot visual object
categorization.
PAMI, 2013.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir
Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim
Rocktäschel, et al.
Retrieval-augmented generation for knowledge-intensive NLP tasks.
NeurIPS, 2020.
Ang Li, Allan Jabri, Armand Joulin, and Laurens Van Der Maaten.
Learning visual n-grams from web data.
In ICCV, 2017.
Chunyuan Li, Haotian Liu, Liunian Harold Li, Pengchuan Zhang, Jyoti Aneja,
Jianwei Yang, Ping Jin, Houdong Hu, Zicheng Liu, Yong Jae Lee, and Jianfeng
Gao.
ELEVATER: A benchmark and toolkit for evaluating language-augmented
visual models.
In NeurIPS Track on Datasets and Benchmarks, 2022.
Junnan Li, Ramprasaath R Selvaraju, Akhilesh Deepak Gotmare, Shafiq Joty,
Caiming Xiong, and Steven Hoi.
Align before fuse: Vision and language representation learning with
momentum distillation.
arXiv preprint arXiv:2107.07651, 2021.
Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang.
Visualbert: A simple and performant baseline for vision and language.
arXiv preprint arXiv:1908.03557, 2019.
Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li,
Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, et al.
Grounded language-image pre-training.
CVPR, 2022.
Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan
Wang, Houdong Hu, Li Dong, Furu Wei, et al.
Oscar: Object-semantics aligned pre-training for vision-language
tasks.
In ECCV, 2020.
Yangguang Li, Feng Liang, Lichen Zhao, Yufeng Cui, Wanli Ouyang, Jing Shao,
Fengwei Yu, and Junjie Yan.
Supervision exists everywhere: A data efficient contrastive
language-image pre-training paradigm.
arXiv preprint arXiv:2110.05208, 2021.
Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár.
Focal loss for dense object detection.
In ICCV, 2017.
Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva
Ramanan, Piotr Dollár, and C Lawrence Zitnick.
Microsoft coco: Common objects in context.
In ECCV, 2014.
Angli Liu, Jingfei Du, and Veselin Stoyanov.
Knowledge-augmented language model and its application to
unsupervised named-entity recognition.
arXiv preprint arXiv:1904.04458, 2019.
Weijie Liu, Peng Zhou, Zhe Zhao, Zhiruo Wang, Qi Ju, Haotang Deng, and Ping
Wang.
K-BERT: Enabling language representation with knowledge graph.
In AAAI, 2020.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and
Baining Guo.
Swin transformer: Hierarchical vision transformer using shifted
windows.
ICCV, 2021.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee.
Vilbert: Pretraining task-agnostic visiolinguistic representations
for vision-and-language tasks.
arXiv preprint arXiv:1908.02265, 2019.
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi.
Fine-grained visual classification of aircraft.
arXiv preprint arXiv:1306.5151, 2013.
Kenneth Marino, Xinlei Chen, Devi Parikh, Abhinav Gupta, and Marcus Rohrbach.
Krisp: Integrating implicit and symbolic knowledge for open-domain
knowledge-based VQA.
In CVPR, 2021.
Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi.
OK-VQA: A visual question answering benchmark requiring external
knowledge.
In CVPR, 2019.
Christian M Meyer and Iryna Gurevych.
Wiktionary: A new rival for expert-built lexicons? Exploring the
possibilities of collaborative lexicography.
na, 2012.
George A Miller.
WordNet: An electronic lexical database.
MIT press, 1998.
Norman Mu, Alexander Kirillov, David Wagner, and Saining Xie.
Slip: Self-supervision meets language-image pre-training.
arXiv preprint arXiv:2112.12750, 2021.
Maria-Elena Nilsback and Andrew Zisserman.
Automated flower classification over a large number of classes.
In Indian Conference on Computer Vision, Graphics & Image
Processing. IEEE, 2008.
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar.
Cats and dogs.
In CVPR, 2012.
Genevieve Patterson and James Hays.
SUN attribute database: Discovering, annotating, and recognizing
scene attributes.
In CVPR, 2012.
Matthew E Peters, Mark Neumann, Robert L Logan IV, Roy Schwartz, Vidur Joshi,
Sameer Singh, and Noah A Smith.
Knowledge enhanced contextual word representations.
arXiv preprint arXiv:1909.04164, 2019.
Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani,
Nicola De Cao, James Thorne, Yacine Jernite, Vladimir Karpukhin, Jean
Maillard, et al.
KILT: a benchmark for knowledge intensive language tasks.
arXiv preprint arXiv:2009.02252, 2020.
Ruizhi Qiao, Lingqiao Liu, Chunhua Shen, and Anton Van Den Hengel.
Less is more: zero-shot learning from online textual documents with
noise suppression.
In CVPR, 2016.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh,
Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark,
et al.
Learning transferable visual models from natural language
supervision.
arXiv preprint arXiv:2103.00020, 2021.
Scott Reed, Zeynep Akata, Honglak Lee, and Bernt Schiele.
Learning deep representations of fine-grained visual descriptions.
In CVPR, 2016.
Marcus Rohrbach, Michael Stark, and Bernt Schiele.
Evaluating knowledge transfer and zero-shot learning in a large-scale
setting.
In CVPR, 2011.
Ruslan Salakhutdinov, Antonio Torralba, and Josh Tenenbaum.
Learning to share visual appearance for multiclass object detection.
In CVPR, 2011.
Shuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng, Gang Yu, Xiangyu Zhang, Jing
Li, and Jian Sun.
Objects365: A large-scale, high-quality dataset for object detection.
In ICCV, 2019.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut.
Conceptual captions: A cleaned, hypernymed, image alt-text dataset
for automatic image captioning.
In ACL, 2018.
Sheng Shen, Liunian Harold Li, Hao Tan, Mohit Bansal, Anna Rohrbach, Kai-Wei
Chang, Zhewei Yao, and Kurt Keutzer.
How much can clip benefit vision-and-language tasks?
ICLR, 2022.
Richard Socher, Milind Ganjoo, Christopher D Manning, and Andrew Ng.
Zero-shot learning through cross-modal transfer.
NeurIPS, 2013.
Robyn Speer, Joshua Chin, and Catherine Havasi.
Conceptnet 5.5: An open multilingual graph of general knowledge.
In AAAI, 2017.
Johannes Stallkamp, Marc Schlipsing, Jan Salmen, and Christian Igel.
The german traffic sign recognition benchmark: a multi-class
classification competition.
In IJCNN, 2011.
Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai.
Vl-bert: Pre-training of generic visual-linguistic representations.
arXiv preprint arXiv:1908.08530, 2019.
Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta.
Revisiting unreasonable effectiveness of data in deep learning era.
In ICCV, 2017.
Hao Tan and Mohit Bansal.
Lxmert: Learning cross-modality encoder representations from
transformers.
In EMNLP, 2019.
Bart Thomee, David A Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni,
Douglas Poland, Damian Borth, and Li-Jia Li.
Yfcc100m: The new data in multimedia research.
Communications of the ACM, 2016.
Changyao Tian, Wenhai Wang, Xizhou Zhu, Xiaogang Wang, Jifeng Dai, and Yu Qiao.
Vl-ltr: Learning class-wise visual-linguistic representation for
long-tailed visual recognition.
arXiv preprint arXiv:2111.13579, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones,
Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.
Attention is all you need.
In NeurIPS, 2017.
Bastiaan S Veeling, Jasper Linmans, Jim Winkens, Taco Cohen, and Max Welling.
Rotation equivariant cnns for digital pathology.
In MICCAI, 2018.
Denny Vrandečić.
Wikidata: A new platform for collaborative data collection.
In WWW, 2012.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge
Belongie.
The caltech-ucsd birds-200-2011 dataset, 2011.
Ruize Wang, Duyu Tang, Nan Duan, Zhongyu Wei, Xuanjing Huang, Guihong Cao,
Daxin Jiang, Ming Zhou, et al.
K-adapter: Infusing knowledge into pre-trained models with
adapters.
arXiv preprint arXiv:2002.01808, 2020.
Xiaolong Wang, Yufei Ye, and Abhinav Gupta.
Zero-shot recognition via semantic embeddings and knowledge graphs.
In CVPR, 2018.
Jason Weston, Samy Bengio, and Nicolas Usunier.
Large scale image annotation: learning to rank with joint word-image
embeddings.
Machine learning, 2010.
Jialin Wu, Jiasen Lu, Ashish Sabharwal, and Roozbeh Mottaghi.
Multi-modal answer validation for knowledge-based VQA.
arXiv preprint arXiv:2103.12248, 2021.
Yongqin Xian, Christoph H Lampert, Bernt Schiele, and Zeynep Akata.
Zero-shot learning—a comprehensive evaluation of the good, the bad
and the ugly.
PAMI, 2018.
Jianwei Yang, Chunyuan Li, Pengchuan Zhang, Bin Xiao, Lu Yuan, Ce Liu, and
Jianfeng Gao.
Unified contrastive learning in image-text-label space.
CVPR, 2022.
Zhengyuan Yang, Zhe Gan, Jianfeng Wang, Xiaowei Hu, Yumao Lu, Zicheng Liu, and
Lijuan Wang.
An empirical study of GPT-3 for few-shot knowledge-based VQA.
arXiv preprint arXiv:2109.05014, 2021.
Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan
Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu.
Filip: Fine-grained interactive language-image pre-training.
arXiv preprint arXiv:2111.07783, 2021.
Da Yin, Li Dong, Hao Cheng, Xiaodong Liu, Kai-Wei Chang, Furu Wei, and Jianfeng
Gao.
A survey of knowledge-intensive nlp with pre-trained language models.
arXiv preprint arXiv:2202.08772, 2022.
Fei Yu, Jiji Tang, Weichong Yin, Yu Sun, Hao Tian, Hua Wu, and Haifeng Wang.
Ernie-vil: Knowledge enhanced vision-language representations through
scene graph.
arXiv preprint arXiv:2006.16934, 2020.
Wenhao Yu, Chenguang Zhu, Yuwei Fang, Donghan Yu, Shuohang Wang, Yichong Xu,
Michael Zeng, and Meng Jiang.
Dict-bert: Enhancing language model pre-training with dictionary.
arXiv preprint arXiv:2110.06490, 2021.
Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao,
Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, et al.
Florence: A new foundation model for computer vision.
arXiv preprint arXiv:2111.11432, 2021.
Hongming Zhang, Daniel Khashabi, Yangqiu Song, and Dan Roth.
Transomcs: From linguistic graphs to commonsense knowledge.
arXiv preprint arXiv:2005.00206, 2020.
Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang,
Yejin Choi, and Jianfeng Gao.
Vinvl: Revisiting visual representations in vision-language models.
In CVPR, 2021.

Supplementary Material for
“K-Lite: Learning Transferable Visual Models with External Knowledge”

This appendix is organized as follows.

In Section  (referred by CheckList), we discuss the societal impact.

In Section REF  (referred by Section 4.1), we summarize the statistics of the datasets used in in pre-training and evaluation stages.

In Section REF  (referred by Section 4), we introduce the pre-training and model adaptation details of our experiments on image classification (IC) and object detection (OD).

In Section REF , we provide zero-shot retrieval comparison by introducing knowledge.

In Section REF , we provide quantitative analysis on how external knowledge benefit transfer.

In Section REF  (referred by Section 4.2 and 4.3), we provide more visualizations of success and failure examples for K-Lite.

In Section REF , we provide more object detection results by training on larger dataset for K-Lite.

Societal Impact
We do not anticipate a specific negative impact, but, as with any Machine Learning method, we recommend to exercise caution. The existing knowledge bases such as Word-Net and Wiktionary are the results of crowd-sourcing various human knowledge or commonsense into a centered place. K-Lite provides evidence to leverage such knowledge bases for AI research. It encourages the community to contribute more to improve the coverage and quality of knowledge items, which will further benefit AI research.

Experimental Results
Dataset statistics
In Table {{table:338c0eaa-e1db-4381-8a1a-95689fc9a91c}}  and Table {{table:29361935-dc3c-4678-9941-79737cf0d332}} , we list the basic characteristics of the 20 IC datasets and the 13 OD datasets used in this paper, respectively.
Statistics of 20 datasets used in image classification.

Dataset
#Concepts
Train size
Test size
Evaluation metric
Source link
Hateful Memes {{cite:4c97072}}
2
8,500
500
ROC AUC
Facebook
PatchCamelyon {{cite:4bfc239}}
2
262,144
32,768
Accuracy
Tensorflow
Rendered-SST2 {{cite:7fe9359}}
2
6,920
1,821
Accuracy
OpenAI
KITTI Distance {{cite:b312c85}}
4
6,347
711
Accuracy
KITTI website
FER 2013 {{cite:e27eb61}}
7
28,709
3,589
Accuracy
Kaggle fer2013
CIFAR-10 {{cite:e3e333e}}
10
50,000
10,000
Accuracy
Tensorflow
EuroSAT {{cite:ec0f3e2}}
10
5,000
5,000
Accuracy
Tensorflow
MNIST {{cite:4a0b22d}}
10
60,000
10,000
Accuracy
Tensorflow
VOC 2007 Classification {{cite:a15e805}}
20
2,501
4,952
11-point mAP
VOC 2007
Oxford-IIIT Pets {{cite:593bb48}}
37
3,680
3,669
Mean-per-class
Oxford-IIIT Pets
GTSRB {{cite:dfd4b11}}
43
26,640
12,630
Accuracy
GTSRB website
Resisc-45 {{cite:095db02}}
45
3,150
25,200
Accuracy
Tensorflow
Describable Textures {{cite:65901c1}}
47
1,880
1,880
Accuracy
DTD website
CIFAR-100 {{cite:e3e333e}}
100
50,000
10,000
Accuracy
Tensorflow
FGVC Aircraft (variants) {{cite:475f691}}
100
3,334
3,333
Mean-per-class
FGVC website
Food-101 {{cite:de80fba}}
101
75,750
25,250
Accuracy
Tensorflow
Caltech-101 {{cite:4941a3f}}
102
3,060
6,084
Mean-per-class
Tensorflow
Oxford Flowers 102 {{cite:914170a}}
102
1,020
6,149
Mean-per-class
Tensorflow
Stanford Cars {{cite:6244429}}
196
8,144
8,041
Accuracy
Stanford Cars
Country-211 {{cite:7fe9359}}
211
31,650
21,100
Accuracy
OpenAI
{{table:250485fe-6531-47f8-8329-66f8fbc3bc8a}}{{table:338c0eaa-e1db-4381-8a1a-95689fc9a91c}}Statistics of 13 datasets used in object detection. Box mAP is used as the evaluation metric. Datasets are downloaded from Roboflow. For the datasets without a citation, we refer to Roboflow links for the original sources.

2*Dataset
2*#Concepts
#Image
#Annotated Regions
2*Source link

Train
Test
Train
Test

CottontailRabbits
1
1980
10
2070
11
Roboflow
EgoHands(generic) {{cite:df5b10b}}
1
3840
480
12015
1514
Roboflow
Packages
1
19
3
31
5
Roboflow
Raccoon
1
150
17
164
20
Roboflow
Pistols
1
2377
297
2728
358
Roboflow
Pothole
1
465
67
1256
154
Roboflow
NorthAmericaMushrooms
2
41
5
67
9
Roboflow
ThermalDogsAndPeople
2
142
20
181
27
Roboflow
ShellfishOpenImages
3
407
58
859
116
Roboflow
AerialMaritimeDrone(large)
5
52
7
873
78
Roboflow
VehiclesOpenImages
5
878
126
1676
258
Roboflow
Aquarium
7
448
63
3324
584
Roboflow
PascalVOC {{cite:a15e805}}
20
13690
3422
31356
7835
Roboflow
{{table:8d1150ad-7c39-45f9-9270-360bbc554118}}{{table:29361935-dc3c-4678-9941-79737cf0d332}}
Training schedule
UniCL for Image classification.
Following UniCL, we use Swin-Tiny {{cite:7c6b33a}} as the visual encoder.
For language encoder, we use a 12-layer Transformer {{cite:3c33bd9}} with hidden dimension of 512 following {{cite:7fe9359}}. Features from visual and textual encoder are projected to the same dimension of 512, using two linear projection layers.
All models including the baseline models are trained on 16 GPUs for 32 epochs, with batch size 4096, initial learning rate {{formula:679aea5e-bcf7-45e1-9e85-5ec4b6039faf}}  and weight decay 0.1. We also used a cosine learning rate scheduler with 5000 warmup iterations.

GLIP for Object Detection.
Following GLIP, we pre-train models
based on Swin-Tiny models with 32 GPUs and a batch size
of 64. We use a base learning rate of {{formula:55781a26-581d-48aa-9a3e-2051bb071667}}
for the language backbone and {{formula:157531f7-c100-42da-904b-74fd46f51b3c}}
for all other parameters. The learning rate is stepped down by a factor of 0.1
at the 67% and 89% of the total training steps.

Adapting to the downstream tasks.
To adapt the pre-trained model for downstream tasks on the evaluation benchmark, we follow the procedure in {{cite:b1dcaee}}, where automatic hyper-parameters are employed (including learning rate and weight decay) to ensure comparison fairness of pre-trained checkpoints, because human-in-the-loop hyper-parameter tuning is excluded in this process. All given training examples are first split into training and validation sets. We perform grid search to select the best hyper-parameter configurations based on the split validation set performance. After it, training is conducted on the entire training set with the selected best hyper-parameters, then the test set performance is reported. Please refer Section 4.1 of  {{cite:b1dcaee}} for details.

 More zero-shot results on retrieval
In Table {{table:ea7b02c3-fffe-4afe-93e4-c8a53a506a82}} , we present additional zero-shot performance of K-Lite on the representative COCO image-text retrieval task. We use COCO 2017 validation set, and show the recall@1 and 5 for image-to-text (I2T) and text-to-image (T2I) retrieval. As shown in the table, K-Lite shows consistent performance improvement over the UniCL baselines with the same amount of pre-training data. This reinforces the generalization ability of K-Lite as we demonstrate in Table {{table:338c0eaa-e1db-4381-8a1a-95689fc9a91c}} .
Overall comparisons of our knowledge-augmented models on zero-shot COCO Retrieval Evaluation. Each model is pre-trained with 32 epochs following UniCL {{cite:6ce06ae}}.
 Training Data
2*Method
 COCO Retrieval
Dataset
# Samples

I2T R@1
I2T R@5
T2I R@1
T2I R@5
22.5cmImageNet-21K
13M (full)
UniCL
2.66
7.46
0.98
3.54

13M (full)
K-Lite
Gray4.04
Gray12.20
Gray1.91
Gray6.48
22.5cmYFCC-14M + ImageNet-21K
14M (half)
UniCL
21.80
45.38
13.33
32.14

14M (half)
K-Lite Gray22.44
Gray47.28
Gray14.38
Gray33.77
12.5cmGCC-15M + ImageNet-21K
15M (half)
UniCL
31.88
57.76
21.41
44.69

15M (half)
K-Lite Gray32.68
Gray58.88
Gray22.08
Gray45.41
{{table:4406ce27-f9be-4403-951f-b94ffb66eafa}}
{{table:ea7b02c3-fffe-4afe-93e4-c8a53a506a82}}
 How does external knowledge affect visual prediction? Quantitative studies
To further quantitatively study how external knowledge affects the visual prediction, we first identify three factors:
{{formula:f6ac4201-5dc7-4c05-bef2-adf82502dbc4}}  Rareness: the inverse frequency of a downstream concept with respect to the pre-training corpus. Specifically, for rareness, we first count the frequency of the concept in the pre-training corpus (#images containing this concept) and the frequency of the concept in the downstream validation set (#images belonging to this concept). We then compute the weighted sum of the frequency, by summing the multiplication result of the two terms. Rareness score of a dataset is computed as the inverse of the weighted sum. A final normalization will be applied so that each number is in the range of 0 and 1. Higher value here denotes higher rareness.
{{formula:d0c733b2-b4fd-4218-a0ad-0d32477258e7}}  Overlap Difference: Overlap is defined as the percentage of concepts appearing in both training and evaluation; The overlap difference is defined as the absolute overlap improvement after adding external knowledge.
{{formula:76d496d3-4afd-40e4-845d-d09aba932791}}  Coverage: The percentage of concepts are covered by the external knowledge bases;
We compute the normalized number for each factor in Table {{table:4537e48a-d287-49b4-8a89-adad274a91f4}} .
We notice that the largest performance boost comes with Flowers2012 (+30.2), The Flowers2012 dataset has the 7th rarest concepts, high coverage and highest overlap difference (+34%). On the other side, while the StanfordCars dataset has the rarest concerts, the low overlap difference brings only +0.43 performance boost.
Furthermore, We computed the Pearson correlation coefficient and found that each factor has a positive correlation while the overlap (especially the difference of the overlap after introducing external knowledge) has the most significant (p=0.001<0.1) correlation with the performance improvement. This suggests that all three (rareness, coverage and overlap) affect the performance of K-Lite, and the overlap difference plays a major role.
Correlation of the improvement of K-Lite over UniCL on 20 datasets with normalized rareness, overlap difference and coverage.

Dataset
#Concepts
Improvement
Rareness
Overlap-Difference
Coverage
Hateful Memes {{cite:4c97072}}
2
6.64
0.33
0.00
1.00
PatchCamelyon {{cite:4bfc239}}
2
14.00
0.33
0.00
1.00
Rendered-SST2 {{cite:7fe9359}}
2
0.16
0.00
0.00
1.00
KITTI Distance {{cite:b312c85}}
4
13.30
0.01
0.01
1.00
FER 2013 {{cite:e27eb61}}
7
0.19
0.00
0.00
1.00
CIFAR-10 {{cite:e3e333e}}
10
0.15
0.00
0.00
1.00
EuroSAT {{cite:ec0f3e2}}
10
-2.50
0.00
0.03
1.00
MNIST {{cite:4a0b22d}}
10
0.09
0.00
0.01
0.60
VOC 2007 Classification {{cite:a15e805}}
20
-1.60
0.02
0.00
0.85
Oxford-IIIT Pets {{cite:593bb48}}
37
17.80
0.06
0.09
0.97
GTSRB {{cite:dfd4b11}}
43
3.95
0.04
0.02
1.00
Resisc-45 {{cite:095db02}}
45
3.29
0.03
0.06
0.98
Describable Textures {{cite:65901c1}}
47
4.57
0.02
0.11
1.00
CIFAR-100 {{cite:e3e333e}}
100
5.50
0.01
0.03
1.00
FGVC Aircraft (variants) {{cite:475f691}}
100
-0.10
0.00
0.01
0.10
Food-101 {{cite:de80fba}}
101
26.40
0.01
0.21
1.00
Caltech-101 {{cite:4941a3f}}
102
1.81
0.01
0.11
1.00
Oxford Flowers 102 {{cite:914170a}}
102
30.20
0.02
0.26
0.97
Stanford Cars {{cite:6244429}}
196
0.43
0.34
0.00
0.00
Country-211 {{cite:7fe9359}}
211
1.09
0.02
0.06
1.00
{{table:ce63684d-d1fb-44e9-9dac-5bc41e11de98}}{{table:4537e48a-d287-49b4-8a89-adad274a91f4}}
How does external knowledge affect visual prediction? Case studies
Image Classification.
In Figure {{figure:9a7d6bf1-50e5-4026-b965-9524c87a668c}} , we show more examples to illustrate how external knowledge affects visual recognition for the three datasets that knowledge benefit the most. Both success and failure examples are showed.
In Figure {{figure:c5a725a3-7928-4a7e-ad77-65cd6623ae74}} , we illustrate some cases that the current knowledge quality is low and its improvement is minor or unclear. Improving the knowledge quality can be one interesting research direction to boost the performance for these datasets.
Success examples. The description of the parent concept, material, shape, color etc. clarifies the concepts, boosting performance for the fine-grained classification tasks.

{{figure:e2c26e54-82c7-4ce0-b298-2b0444087d2f}}
{{figure:f5354371-dda9-4632-8425-ae270b2995a7}}

{{figure:3b291c66-b54b-4884-9a3a-0e5d094ee8d9}}
{{table:8ef5b072-b001-4b03-9b11-8973d0a86bfd}}
{{figure:06b2664d-544a-4a0d-bdc1-42d3262bb006}}
{{table:ea7db6a7-e270-4600-acce-a59358d82e7a}}[h]0.98

[h]0.98

Failure examples. The knowledge contains certain spurious description that confuse the models.The three datasets with the largest improvement according to Fig. {{figure:f5458e36-e52f-4560-93bb-af12bfbe21f0}} : Flowers102, Food101 and OxfordPets. For each dataset, two success examples and one failure example are shown in (a) and (b) respectively. For each image, the top row is the knowledge-based prediction, and the bottom row is the baseline prediction (no knowledge).{{figure:9a7d6bf1-50e5-4026-b965-9524c87a668c}}More case studies for the knowledge-augmented model: Top (Country211), Middle (MNIST), Bottom (PatchCamelyon). External knowledge does not benefit the first 2 datasets significantly (performance gain +1.09% and 0.0%), as the extracted knowledge is not quite relevant to the specific classification task. In PatchCamelyon, the external knowledge improves the baseline with +14.0% absolutely accuracy, we hypothesize the knowledge describes visual appearance of tumor tissue.

{{figure:b4b09ad5-e9ad-4532-832a-105262b6a117}}
{{figure:5ff853b3-05df-41c7-b42c-34557da3f62b}}
{{figure:253fe753-43c7-4d8b-8ba3-fca80f756d02}}
{{table:74db84e5-e53e-4dd5-afce-4da97750d2a5}}

{{figure:c5a725a3-7928-4a7e-ad77-65cd6623ae74}}Case study of success (Left & Middle) and failure (Right) for the knowledge-augmented model in object detection. For each image pair, the first one is the knowledge-based prediction (W), and the second one is the baseline prediction without knowledge (W/O). In the inference stage, the prediction box and category above a given threshold is used as the detection result. Only the categories with detected box regions are displayed.
{{figure:eb1e29ce-3132-43c7-9e8d-5bc476daab58}}
Object Detection.
In Figure {{figure:eb1e29ce-3132-43c7-9e8d-5bc476daab58}} , we provide examples of object detection results. The pre-training dataset Object365 has concepts “bread/bun, fire hydrant”, but does not contain concepts “doughnut, fireplug”. With external knowledge, the unseen concepts “doughnut, fireplug” are explained with their shape and similar seen concepts, helping the model to precisely locate the object regions and categorize them into the correct classes with higher confidence. However, similar to IC, there are failure cases due to the fact that external knowledge contains spurious words (the last example in Figure {{figure:eb1e29ce-3132-43c7-9e8d-5bc476daab58}} ), e.g., “water” appears in the definition of both “fireplug” and “garden hose”. It confuses the model in the categorization task, though the localization task is better executed.

Adding Grounding Data into Object Detection Training
In the main paper, we focus on pre-training on Object365 dataset. We further train our K-Lite on the combination of Object365 and gold grounding data (GoldG). Follow {{cite:acc5d0b}}, GoldG is 0.8M
human-annotated gold grounding data curated, including Flickr30K, VG Caption, and GQA, while COCO images are removed from the dataset. It is designed to verify the effectiveness of gold grounding data. The zero-shot performance on LVIS is shown in Table {{table:85acd9eb-b978-4f4e-9955-7bbfad504697}} , where the best numbers of GLIP and K-Lite are reported with their best settings (e.g., adding knowledge or not). By observing more concepts in grounding data, our proposed knowledge-augmented model can achieve superior performance in comparison to the baseline GLIP model. It shows that K-Lite can benefit from further scaling the training data.
Zero-shot task transfer performance on LVIS dataset. In {{cite:acc5d0b}}, GLIP-A is a two-encoder model without fusion module trained on Object365, and GLIP-C is a two-encoder model with fusion module trained on Object365 + Gold Grounding Data.
Method
# Parameters
# Training Data
LVIS

APr
APc
APf
AP
GLIP-A {{cite:acc5d0b}}
151M
2*Object-365
14.2
13.9
23.4
18.5
K-Lite
151M

14.8
18.6
24.8
21.3
GLIP-C {{cite:acc5d0b}}
231M
2*Object-365 + Gold Grounding Data
17.7
19.5
31.0
24.9
K-Lite
151M

17.2
24.6
29.0
26.1
{{table:0f9199a4-cc59-4b84-a68e-4cb278070535}}
{{table:85acd9eb-b978-4f4e-9955-7bbfad504697}}


Deep Convolutional Neural Networks for Computer-Aided Detection: CNN Architectures, Dataset Characteristics and Transfer Learning

Abstract: Remarkable progress has been made in image recognition, primarily due to the availability of large-scale annotated datasets and the revival of deep CNN. CNNs enable learning data-driven, highly representative, layered hierarchical image features from sufficient training data. However, obtaining datasets as comprehensively annotated as ImageNet in the medical imaging domain remains a challenge. There are currently three major techniques that successfully employ CNNs to medical image classification: training the CNN from scratch, using off-the-shelf pre-trained CNN features, and conducting unsupervised CNN pre-training with supervised fine-tuning. Another effective method is transfer learning, i.e., fine-tuning CNN models pre-trained from natural image dataset to medical image tasks. In this paper, we exploit three important, but previously understudied factors of employing deep convolutional neural networks to computer-aided detection problems. We first explore and evaluate different CNN architectures. The studied models contain 5 thousand to 160 million parameters, and vary in numbers of layers. We then evaluate the influence of dataset scale and spatial image context on performance. Finally, we examine when and why transfer learning from pre-trained ImageNet (via fine-tuning) can be useful. We study two specific computer-aided detection (CADe) problems, namely thoraco-abdominal lymph node (LN) detection and interstitial lung disease (ILD) classification. We achieve the state-of-the-art performance on the mediastinal LN detection, with 85% sensitivity at 3 false positive per patient, and report the first five-fold cross-validation classification results on predicting axial CT slices with ILD categories. Our extensive empirical evaluation, CNN model analysis and valuable insights can be extended to the design of high performance CAD systems for other medical imaging tasks.

CheXNet: Radiologist-Level Pneumonia Detection on Chest X-Rays with Deep Learning

Abstract: We develop an algorithm that can detect pneumonia from chest X-rays at a level exceeding practicing radiologists. Our algorithm, CheXNet, is a 121-layer convolutional neural network trained on ChestX-ray14, currently the largest publicly available chest X-ray dataset, containing over 100,000 frontal-view X-ray images with 14 diseases. Four practicing academic radiologists annotate a test set, on which we compare the performance of CheXNet to that of radiologists. We find that CheXNet exceeds average radiologist performance on the F1 metric. We extend CheXNet to detect all 14 diseases in ChestX-ray14 and achieve state of the art results on all 14 diseases.

Detecting Cancer Metastases on Gigapixel Pathology Images

Abstract: Each year, the treatment decisions for more than 230,000 breast cancer patients in the U.S. hinge on whether the cancer has metastasized away from the breast. Metastasis detection is currently performed by pathologists reviewing large expanses of biological tissues. This process is labor intensive and error-prone. We present a framework to automatically detect and localize tumors as small as 100 x 100 pixels in gigapixel microscopy images sized 100,000 x 100,000 pixels. Our method leverages a convolutional neural network (CNN) architecture and obtains state-of-the-art results on the Camelyon16 dataset in the challenging lesion-level tumor detection task. At 8 false positives per image, we detect 92.4% of the tumors, relative to 82.7% by the previous best automated approach. For comparison, a human pathologist attempting exhaustive search achieved 73.2% sensitivity. We achieve image-level AUC scores above 97% on both the Camelyon16 test set and an independent set of 110 slides. In addition, we discover that two slides in the Camelyon16 training set were erroneously labeled normal. Our approach could considerably reduce false negative rates in metastasis detection.

Deep Multi-scale Location-aware 3D Convolutional Neural Networks for Automated Detection of Lacunes of Presumed Vascular Origin

Abstract: Lacunes of presumed vascular origin (lacunes) are associated with an increased risk of stroke, gait impairment, and dementia and are a primary imaging feature of the small vessel disease. Quantification of lacunes may be of great importance to elucidate the mechanisms behind neuro-degenerative disorders and is recommended as part of study standards for small vessel disease research. However, due to the different appearance of lacunes in various brain regions and the existence of other similar-looking structures, such as perivascular spaces, manual annotation is a difficult, elaborative and subjective task, which can potentially be greatly improved by reliable and consistent computer-aided detection (CAD) routines. In this paper, we propose an automated two-stage method using deep convolutional neural networks (CNN). We show that this method has good performance and can considerably benefit readers. We first use a fully convolutional neural network to detect initial candidates. In the second step, we employ a 3D CNN as a false positive reduction tool. As the location information is important to the analysis of candidate structures, we further equip the network with contextual information using multi-scale analysis and integration of explicit location features. We trained, validated and tested our networks on a large dataset of 1075 cases obtained from two different studies. Subsequently, we conducted an observer study with four trained observers and compared our method with them using a free-response operating characteristic analysis. Shown on a test set of 111 cases, the resulting CAD system exhibits performance similar to the trained human observers and achieves a sensitivity of 0.974 with 0.13 false positives per slice. A feasibility study also showed that a trained human observer would considerably benefit once aided by the CAD system.

Summary of highly influential papers appeared in 2016 and 2017 (based on Google Scholar citation index in January 2019) that exploit deep learning for Detection/Localization in Medical Image Analysis.

Zhang et al. {{cite:ce72db4}} achieved {{formula:2a62f5c6-d6a5-49d4-8567-012d9a292552}} % accuracy and a localization error {{formula:0e5788f7-cbcd-4b0a-b817-7a1627915e62}} mm for the detection of inner ear in CT images. They used 3D U-Net {{cite:e839d44}} to map the whole 3D image which consists of multiple convolution-pooling layers that convert the raw input image into the low resolution and highly abstracted feature maps. They applied false positive suppression technique in the training process and used a shape based constraint during training.
Rajpurkar et al. {{cite:7ddb51b}} recently released a data set MURA which consists of 40,561 images from 14,863 musculoskeletal studies labeled by radiologists as either normal or abnormal. The authors used CNN with 169-layers for the detection of normality and abnormality in each image study. Li et al. {{cite:9e3740d}} proposed a Neural Conditional Random Field (NCRF) technique for the metastasis cancer detection in whole slide images. Their model was trained end-to-end using back-propagation and it obtained successful FROC score of 0.8096 in testing using Camelyon16 dataset {{cite:972ddb7}}.
Codella et al. {{cite:fdc083c}} recently organized a challenge at the International Symposium on Biomedical Imaging (ISBI), 2017 for skin lession analysis for melanoma detection. The challenge task provided 2,000 training images, 150 validation images, and 600 images for testing. It eventually published the results of 46 submission. We refer to {{cite:fdc083c}} for further details on the challenge itself and the submissions.
We also mention few techniques in Table {{table:c868d25a-4861-45c5-a7d0-fde5473239aa}}  related to the task of detection/localization. These methods appeared in the literature in the years 2016-17. Based on Google Scholar's citation index, these methods are among the highly influential techniques in the current related literature. This article also provides similar summaries of the highly influential papers from the years 2016-17 for each pattern recognition task considered in the Sections to follow.


Yummly-28K, CIFAR-10/-100, ESC-50, Wilderness

CUB-200, miniImageNet, AWA1, AWA2, Omniglot

SMIL: Multimodal Learning with Severely Missing Modality

Abstract: A common assumption in multimodal learning is the completeness of training data, i.e., full modalities are available in all training examples. Although there exists research endeavor in developing novel methods to tackle the incompleteness of testing data, e.g., modalities are partially missing in testing examples, few of them can handle incomplete training modalities. The problem becomes even more challenging if considering the case of severely missing, e.g., ninety percent of training examples may have incomplete modalities. For the first time in the literature, this paper formally studies multimodal learning with missing modality in terms of flexibility (missing modalities in training, testing, or both) and efficiency (most training data have incomplete modality). Technically, we propose a new method named SMIL that leverages Bayesian meta-learning in uniformly achieving both objectives. To validate our idea, we conduct a series of experiments on three popular benchmarks: MM-IMDb, CMU-MOSI, and avMNIST. The results prove the state-of-the-art performance of SMIL over existing methods and generative baselines including autoencoders and generative adversarial networks.

Learning from Multiple Cities: A Meta-Learning Approach for Spatial-Temporal Prediction

Abstract: Spatial-temporal prediction is a fundamental problem for constructing smart city, which is useful for tasks such as traffic control, taxi dispatching, and environment policy making. Due to data collection mechanism, it is common to see data collection with unbalanced spatial distributions. For example, some cities may release taxi data for multiple years while others only release a few days of data; some regions may have constant water quality data monitored by sensors whereas some regions only have a small collection of water samples. In this paper, we tackle the problem of spatial-temporal prediction for the cities with only a short period of data collection. We aim to utilize the long-period data from other cities via transfer learning. Different from previous studies that transfer knowledge from one single source city to a target city, we are the first to leverage information from multiple cities to increase the stability of transfer. Specifically, our proposed model is designed as a spatial-temporal network with a meta-learning paradigm. The meta-learning paradigm learns a well-generalized initialization of the spatial-temporal network, which can be effectively adapted to target cities. In addition, a pattern-based spatial-temporal memory is designed to distill long-term temporal information (i.e., periodicity). We conduct extensive experiments on two tasks: traffic (taxi and bike) prediction and water quality prediction. The experiments demonstrate the effectiveness of our proposed model over several competitive baseline models.

Multimodal Aggregation Approach for Memory Vision-Voice Indoor Navigation with Meta-Learning

Abstract: Vision and voice are two vital keys for agents’ interaction and learning. In this paper, we present a novel indoor navigation model called Memory Vision-Voice Indoor Navigation (MVV-IN), which receives voice commands and analyzes multimodal information of visual observation in order to enhance robots’ environment understanding. We make use of single RGB images taken by a rst-view monocular camera. We also apply a self-attention mechanism to keep the agent focusing on key areas. Memory is important for the agent to avoid repeating certain tasks unnecessarily and in order for it to adapt adequately to new scenes, therefore, we make use of meta-learning. We have experimented with various functional features extracted from visual observation. Comparative experiments prove that our methods outperform state-of-the-art baselines.

Meta-Learned Attribute Self-Gating for Continual Generalized Zero-Shot Learning

Abstract: Zero-shot learning (ZSL) has been shown to be a promising approach to generalizing a model to categories unseen during training by leveraging class attributes, but challenges still remain. Recently, methods using generative models to combat bias towards classes seen during training have pushed the state of the art of ZSL, but these generative models can be slow or computationally expensive to train. Additionally, while many previous ZSL methods assume a one-time adaptation to unseen classes, in reality, the world is always changing, necessitating a constant adjustment for deployed models. Models unprepared to handle a sequential stream of data are likely to experience catastrophic forgetting. We propose a meta-continual zero-shot learning (MCZSL) approach to address both these issues. In particular, by pairing self-gating of attributes and scaled class normalization with meta-learning based training, we are able to outperform state-of-the-art results while being able to train our models substantially faster ($>100\times$) than expensive generative-based approaches. We demonstrate this by performing experiments on five standard ZSL datasets (CUB, aPY, AWA1, AWA2 and SUN) in both generalized zero-shot learning and generalized continual zero-shot learning settings.

HetMAML: Task-Heterogeneous Model-Agnostic Meta-Learning for Few-Shot Learning Across Modalities

Abstract: Most of existing gradient-based meta-learning approaches to few-shot learning assume that all tasks have the same input feature space. However, in the real world scenarios, there are many cases that the input structures of tasks can be different, that is, different tasks may vary in the number of input modalities or data types. Existing meta-learners cannot handle the heterogeneous task distribution (HTD) as there is not only global meta-knowledge shared across tasks but also type-specific knowledge that distinguishes each type of tasks. To deal with task heterogeneity and promote fast within-task adaptions for each type of tasks, in this paper, we propose HetMAML, a task-heterogeneous model-agnostic meta-learning framework, which can capture both the type-specific and globally shared knowledge and can achieve the balance between knowledge customization and generalization. Specifically, we design a multi-channel backbone module that encodes the input of each type of tasks into the same length sequence of modality-specific embeddings. Then, we propose a task-aware iterative feature aggregation network which can automatically take into account the context of task-specific input structures and adaptively project the heterogeneous input spaces to the same lower-dimensional embedding space of concepts. Our experiments on six task-heterogeneous datasets demonstrate that HetMAML successfully leverages type-specific and globally shared meta-parameters for heterogeneous tasks and achieves fast within-task adaptions for each type of tasks.

Cross-Modal Generalization: Learning in Low Resource Modalities via Meta-Alignment

Abstract: How can we generalize to a new prediction task at test time when it also uses a new modality as input? More importantly, how can we do this with as little annotated data as possible? This problem of cross-modal generalization is a new research milestone with concrete impact on real-world applications. For example, can an AI system start understanding spoken language from mostly written text? Or can it learn the visual steps of a new recipe from only text descriptions? In this work, we formalize cross-modal generalization as a learning paradigm to train a model that can (1) quickly perform new tasks (from new domains) while (2) being originally trained on a different input modality. Such a learning paradigm is crucial for generalization to low-resource modalities such as spoken speech in rare languages while utilizing a different high-resource modality such as text. One key technical challenge that makes it different from other learning paradigms such as meta-learning and domain adaptation is the presence of different source and target modalities which will require different encoders. We propose an effective solution based on meta-alignment, a novel method to align representation spaces using strongly and weakly paired cross-modal data while ensuring quick generalization to new tasks across different modalities. This approach uses key ideas from cross-modal learning and meta-learning, and presents strong results on the cross-modal generalization problem. We benchmark several approaches on 3 real-world classification tasks: few-shot recipe classification from text to images of recipes, object classification from images to audio of objects, and language classification from text to spoken speech across 100 languages spanning many rare languages. Our results demonstrate strong performance even when the new target modality has only a few (1-10) labeled samples and in the presence of noisy labels, a scenario particularly prevalent in low-resource modalities.

Using Sensory Time-cue to enable Unsupervised Multimodal Meta-learning

Abstract: As data from IoT (Internet of Things) sensors become ubiquitous, state-of-the-art machine learning algorithms face many challenges on directly using sensor data. To overcome these challenges, methods must be designed to learn directly from sensors without manual annotations. This paper introduces Sensory Time-cue for Unsupervised Meta-learning (STUM). Different from traditional learning approaches that either heavily depend on labels or on time-independent feature extraction assumptions, such as Gaussian distribution features, the STUM system uses time relation of inputs to guide the feature space formation within and across modalities. The fact that STUM learns from a variety of small tasks may put this method in the camp of Meta-Learning. Different from existing Meta-Learning approaches, STUM learning tasks are composed within and across multiple modalities based on time-cue co-exist with the IoT streaming data. In an audiovisual learning example, because consecutive visual frames usually comprise the same object, this approach provides a unique way to organize features from the same object together. The same method can also organize visual object features with the object's spoken-name features together if the spoken name is presented with the object at about the same time. This cross-modality feature organization may further help the organization of visual features that belong to similar objects but acquired at different location and time. Promising results are achieved through evaluations.

Direct multimodal few-shot learning of speech and images

Abstract: We propose direct multimodal few-shot models that learn a shared embedding space of spoken words and images from only a few paired examples. Imagine an agent is shown an image along with a spoken word describing the object in the picture, e.g. pen, book and eraser. After observing a few paired examples of each class, the model is asked to identify the "book" in a set of unseen pictures. Previous work used a two-step indirect approach relying on learned unimodal representations: speech-speech and image-image comparisons are performed across the support set of given speech-image pairs. We propose two direct models which instead learn a single multimodal space where inputs from different modalities are directly comparable: a multimodal triplet network (MTriplet) and a multimodal correspondence autoencoder (MCAE). To train these direct models, we mine speech-image pairs: the support set is used to pair up unlabelled in-domain speech and images. In a speech-to-image digit matching task, direct models outperform indirect models, with the MTriplet achieving the best multimodal five-shot accuracy. We show that the improvements are due to the combination of unsupervised and transfer learning in the direct models, and the absence of two-step compounding errors.

Multimodal Prototypical Networks for Few-shot Learning

Abstract: Although providing exceptional results for many computer vision tasks, state-of-the-art deep learning algorithms catastrophically struggle in low data scenarios. However, if data in additional modalities exist (e.g. text) this can compensate for the lack of data and improve the classification results. To overcome this data scarcity, we design a cross-modal feature generation framework capable of enriching the low populated embedding space in few-shot scenarios, leveraging data from the auxiliary modality. Specifically, we train a generative model that maps text data into the visual feature space to obtain more reliable prototypes. This allows to exploit data from additional modalities (e.g. text) during training while the ultimate task at test time remains classification with exclusively visual data. We show that in such cases nearest neighbor classification is a viable approach and outperform state-of-the-art single-modal and multimodal few-shot learning methods on the CUB-200 and Oxford-102 datasets.

Adaptive Cross-Modal Few-Shot Learning

Abstract: Metric-based meta-learning techniques have successfully been applied to few-shot classification problems. In this paper, we propose to leverage cross-modal information to enhance metric-based few-shot learning methods. Visual and semantic feature spaces have different structures by definition. For certain concepts, visual features might be richer and more discriminative than text ones. While for others, the inverse might be true. Moreover, when the support from visual information is limited in image classification, semantic representations (learned from unsupervised text corpora) can provide strong prior knowledge and context to help learning. Based on these two intuitions, we propose a mechanism that can adaptively combine information from both modalities according to new image categories to be learned. Through a series of experiments, we show that by this adaptive combination of the two modalities, our model outperforms current uni-modality few-shot learning methods and modality-alignment methods by a large margin on all benchmarks and few-shot scenarios tested. Experiments also show that our model can effectively adjust its focus on the two modalities. The improvement in performance is particularly large when the number of shots is very small.

Dizygotic Conditional Variational AutoEncoder for Multi-Modal and Partial Modality Absent Few-Shot Learning

Abstract: Data augmentation is a powerful technique for improving the performance of the few-shot classification task. It generates more samples as supplements, and then this task can be transformed into a common supervised learning issue for solution. However, most mainstream data augmentation based approaches only consider the single modality information, which leads to the low diversity and quality of generated features. In this paper, we present a novel multi-modal data augmentation approach named Dizygotic Conditional Variational AutoEncoder (DCVAE) for addressing the aforementioned issue. DCVAE conducts feature synthesis via pairing two Conditional Variational AutoEncoders (CVAEs) with the same seed but different modality conditions in a dizygotic symbiosis manner. Subsequently, the generated features of two CVAEs are adaptively combined to yield the final feature, which can be converted back into its paired conditions while ensuring these conditions are consistent with the original conditions not only in representation but also in function. DCVAE essentially provides a new idea of data augmentation in various multi-modal scenarios by exploiting the complement of different modality prior information. Extensive experimental results demonstrate our work achieves state-of-the-art performances on miniImageNet, CIFAR-FS and CUB datasets, and is able to work well in the partial modality absence case.

Shaping Visual Representations with Language for Few-Shot Classification

Abstract: By describing the features and abstractions of our world, language is a crucial tool for human learning and a promising source of supervision for machine learning models. We use language to improve few-shot visual classification in the underexplored scenario where natural language task descriptions are available during training, but unavailable for novel tasks at test time. Existing models for this setting sample new descriptions at test time and use those to classify images. Instead, we propose language-shaped learning (LSL), an end-to-end model that regularizes visual representations to predict language. LSL is conceptually simpler, more data efficient, and outperforms baselines in two challenging few-shot domains.

Improving One-Shot Learning through Fusing Side Information

Abstract: Deep Neural Networks (DNNs) often struggle with one-shot learning where we have only one or a few labeled training examples per category. In this paper, we argue that by using side information, we may compensate the missing information across classes. We introduce two statistical approaches for fusing side information into data representation learning to improve one-shot learning. First, we propose to enforce the statistical dependency between data representations and multiple types of side information. Second, we introduce an attention mechanism to efficiently treat examples belonging to the 'lots-of-examples' classes as quasi-samples (additional training samples) for 'one-example' classes. We empirically show that our learning architecture improves over traditional softmax regression networks as well as state-of-the-art attentional regression networks on one-shot recognition tasks.

Learning to Compare: Relation Network for Few-Shot Learning

Abstract: We present a conceptually simple, flexible, and general framework for few-shot learning, where a classifier must learn to recognise new classes given only few examples from each. Our method, called the Relation Network (RN), is trained end-to-end from scratch. During meta-learning, it learns to learn a deep distance metric to compare a small number of images within episodes, each of which is designed to simulate the few-shot setting. Once trained, a RN is able to classify images of new classes by computing relation scores between query images and the few examples of each new class without further updating the network. Besides providing improved performance on few-shot learning, our framework is easily extended to zero-shot learning. Extensive experiments on five benchmarks demonstrate that our simple approach provides a unified and effective approach for both of these two tasks.

Cross-modal Hallucination for Few-shot Fine-grained Recognition

Abstract: State-of-the-art deep learning algorithms generally require large amounts of data for model training. Lack thereof can severely deteriorate the performance, particularly in scenarios with fine-grained boundaries between categories. To this end, we propose a multimodal approach that facilitates bridging the information gap by means of meaningful joint embeddings. Specifically, we present a benchmark that is multimodal during training (i.e. images and texts) and single-modal in testing time (i.e. images), with the associated task to utilize multimodal data in base classes (with many samples), to learn explicit visual classifiers for novel classes (with few samples). Next, we propose a framework built upon the idea of cross-modal data hallucination. In this regard, we introduce a discriminative text-conditional GAN for sample generation with a simple self-paced strategy for sample selection. We show the results of our proposed discriminative hallucinated method for 1-, 2-, and 5- shot learning on the CUB dataset, where the accuracy is improved by employing multimodal data.

Overview of papers using meta-learning methods for various applications with multimodal datasets.

In this section, we provide a summary of recent works about multimodality-based meta-learning methods. The attributes of each method are concluded in Table {{table:e33aec7a-4951-4ddb-8fa2-c95e789d476c}} . Specifically, we highlight the following attributes and discuss some common applications.

The lack of multimodal training datasets with few samples is often regarded as an obstacle to the application of multimodality-based meta-learning algorithms. However, this notion is only partially correct. It can be seen from Table {{table:e33aec7a-4951-4ddb-8fa2-c95e789d476c}}  that the application fields are extensive, and the datasets are not limited to image classification. As the applications of meta-learning in real environments are introduced with more public datasets available, the flexibility of the optimization-based framework is expected to play a more important role in adapting to new changes of modality patterns.

From the number of article distributions (See Table {{table:e33aec7a-4951-4ddb-8fa2-c95e789d476c}} ), we have seen that the metric-based meta-learning algorithms have become the preferred approach for interpreting multimodal data. In fact, the metric-based algorithms provide exactly what can help multiple modalities map to the same embedding space, thus implicitly solving the problem originated from traditional multimodal machine learning and providing convenience to compare distances using the metric. Specifically, after reviewing relevant papers using the prototypical networks to capture the changes brought by multimodal prototypes, one would expect to be able to summarize the perfect prototype fusion method. Although different methods have proved that their performance exceeds that of unimodal prototype applications, a striking conclusion we can draw is that there is no exact solution to uniformly solve the problem of modifying multimodal prototypes in different applications. We have seen that, for example, in the adaptive modality mixture mechanism (AM3), many researchers {{cite:d58fbd3}}, {{cite:61ffdbe}}, {{cite:8ed6172}} use exactly the same prototypical networks and modality types, but the results are very different. A key aspect that is often overlooked is that the discriminative features that different modalities can provide are very different in various scenarios. Moreover, it also depends on the dataset itself and we cannot decide which modality is more important in a simple procedure. The benefits that multimodal prototypes can provide are not only to expand the possibility of more modality combinations in the embedding space, but may also be affected by the previous encoding models of multimodal data. However, these factors hardly appear in the current works.