Emotion recognition, dialogue act recog.

Turn-taking prediction: barge-in detection

Improving Long Distance Slot Carryover in Spoken Dialogue Systems

Abstract: Tracking the state of the conversation is a central component in task-oriented spoken dialogue systems. One such approach for tracking the dialogue state is slot carryover, where a model makes a binary decision if a slot from the context is relevant to the current turn. Previous work on the slot carryover task used models that made independent decisions for each slot. A close analysis of the results show that this approach results in poor performance over longer context dialogues. In this paper, we propose to jointly model the slots. We propose two neural network architectures, one based on pointer networks that incorporate slot ordering information, and the other based on transformer networks that uses self attention mechanism to model the slot interdependencies. Our experiments on an internal dialogue benchmark dataset and on the public DSTC2 dataset demonstrate that our proposed models are able to resolve longer distance slot references and are able to achieve competitive performance.

TurnGPT: a Transformer-based Language Model for Predicting Turn-taking in Spoken Dialog

Abstract: Syntactic and pragmatic completeness is known to be important for turn-taking prediction, but so far machine learning models of turn-taking have used such linguistic information in a limited way. In this paper, we introduce TurnGPT, a transformer-based language model for predicting turn-shifts in spoken dialog. The model has been trained and evaluated on a variety of written and spoken dialog datasets. We show that the model outperforms two baselines used in prior work. We also report on an ablation study, as well as attention and gradient analyses, which show that the model is able to utilize the dialog context and pragmatic completeness for turn-taking prediction. Finally, we explore the model's potential in not only detecting, but also projecting, turn-completions.

Are Neural Open-Domain Dialog Systems Robust to Speech Recognition Errors in the Dialog History? An Empirical Study

Abstract: Large end-to-end neural open-domain chatbots are becoming increasingly popular. However, research on building such chatbots has typically assumed that the user input is written in nature and it is not clear whether these chatbots would seamlessly integrate with automatic speech recognition (ASR) models to serve the speech modality. We aim to bring attention to this important question by empirically studying the effects of various types of synthetic and actual ASR hypotheses in the dialog history on TransferTransfo, a state-of-the-art Generative Pre-trained Transformer (GPT) based neural open-domain dialog system from the NeurIPS ConvAI2 challenge. We observe that TransferTransfo trained on written data is very sensitive to such hypotheses introduced to the dialog history during inference time. As a baseline mitigation strategy, we introduce synthetic ASR hypotheses to the dialog history during training and observe marginal improvements, demonstrating the need for further research into techniques to make end-to-end open-domain chatbots fully speech-robust. To the best of our knowledge, this is the first study to evaluate the effects of synthetic and actual ASR hypotheses on a state-of-the-art neural open-domain dialog system and we hope it promotes speech-robustness as an evaluation criterion in open-domain dialog.

Controllable Time-Delay Transformer for Real-Time Punctuation Prediction and Disfluency Detection

Abstract: With the increased applications of automatic speech recognition (ASR) in recent years, it is essential to automatically insert punctuation marks and remove disfluencies in transcripts, to improve the readability of the transcripts as well as the performance of subsequent applications, such as machine translation, dialogue systems, and so forth. In this paper, we propose a Controllable Time-delay Transformer (CT-Transformer) model that jointly completes the punctuation prediction and disfluency detection tasks in real time. The CT-Transformer model facilitates freezing partial outputs with controllable time delay to fulfill the real-time constraints in partial decoding required by subsequent applications. We further propose a fast decoding strategy to minimize latency while maintaining competitive performance. Experimental results on the IWSLT2011 benchmark dataset and an in-house Chinese annotated dataset demonstrate that the proposed approach outperforms the previous state-of-the-art models on F-scores and achieves a competitive inference speed.

A Comparative Study on Language Models for Task-Oriented Dialogue Systems

Abstract: The recent development of language models has shown promising results by achieving state-of-the-art performance on various natural language tasks by fine-tuning pretrained models. In task-oriented dialogue (ToD) systems, language models can be used for end-to-end training without relying on dialogue state tracking to track the dialogue history but allowing the language models to generate responses according to the context given as input. This paper conducts a comparative study to show the effectiveness and strength of using recent pretrained models for fine-tuning, such as BART and T5, on endto-end ToD systems. The experimental results show substantial performance improvements after language model fine-tuning. The models produce more fluent responses after adding knowledge to the context that guides the model to avoid hallucination and generate accurate entities in the generated responses. Furthermore, we found that BART and T5 outperform GPT-based models in BLEU and F1 scores and achieve state-of-the-art performance in a ToD system.

Hierarchical Pre-training for Sequence Labelling in Spoken Dialog

Abstract: Sequence labelling tasks like Dialog Act and Emotion/Sentiment identification are a key component of spoken dialog systems. In this work, we propose a new approach to learn generic representations adapted to spoken dialog, which we evaluate on a new benchmark we call Sequence labellIng evaLuatIon benChmark fOr spoken laNguagE benchmark (\texttt{SILICONE}). \texttt{SILICONE} is model-agnostic and contains 10 different datasets of various sizes. We obtain our representations with a hierarchical encoder based on transformer architectures, for which we extend two well-known pre-training objectives. Pre-training is performed on OpenSubtitles: a large corpus of spoken dialog containing over $2.3$ billion of tokens. We demonstrate how hierarchical encoders achieve competitive results with consistently fewer parameters compared to state-of-the-art models and we show their importance for both pre-training and fine-tuning.

A Unified Transformer-based Framework for Duplex Text Normalization

Abstract: Text normalization (TN) and inverse text normalization (ITN) are essential preprocessing and postprocessing steps for text-to-speech synthesis and automatic speech recognition, respectively. Many methods have been proposed for either TN or ITN, ranging from weighted finite-state transducers to neural networks. Despite their impressive performance, these methods aim to tackle only one of the two tasks but not both. As a result, in a complete spoken dialog system, two separate models for TN and ITN need to be built. This heterogeneity increases the technical complexity of the system, which in turn increases the cost of maintenance in a production setting. Motivated by this observation, we propose a unified framework for building a single neural duplex system that can simultaneously handle TN and ITN. Combined with a simple but effective data augmentation method, our systems achieve state-of-the-art results on the Google TN dataset for English and Russian. They can also reach over 95% sentence-level accuracy on an internal English TN dataset without any additional fine-tuning. In addition, we also create a cleaned dataset from the Spoken Wikipedia Corpora for German and report the performance of our systems on the dataset. Overall, experimental results demonstrate the proposed duplex text normalization framework is highly effective and applicable to a range of domains and languages

Two-stage Textual Knowledge Distillation for End-to-End Spoken Language Understanding

Abstract: End-to-end approaches open a new way for more accurate and efficient spoken language understanding (SLU) systems by alleviating the drawbacks of traditional pipeline systems. Previous works exploit textual information for an SLU model via pre-training with automatic speech recognition or fine-tuning with knowledge distillation. To utilize textual information more effectively, this work proposes a two-stage textual knowledge distillation method that matches utterance-level representations and predicted logits of two modalities during pre-training and fine-tuning, sequentially. We use vq-wav2vec BERT as a speech encoder because it captures general and rich features. Furthermore, we improve the performance, especially in a low-resource scenario, with data augmentation methods by randomly masking spans of discrete audio tokens and contextualized hidden representations. Consequently, we push the state-of-the-art on the Fluent Speech Commands, achieving 99.7% test accuracy in the full dataset setting and 99.5% in the 10% subset setting. Throughout the ablation studies, we empirically verify that all used methods are crucial to the final performance, providing the best practice for spoken language understanding. Code is available at https://github.com/clovaai/textual-kd-slu.

End-to-End Evaluation of a Spoken Dialogue System for Learning Basic Mathematics

Abstract: The advances in language-based Artificial Intelligence (AI) technologies applied to build educational applications can present AI for social-good opportunities with a broader positive impact. Across many disciplines, enhancing the quality of mathematics education is crucial in building critical thinking and problem-solving skills at younger ages. Conversational AI systems have started maturing to a point where they could play a significant role in helping students learn fundamental math concepts. This work presents a task-oriented Spoken Dialogue System (SDS) built to support play-based learning of basic math concepts for early childhood education. The system has been evaluated via real-world deployments at school while the students are practicing early math concepts with multimodal interactions. We discuss our efforts to improve the SDS pipeline built for math learning, for which we explore utilizing MathBERT representations for potential enhancement to the Natural Language Understanding (NLU) module. We perform an end-to-end evaluation using real-world deployment outputs from the Automatic Speech Recognition (ASR), Intent Recognition, and Dialogue Manager (DM) components to understand how error propagation affects the overall performance in real-world scenarios.

Duplex Conversation: Towards Human-like Interaction in Spoken Dialogue Systems

Abstract: In this paper, we present Duplex Conversation, a multi-turn, multimodal spoken dialogue system that enables telephone-based agents to interact with customers like a human. We use the concept of full-duplex in telecommunication to demonstrate what a human-like interactive experience should be and how to achieve smooth turn-taking through three subtasks: user state detection, backchannel selection, and barge-in detection. Besides, we propose semi-supervised learning with multimodal data augmentation to leverage unlabeled data to increase model generalization. Experimental results on three sub-tasks show that the proposed method achieves consistent improvements compared with baselines. We deploy the Duplex Conversation to Alibaba intelligent customer service and share lessons learned in production. Online A/B experiments show that the proposed system can significantly reduce response latency by 50%.

Device Directedness with Contextual Cues for Spoken Dialog Systems

Abstract: In this work, we define barge-in verification as a supervised learning task where audio-only information is used to classify user spoken dialogue into true and false barge-ins. Following the success of pre-trained models, we use low-level speech representations from a self-supervised representation learning model for our downstream classification task. Further, we propose a novel technique to infuse lexical information directly into speech representations to improve the domain-specific language information implicitly learned during pre-training. Experiments conducted on spoken dialog data show that our proposed model trained to validate barge-in entirely from speech representations is faster by 38% relative and achieves 4.5% relative F1 score improvement over a baseline LSTM model that uses both audio and Automatic Speech Recognition (ASR) 1-best hypotheses. On top of this, our best proposed model with lexically infused representations along with contextual features provides a further relative improvement of 5.7% in the F1 score but only 22% faster than the baseline.

A Transformer-Based User Satisfaction Prediction for Proactive Interaction Mechanism in DuerOS

Abstract: Recently, spoken dialogue systems have been widely deployed in a variety of applications, serving a huge number of end-users. A common issue is that the errors resulting from noisy utterances, semantic misunderstandings, or lack of knowledge make it hard for a real system to respond properly, possibly leading to an unsatisfactory user experience. To avoid such a case, we consider a proactive interaction mechanism where the system predicts the user satisfaction with the candidate response before giving it to the user. If the user is not likely to be satisfied according to the prediction, the system will ask the user a suitable question to determine the real intent of the user instead of providing the response directly. With such an interaction with the user, the system can give a better response to the user. Previous models that predict the user satisfaction are not applicable to DuerOS which is a large-scale commercial dialogue system. They are based on hand-crafted features and thus can hardly learn the complex patterns lying behind millions of conversations and temporal dependency in multiple turns of the conversation. Moreover, they are trained and evaluated on the benchmark datasets with adequate labels, which are expensive to obtain in a commercial dialogue system. To face these challenges, we propose a pipeline to predict the user satisfaction to help DuerOS decide whether to ask for clarification in each turn. Specifically, we propose to first generate a large number of weak labels and then train a transformer-based model to predict the user satisfaction with these weak labels. Empirically, we deploy and evaluate our model on DuerOS, and observe a 19% relative improvement on the accuracy of user satisfaction prediction and 2.3% relative improvement on user experience.

Gated Multimodal Fusion with Contrastive Learning for Turn-taking Prediction in Human-robot Dialogue

Abstract: Turn-taking, aiming to decide when the next speaker can start talking, is an essential component in building human-robot spoken dialogue systems. Previous studies indicate that multimodal cues can facilitate this challenging task. However, due to the paucity of public multimodal datasets, current methods are mostly limited to either utilizing unimodal features or simplistic multimodal ensemble models. Besides, the inherent class imbalance in real scenario, e.g. sentence ending with short pause will be mostly regarded as the end of turn, also poses great challenge to the turn-taking decision. In this paper, we first collect a large-scale annotated corpus for turn-taking with over 5,000 real human-robot dialogues in speech and text modalities. Then, a novel gated multimodal fusion mechanism is devised to utilize various information seamlessly for turn-taking prediction. More importantly, to tackle the data imbalance issue, we design a simple yet effective data augmentation method to construct negative instances without supervision and apply contrastive learning to obtain better feature representations. Extensive experiments are conducted and the results demonstrate the superiority and competitiveness of our model over several state-of-the-art baselines.

Towards End-to-End Integration of Dialog History for Improved Spoken Language Understanding

Abstract: Dialog history plays an important role in spoken language understanding (SLU) performance in a dialog system. For end-to-end (E2E) SLU, previous work has used dialog history in text form, which makes the model dependent on a cascaded automatic speech recognizer (ASR). This rescinds the benefits of an E2E system which is intended to be compact and robust to ASR errors. In this paper, we propose a hierarchical conversation model that is capable of directly using dialog history in speech form, making it fully E2E. We also distill semantic knowledge from the available gold conversation transcripts by jointly training a similar text-based conversation model with an explicit tying of acoustic and semantic embeddings. We also propose a novel technique that we call DropFrame to deal with the long training time incurred by adding dialog history in an E2E manner. On the HarperValleyBank dialog dataset, our E2E history integration outperforms a history independent baseline by 7.7% absolute F1 score on the task of dialog action recognition. Our model performs competitively with the state-of-the-art history based cascaded baseline, but uses 48% fewer parameters. In the absence of gold transcripts to fine-tune an ASR model, our model outperforms this baseline by a significant margin of 10% absolute F1 score.

Representative list of publications in Transformer-based Spoken Dialogue Systems in the period 2019-2023.

Table {{table:cf85aefe-8ec6-471d-b2b0-984e08768741}}  shows a list of related works on spoken dialogue systems using Transformer networks. But it should be noted that most of those neural architectures have been originally applied to text-based language processing tasks, not to speech data with some exceptions {{cite:7fc1eb3}}, {{cite:4595b61}}. From the popularity chart in Fig. {{figure:3bd7daca-0409-4148-bd76-43d4ea612574}} , it can be noted that the most popular neural architecture is BERT (Bidirectional Encoder Transformer) {{cite:f9be9f2}} and the second most popular choice of architecture is either the original/vanilla transformer architecture {{cite:790bb4b}} or GPT-2 (Generative Pre-Trained Transformer) {{cite:4ceaac3}}. Whilst BERT and GPT-2 are generalizations of the vanilla transformer networks, BERT uses encoder blocks (no decoder blocks) and bidirectional representations whilst GPT-2 uses decoder blocks (no encoder blocks) and left-to-right representations. Other generalizations of Transformers include the following: XLM (Cross-lingual Language Model) {{cite:df3825e}} to benefit from data in different languages; DistillBERT (Distilled version of BERT) {{cite:a805ecc}} to train more compact models via knowledge distillation; ConvERT {{cite:e334295}} to perform faster training via pre-trained response selection; BART (Bidirectional Auto-Regressive Transformers) {{cite:b59342b}} to pre-train sequence-to-sequence models via a denoising autoencoder (by predicting outputs without noise from noisy inputs); T5 (Text-to-Text Transfer Transformer) {{cite:6d6b64b}} to learn from data in multiple language tasks by converting it to text-to-text format and then carrying out transfer learning to a specific language task; Conformer (Convolution-augmented Transformer) {{cite:7fc1eb3}} to bring the advantages of Transformer and Convolutional neural nets into a single architecture suitable for audio sequences; DIET (Dual Intent and Entity Transformer) {{cite:4595b61}} to perform language understanding of intents and entities in utterances without pretraining; GPT-3 (large Generative Pre-Trained Transformer) {{cite:8af966a}} to learn large language models without the need of fine-tuning; and RoBERTa (Robustly optimised BERT approach) {{cite:6f4a4d4}} to learn language models with an improved methodology over BERT. Some other related architectures have been proposed and trained with text instead of speech data {{cite:b84eebb}}, {{cite:b64bc47}}, {{cite:e756cd5}}, {{cite:9d7f57c}}, {{cite:66bbc11}}, {{cite:2e1a14b}}, {{cite:de8f9fb}}, {{cite:86221dd}}, {{cite:1f85983}}, {{cite:99c7498}}, {{cite:b2896d4}}, {{cite:f5876cc}}, {{cite:ae3196f}}, {{cite:34a4356}}, {{cite:8fdcec0}}, {{cite:1cd7b35}}.

One significant challenge faced by transformers-based speech models is the requirement for a large amount of data for effective training.
While recent text-based conversational AI models have been trained on large amounts of data (e.g., GODEL, which used around 800GB of data, and GPT-3, which was pretrained on 570GB of data), the amount of speech data available for training models for spoken technology is more limited. For instance, the works in Table {{table:cf85aefe-8ec6-471d-b2b0-984e08768741}}  (with some exceptions) use datasets consisting of several thousand spoken dialogues for various tasks, which is far less compared to the hundreds of gigabytes of text data used by text-based models. This highlights the need to either develop more efficient ways of creating datasets to train speech-related systems more efficiently.


MedAug: Contrastive learning leveraging patient metadata improves representations for chest X-ray interpretation

Abstract: Self-supervised contrastive learning between pairs of multiple views of the same image has been shown to successfully leverage unlabeled data to produce meaningful visual representations for both natural and medical images. However, there has been limited work on determining how to select pairs for medical images, where availability of patient metadata can be leveraged to improve representations. In this work, we develop a method to select positive pairs coming from views of possibly different images through the use of patient metadata. We compare strategies for selecting positive pairs for chest X-ray interpretation including requiring them to be from the same patient, imaging study or laterality. We evaluate downstream task performance by fine-tuning the linear layer on 1% of the labeled dataset for pleural effusion classification. Our best performing positive pair selection strategy, which involves using images from the same patient from the same study across all lateralities, achieves a performance increase of 14.4% in mean AUC from the ImageNet pretrained baseline. Our controlled experiments show that the keys to improving downstream performance on disease classification are (1) using patient metadata to appropriately create positive pairs from different images with the same underlying pathologies, and (2) maximizing the number of different images used in query pairing. In addition, we explore leveraging patient metadata to select hard negative pairs for contrastive learning, but do not find improvement over baselines that do not use metadata. Our method is broadly applicable to medical image interpretation and allows flexibility for incorporating medical insights in choosing pairs for contrastive learning.

Big Self-Supervised Models Advance Medical Image Classification

Abstract: Self-supervised pretraining followed by supervised fine-tuning has seen success in image recognition, especially when labeled examples are scarce, but has received limited attention in medical image analysis. This paper studies the effectiveness of self-supervised learning as a pretraining strategy for medical image classification. We conduct experiments on two distinct tasks: dermatology skin condition classification from digital camera images and multi-label chest X-ray classification, and demonstrate that self-supervised learning on ImageNet, followed by additional self-supervised learning on unlabeled domain-specific medical images significantly improves the accuracy of medical image classifiers. We introduce a novel Multi-Instance Contrastive Learning (MICLe) method that uses multiple images of the underlying pathology per patient case, when available, to construct more informative positive pairs for self-supervised learning. Combining our contributions, we achieve an improvement of 6.7% in top-1 accuracy and an improvement of 1.1% in mean AUC on dermatology and chest X-ray classification respectively, outperforming strong supervised baselines pretrained on ImageNet. In addition, we show that big self-supervised models are robust to distribution shift and can learn efficiently with a small number of labeled medical images.

Performance comparison on CheXpert dataset.

Classification tasks performance comparison

For the sake of performance comparison of the proposed methods from the previous section on the classification tasks, three datasets are selected in which two of them are public datasets which are Lung Nodule Analysis LUNA {{cite:295fc03}}, and CheXpert dataset {{cite:15ee410}}, while the third one is a private dataset which is Brain Hemorrhage classification dataset. For consistent comparison, the most common performance measure among all proposed approaches is reported. The area under curve (AUC) for receiver operating characteristic curve (ROC) is reported for both LUNA and CheXpert datasets, while overall accuracy is reported for the brain hemorrhage classification dataset. Further, all reported results are in terms of fine-tuning the whole model on the downstream task rather than the results of the fine-tuning the classification layer.
Table {{table:da936c1b-9adf-481b-95ab-7bf33ec6937c}}  reports the results on the LUNA dataset while Table {{table:b0c344c7-01d9-4124-954e-e413c7da178b}}  reports the results on the brain hemorrhage classification dataset, and Table {{table:e1aa8b8e-2801-48c7-9bfb-c77d5689b656}}  reports the results on the CheXpert dataset.

It can be clearly observed from tables {{table:da936c1b-9adf-481b-95ab-7bf33ec6937c}} , {{table:b0c344c7-01d9-4124-954e-e413c7da178b}}  and {{table:e1aa8b8e-2801-48c7-9bfb-c77d5689b656}}  that self-supervised learning approaches provide better performance when compared to either training from scratch or using ImageNet pre-trained models. In addition, designing new approaches or modifying existing methods in a way that considers the properties of medical images provides better performance than directly adopting pretext tasks from the computer vision field. Contrastive learning based algorithms tend to show better results when compared to other categories as shown in Table {{figure:3c9a8888-385f-4edc-bf01-b00361cb1102}} . However, this is not always the case where the TCPC approach showed the best and the worst results on the brain hemorrhage classification dataset when using different backbones as shown in Table {{table:b0c344c7-01d9-4124-954e-e413c7da178b}} . In Table {{table:e1aa8b8e-2801-48c7-9bfb-c77d5689b656}} , it can be clearly seen that directly adopting pretext task from computer vision to the medical images analysis field provides marginal improvements on the down-stream task as compared to the ImageNet pre-trained models as given by both {{cite:2ada0c2}} and {{cite:cdd9764}}. On the other side, modifying computer vision tasks by incorporating medical knowledge, as given by {{cite:d49369e}}, provided significant improvements on the performance when compared to ImageNet pre-trained models.


High precision grasp pose detection in dense clutter

Abstract: This paper considers the problem of grasp pose detection in point clouds. We follow a general algorithmic structure that first generates a large set of 6-DOF grasp candidates and then classifies each of them as a good or a bad grasp. Our focus in this paper is on improving the second step by using depth sensor scans from large online datasets to train a convolutional neural network. We propose two new representations of grasp candidates, and we quantify the effect of using prior knowledge of two forms: instance or category knowledge of the object to be grasped, and pretraining the network on simulated depth data obtained from idealized CAD models. Our analysis shows that a more informative grasp candidate representation as well as pretraining and prior knowledge significantly improve grasp detection. We evaluate our approach on a Baxter Research Robot and demonstrate an average grasp success rate of 93% in dense clutter. This is a 20% improvement compared to our prior work.

Grasp Pose Detection in Point Clouds

Abstract: Recently, a number of grasp detection methods have been proposed that can be used to localize robotic grasp configurations directly from sensor data without estimating object pose. The underlying idea is to treat grasp perception analogously to object detection in computer vision. These methods take as input a noisy and partially occluded RGBD image or point cloud and produce as output pose estimates of viable grasps, without assuming a known CAD model of the object. Although these methods generalize grasp knowledge to new objects well, they have not yet been demonstrated to be reliable enough for wide use. Many grasp detection methods achieve grasp success rates (grasp successes as a fraction of the total number of grasp attempts) between 75% and 95% for novel objects presented in isolation or in light clutter. Not only are these success rates too low for practical grasping applications, but the light clutter scenarios that are evaluated often do not reflect the realities of real world grasping. This paper proposes a number of innovations that together result in a significant improvement in grasp detection performance. The specific improvement in performance due to each of our contributions is quantitatively measured either in simulation or on robotic hardware. Ultimately, we report a series of robotic experiments that average a 93% end-to-end grasp success rate for novel objects presented in dense clutter.

6-DOF GraspNet: Variational Grasp Generation for Object Manipulation

Abstract: Generating grasp poses is a crucial component for any robot object manipulation task. In this work, we formulate the problem of grasp generation as sampling a set of grasps using a variational autoencoder and assess and refine the sampled grasps using a grasp evaluator model. Both Grasp Sampler and Grasp Refinement networks take 3D point clouds observed by a depth camera as input. We evaluate our approach in simulation and real-world robot experiments. Our approach achieves 88\% success rate on various commonly used objects with diverse appearances, scales, and weights. Our model is trained purely in simulation and works in the real world without any extra steps. The video of our experiments can be found at: https://research.nvidia.com/publication/2019-10_6-DOF-GraspNet\%3A-Variational

PointNetGPD: Detecting Grasp Configurations from Point Sets

Abstract: In this paper, we propose an end-to-end grasp evaluation model to address the challenging problem of localizing robot grasp configurations directly from the point cloud. Compared to recent grasp evaluation metrics that are based on handcrafted depth features and a convolutional neural network (CNN), our proposed PointNetGPD is lightweight and can directly process the 3D point cloud that locates within the gripper for grasp evaluation. Taking the raw point cloud as input, our proposed grasp evaluation network can capture the complex geometric structure of the contact area between the gripper and the object even if the point cloud is very sparse. To further improve our proposed model, we generate a large-scale grasp dataset with 350k real point cloud and grasps with the YCB object set for training. The performance of the proposed model is quantitatively measured both in simulation and on robotic hardware. Experiments on object grasping and clutter removal show that our proposed model generalizes well to novel objects and outperforms state-of-the-art methods. Code and video are available at https://lianghongzhuo.github.io/PointNetGPD.

Deep Dexterous Grasping of Novel Objects from a Single View

Abstract: In this thesis, a novel generative-evaluative method was proposed to solve the problem of dexterous grasping of the novel object with a single view. The generative model is learned from human demonstration. The grasps generated by the generative model are used to train the evaluative model. Two novel evaluative network architectures are proposed. The evaluative model is a deep evaluative network that is trained in the simulation. The generative-evaluative method is tested in a real grasp data set with 49 previously unseen challenging objects. The generative-evaluative method achieves a success rate of 78% that outperforms the purely generative method, that has a success rate of 57%. The thesis provides insights into the strengths and weaknesses of the generative-evaluative method by comparing different deep network architectures.

6-DOF Grasping for Target-driven Object Manipulation in Clutter

Abstract: Grasping in cluttered environments is a fundamental but challenging robotic skill. It requires both reasoning about unseen object parts and potential collisions with the manipulator. Most existing data-driven approaches avoid this problem by limiting themselves to top-down planar grasps which is insufficient for many real-world scenarios and greatly limits possible grasps. We present a method that plans 6-DOF grasps for any desired object in a cluttered scene from partial point cloud observations. Our method achieves a grasp success of 80.3%, outperforming baseline approaches by 17.6% and clearing 9 cluttered table scenes (which contain 23 unknown objects and 51 picks in total) on a real robotic platform. By using our learned collision checking module, we can even reason about effective grasp sequences to retrieve objects that are not immediately accessible. Supplementary video can be found at https://youtu.be/w0B5S-gCsJk.

Learning Continuous 3D Reconstructions for Geometrically Aware Grasping

Abstract: Deep learning has enabled remarkable improvements in grasp synthesis for previously unseen objects from partial object views. However, existing approaches lack the ability to explicitly reason about the full 3D geometry of the object when selecting a grasp, relying on indirect geometric reasoning derived when learning grasp success networks. This abandons explicit geometric reasoning, such as avoiding undesired robot object collisions. We propose to utilize a novel, learned 3D reconstruction to enable geometric awareness in a grasping system. We leverage the structure of the reconstruction network to learn a grasp success classifier which serves as the objective function for a continuous grasp optimization. We additionally explicitly constrain the optimization to avoid undesired contact, directly using the reconstruction. We examine the role of geometry in grasping both in the training of grasp metrics and through 96 robot grasping trials. Our results can be found on https://sites.google.com/view/reconstruction-grasp/.

Beyond Top-Grasps Through Scene Completion

Abstract: Current end-to-end grasp planning methods propose grasps in the order of seconds that attain high grasp success rates on a diverse set of objects, but often by constraining the workspace to top-grasps. In this work, we present a method that allows end-to-end top-grasp planning methods to generate full six-degree-of-freedom grasps using a single RGB-D view as input. This is achieved by estimating the complete shape of the object to be grasped, then simulating different viewpoints of the object, passing the simulated viewpoints to an end-to-end grasp generation method, and finally executing the overall best grasp. The method was experimentally validated on a Franka Emika Panda by comparing 429 grasps generated by the state-of-the-art Fully Convolutional Grasp Quality CNN, both on simulated and real camera images. The results show statistically significant improvements in terms of grasp success rate when using simulated images over real camera images, especially when the real camera viewpoint is angled. Code and video are available at https://irobotics.aalto.fi/beyond-top-grasps-through-scene-completion/.

Learning to Generate 6-DoF Grasp Poses with Reachability Awareness

Abstract: Motivated by the stringent requirements of unstructured real-world where a plethora of unknown objects reside in arbitrary locations of the surface, we propose a voxel-based deep 3D Convolutional Neural Network (3D CNN) that generates feasible 6-DoF grasp poses in unrestricted workspace with reachability awareness. Unlike the majority of works that predict if a proposed grasp pose within the restricted workspace will be successful solely based on grasp pose stability, our approach further learns a reachability predictor that evaluates if the grasp pose is reachable or not from robot's own experience. To avoid the laborious real training data collection, we exploit the power of simulation to train our networks on a large-scale synthetic dataset. This work is an early attempt that simultaneously evaluates grasping reachability from learned knowledge while proposing feasible grasp poses with 3D CNN. Experimental results in both simulation and real-world demonstrate that our approach outperforms several other methods and achieves 82.5% grasping success rate on unknown objects.

Same Object, Different Grasps: Data and Semantic Knowledge for Task-Oriented Grasping

Abstract: Despite the enormous progress and generalization in robotic grasping in recent years, existing methods have yet to scale and generalize task-oriented grasping to the same extent. This is largely due to the scale of the datasets both in terms of the number of objects and tasks studied. We address these concerns with the TaskGrasp dataset which is more diverse both in terms of objects and tasks, and an order of magnitude larger than previous datasets. The dataset contains 250K task-oriented grasps for 56 tasks and 191 objects along with their RGB-D information. We take advantage of this new breadth and diversity in the data and present the GCNGrasp framework which uses the semantic knowledge of objects and tasks encoded in a knowledge graph to generalize to new object instances, classes and even new tasks. Our framework shows a significant improvement of around 12% on held-out settings compared to baseline methods which do not use semantics. We demonstrate that our dataset and model are applicable for the real world by executing task-oriented grasps on a real robot on unknown objects. Code, data and supplementary video could be found at https://sites.google.com/view/taskgrasp

Learning Task-Oriented Grasping from Human Activity Datasets

Abstract: We propose to leverage a real-world, human activity RGB dataset to teach a robot Task-Oriented Grasping (TOG). We develop a model that takes as input an RGB image and outputs a hand pose and configuration as well as an object pose and a shape. We follow the insight that jointly estimating hand and object poses increases accuracy compared to estimating these quantities independently of each other. Given the trained model, we process an RGB dataset to automatically obtain the data to train a TOG model. This model takes as input an object point cloud and outputs a suitable region for task-specific grasping. Our ablation study shows that training an object pose predictor with the hand pose information (and vice versa) is better than training without this information. Furthermore, our results on a real-world dataset show the applicability and competitiveness of our method over state-of-the-art. Experiments with a robot demonstrate that our method can allow a robot to preform TOG on novel objects.

Multi-FinGAN: Generative Coarse-To-Fine Sampling of Multi-Finger Grasps

Abstract: While there exists many methods for manipulating rigid objects with parallel-jaw grippers, grasping with multi-finger robotic hands remains a quite unexplored research topic. Reasoning and planning collision-free trajectories on the additional degrees of freedom of several fingers represents an important challenge that, so far, involves computationally costly and slow processes. In this work, we present Multi-FinGAN, a fast generative multi-finger grasp sampling method that synthesizes high quality grasps directly from RGB-D images in about a second. We achieve this by training in an end-to-end fashion a coarse-to-fine model composed of a classification network that distinguishes grasp types according to a specific taxonomy and a refinement network that produces refined grasp poses and joint angles. We experimentally validate and benchmark our method against a standard grasp-sampling method on 790 grasps in simulation and 20 grasps on a real Franka Emika Panda. All experimental results using our method show consistent improvements both in terms of grasp quality metrics and grasp success rate. Remarkably, our approach is up to 20-30 times faster than the baseline, a significant improvement that opens the door to feedback-based grasp re-planning and task informative grasping. Code is available at https://irobotics.aalto.fi/multi-fingan/.

Collision-Aware Target-Driven Object Grasping in Constrained Environments

Abstract: Grasping a novel target object in constrained environments (e.g., walls, bins, and shelves) requires intensive reasoning about grasp pose reachability to avoid collisions with the surrounding structures. Typical 6-DoF robotic grasping systems rely on the prior knowledge about the environment and intensive planning computation, which is ungeneralizable and inefficient. In contrast, we propose a novel Collision-Aware Reachability Predictor (CARP) for 6-DoF grasping systems. The CARP learns to estimate the collision-free probabilities for grasp poses and significantly improves grasping in challenging environments. The deep neural networks in our approach are trained fully by self-supervision in simulation. The experiments in both simulation and the real world show that our approach achieves more than 75% grasping rate on novel objects in various surrounding structures. The ablation study demonstrates the effectiveness of the CARP, which improves the 6-DoF grasping rate by 95.7%.

DDGC: Generative Deep Dexterous Grasping in Clutter

Abstract: Recent advances in multi-fingered robotic grasping have enabled fast 6-Degrees-Of-Freedom (DOF) single object grasping. Multi-finger grasping in cluttered scenes, on the other hand, remains mostly unexplored due to the added difficulty of reasoning over obstacles which greatly increases the computational time to generate high-quality collision-free grasps. In this work we address such limitations by introducing DDGC, a fast generative multi-finger grasp sampling method that can generate high quality grasps in cluttered scenes from a single RGB-D image. DDGC is built as a network that encodes scene information to produce coarse-to-fine collision-free grasp poses and configurations. We experimentally benchmark DDGC against the simulated-annealing planner in GraspIt! on 1200 simulated cluttered scenes and 7 real world scenes. The results show that DDGC outperforms the baseline on synthesizing high-quality grasps and removing clutter while being 5 times faster. This, in turn, opens up the door for using multi-finger grasps in practical applications which has so far been limited due to the excessive computation time needed by other methods.

A self-supervised learning-based 6-DOF grasp planning method for manipulator

Abstract: To realize a robust robotic grasping system for unknown objects in an unstructured environment, large amounts of grasp data and 3D model data for the object are required, the sizes of which directly affect the rate of successful grasps. To reduce the time cost of data acquisition and labeling and increase the rate of successful grasps, we developed a self-supervised learning mechanism to control grasp tasks performed by manipulators. First, a manipulator automatically collects the point cloud for the objects from multiple perspectives to increase the efficiency of data acquisition. The complete point cloud for the objects is obtained by utilizing the hand-eye vision of the manipulator, and the TSDF algorithm. Then, the point cloud data for the objects is used to generate a series of six-degrees-of-freedom grasp poses, and the force-closure decision algorithm is used to add the grasp quality label to each grasp pose to realize the automatic labeling of grasp data. Finally, the point cloud in the gripper closing area corresponding to each grasp pose is obtained; it is then used to train the grasp-quality classification model for the manipulator. The results of data acquisition experiments demonstrate that the proposed method allows high-quality data to be obtained. The simulated results prove the effectiveness of the proposed grasp-data acquisition method. The results of performing actual grasping experiments demonstrate that the proposed self-supervised learning method can increase the rate of successful grasps for the manipulator.

Synergies Between Affordance and Geometry: 6-DoF Grasp Detection via Implicit Representations

Abstract: Grasp detection in clutter requires the robot to reason about the 3D scene from incomplete and noisy perception. In this work, we draw insight that 3D reconstruction and grasp learning are two intimately connected tasks, both of which require a fine-grained understanding of local geometry details. We thus propose to utilize the synergies between grasp affordance and 3D reconstruction through multi-task learning of a shared representation. Our model takes advantage of deep implicit functions, a continuous and memory-efficient representation, to enable differentiable training of both tasks. We train the model on self-supervised grasp trials data in simulation. Evaluation is conducted on a clutter removal task, where the robot clears cluttered objects by grasping them one at a time. The experimental results in simulation and on the real robot have demonstrated that the use of implicit neural representations and joint learning of grasp affordance and 3D reconstruction have led to state-of-the-art grasping results. Our method outperforms baselines by over 10% in terms of grasp success rate. Additional results and videos can be found at https://sites.google.com/view/rpl-giga2021

MVGrasp: Real-Time Multi-View 3D Object Grasping in Highly Cluttered Environments

Abstract: Nowadays robots play an increasingly important role in our daily life. In human-centered environments, robots often encounter piles of objects, packed items, or isolated objects. Therefore, a robot must be able to grasp and manipulate different objects in various situations to help humans with daily tasks. In this paper, we propose a multi-view deep learning approach to handle robust object grasping in human-centric domains. In particular, our approach takes a point cloud of an arbitrary object as an input, and then, generates orthographic views of the given object. The obtained views are finally used to estimate pixel-wise grasp synthesis for each object. We train the model end-to-end using a small object grasp dataset and test it on both simulations and real-world data without any further fine-tuning. To evaluate the performance of the proposed approach, we performed extensive sets of experiments in three scenarios, including isolated objects, packed items, and pile of objects. Experimental results show that our approach performed very well in all simulation and real-robot scenarios, and is able to achieve reliable closed-loop grasping of novel objects across various scene configurations.

Learning to Detect Multi-Modal Grasps for Dexterous Grasping in Dense Clutter

Abstract: We propose an approach to multi-modal grasp detection that jointly predicts the probabilities that several types of grasps succeed at a given grasp pose. Given a partial point cloud of a scene, the algorithm proposes a set of feasible grasp candidates, then estimates the probabilities that a grasp of each type would succeed at each candidate pose. Predicting grasp success probabilities directly from point clouds makes our approach agnostic to the number and placement of depth sensors at execution time. We evaluate our system both in simulation and on a real robot with a Robotiq 3-Finger Adaptive Gripper and compare our network against several baselines that perform fewer types of grasps. Our experiments show that a system that explicitly models grasp type achieves an object retrieval rate 8.5% higher in a complex cluttered environment than our highest-performing baseline.

CaTGrasp: Learning Category-Level Task-Relevant Grasping in Clutter from Simulation

Abstract: Task-relevant grasping is critical for industrial assembly, where downstream manipulation tasks constrain the set of valid grasps. Learning how to perform this task, however, is challenging, since task-relevant grasp labels are hard to define and annotate. There is also yet no consensus on proper representations for modeling or off-the-shelf tools for performing task-relevant grasps. This work proposes a framework to learn task-relevant grasping for industrial objects without the need of time-consuming real-world data collection or manual annotation. To achieve this, the entire framework is trained solely in simulation, including supervised training with synthetic label generation and self-supervised, hand-object interaction. In the context of this framework, this paper proposes a novel, object-centric canonical representation at the category level, which allows establishing dense correspondence across object instances and transferring task-relevant grasps to novel instances. Extensive experiments on task-relevant grasping of densely-cluttered industrial objects are conducted in both simulation and real-world setups, demonstrating the effectiveness of the proposed framework. Code and data are available at https://sites.google.com/view/catgrasp.

Publications with deep-learning focused sampling methods. We cluster the papers based on the space the sample through and how the samples are evaluated. Some approaches further consider an optional refinement stage.

Deep-learning approaches employing sampling implement the following steps: sample information; evaluate the sample according to a quality estimation function, parameterized by a deep neural network; and optionally refine the sample using an optimization-based approach to achieve a higher quality grasp. We adopt the term `quality', however, there is no current consensus on the definition. Throughout this paper, we use this term to refer to the confidence of grasp success. Table {{table:111cd96e-79c9-4a1a-8a50-cfba11a2ca4c}}  presents an overview of all the Sampling papers. A popular deep-learning based approach for 4-DOF grasping generates a series of antipodal grasps through sampling while using a neural network to predict grasp quality {{cite:344478d}}. This aims to predict the probability of a successful grasp to generate a series of antipodal grasps.


Scikit-learn: Machine Learning in Python

Abstract: Scikit-learn is a Python module integrating a wide range of state-of-the-art machine learning algorithms for medium-scale supervised and unsupervised problems. This package focuses on bringing machine learning to non-specialists using a general-purpose high-level language. Emphasis is put on ease of use, performance, documentation, and API consistency. It has minimal dependencies and is distributed under the simplified BSD license, encouraging its use in both academic and commercial settings. Source code, binaries, and documentation can be downloaded from http://scikit-learn.sourceforge.net.

Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms

Abstract: We present Fashion-MNIST, a new dataset comprising of 28x28 grayscale images of 70,000 fashion products from 10 categories, with 7,000 images per category. The training set has 60,000 images and the test set has 10,000 images. Fashion-MNIST is intended to serve as a direct drop-in replacement for the original MNIST dataset for benchmarking machine learning algorithms, as it shares the same image size, data format and the structure of training and testing splits. The dataset is freely available at this https URL

Datasets used across all of the analyses and visualizations in the paper. NA refers to a split that was not used in the paper (e.g. no validation or testing set was used for the bison visualization in Fig B.)

We performed experiments over several different datasets varying in complexity. The Cassin's vireo song dataset {{cite:e1de1b7}}, {{cite:c16a3ae}}, {{cite:9012875}} consists of spectrograms of 20 of the most frequently sung elements of Cassin's vireo song (zero-padded to 32 frequency by 31 time bins) produced by several individuals recorded in the Sierra Nevada mountains of California. Despite being recorded in the wild, the Cassin's vireo song is relatively low noise and vocal elements are highly stereotyped. MNIST is a benchmark handwritten digits dataset in 28x28 pixels {{cite:d54fff9}}. Fashion MNIST (FMNIST) is a dataset of fashion items in the same format as Fashion MNIST designed to be a more difficult classification problem than MNIST {{cite:d0699c8}}. CIFAR10 is a natural image dataset (32x32x3 pixels) with 10 classes {{cite:ba8da11}}. CIFAR10 classes are much less structured than the other datasets used. For example, unlike FMNIST, subjects of images are not uniformly centered in the image and can have different background conditions that make neighborhood in pixel-space less likely between members of the same class. The single-cell retina transcriptome dataset consists of PCA projections (50D) of single-cell RNA transcriptome data from mouse retina {{cite:d8ec8b4}}, {{cite:54b7a89}}. The CelebAMask-HQ dataset consists of cropped and aligned photographs of celebrity faces with 40 facial feature annotations {{cite:bb563fb}}, {{cite:52038e8}} and label masks corresponding to face-landmarks. We removed the background of each image to make the task of learning a structured embedding simpler for the neural network. A further description of each dataset is given in Table {{table:922d9133-f613-42d2-b191-b75c784e550f}} .


UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild

Abstract: We introduce UCF101 which is currently the largest dataset of human actions. It consists of 101 action classes, over 13k clips and 27 hours of video data. The database consists of realistic user uploaded videos containing camera motion and cluttered background. Additionally, we provide baseline action recognition results on this new dataset using standard bag of words approach with overall performance of 44.5%. To the best of our knowledge, UCF101 is currently the most challenging dataset of actions due to its large number of classes, large number of clips and also unconstrained nature of such clips.

The Kinetics Human Action Video Dataset

Abstract: We describe the DeepMind Kinetics human action video dataset. The dataset contains 400 human action classes, with at least 400 video clips for each action. Each clip lasts around 10s and is taken from a different YouTube video. The actions are human focussed and cover a broad range of classes including human-object interactions such as playing instruments, as well as human-human interactions such as shaking hands. We describe the statistics of the dataset, how it was collected, and give some baseline performance figures for neural network architectures trained and tested for human action classification on this dataset. We also carry out a preliminary analysis of whether imbalance in the dataset leads to bias in the classifiers.

Fine-Grained Activity Recognition in Baseball Videos

Abstract: In this paper, we introduce a challenging new dataset, MLB-YouTube, designed for fine-grained activity detection. The dataset contains two settings: segmented video classification as well as activity detection in continuous videos. We experimentally compare various recognition approaches capturing temporal structure in activity videos, by classifying segmented videos and extending those approaches to continuous videos. We also compare models on the extremely difficult task of predicting pitch speed and pitch type from broadcast baseball videos. We find that learning temporal structure is valuable for fine-grained activity recognition.

A Short Note on the Kinetics-700 Human Action Dataset

Abstract: We describe an extension of the DeepMind Kinetics human action dataset from 600 classes to 700 classes, where for each class there are at least 600 video clips from different YouTube videos. This paper details the changes introduced for this new release of the dataset, and includes a comprehensive set of statistics as well as baseline results using the I3D neural network architecture.

nblueDatasets used in the ZSAR experiments ordered by year of creation. The number of videos (#V) and the number of classes (#C) are also provided for each dataset.

The first popular video benchmarks were small, with approximately 10k videos {{cite:e768b16}}, as shown in Table {{table:213352e1-91d7-4811-bc8c-5b8fc5e50111}} . Larger and complex datasets are available since 2011, such as HMDB51 {{cite:8e6b8d6}}, UCF101 {{cite:2b1ae92}}, ActivityNet {{cite:c4cc7bf}} and, more recently, Kinetics {{cite:a18784e}}, {{cite:e768b16}}.

As shown in Table {{table:213352e1-91d7-4811-bc8c-5b8fc5e50111}} , there is a group of datasets used only once, such as UIUC {{cite:fa768fb}}, Thumos14 {{cite:733f819}}, and MLB-YouTube {{cite:c1590be}}.
The UIUC dataset is presented in {{cite:fa768fb}}. It consists of 532 high-resolution sequences of 14 activities performed by 8 actors in a single view. The Thumos14 dataset was proposed in the Thumos Challenge context {{cite:d0d7204}}. In this dataset, there are temporally untrimmed videos and background videos, that is, with a similar background but without actions in the scene. The 101 action classes are performed in realistic settings and distributed in 1574 video clips. MLB-YouTube {{cite:c1590be}} is a dataset with activities collected from broadcast baseball videos with a focus on fine-grained activity recognition. More precisely, it is composed of 20 baseball games (42 hours) from the 2017 MLB post-season available on YouTube. In this dataset, the structure of the scene is very similar among activities; often, the only difference is the motion of a single person. Additionally, there is a single camera viewpoint to determine the activity. Due to its objective, this dataset has limited potential in zsl. A complete description of most of datasets can be found in {{cite:201ac7e}}, {{cite:b70f8f0}}, {{cite:9a4bb58}}.

Experimental Protocols and Performance Analysis

There are many experimental protocols to perform zsar in videos. Consequently, it is not easy to compare them. We select works that use HMBD51, UCF101 or Olympic Sports datasets since they are the most popular, as shown in Table {{table:213352e1-91d7-4811-bc8c-5b8fc5e50111}} , which enables comparison among different approaches. The ActivityNet was not selected because we do not discriminate between its two versions (i.e., trimmed and untrimmed) in Table {{table:213352e1-91d7-4811-bc8c-5b8fc5e50111}} . nblueTable  reports the results of selected works, the proportions and amount of runs used in experiments and a comparison of performance in inductive and transductive settings. Additionally, we provide complementary information on how the visual and semantic embedding were performed to acquire that result and what was the classification approach.

From the perspective of visual extraction, with the recent advances in deep learning methods, its use seems to be imperative, especially pre-trained models, recurrent networks and generative models. However, a new problem emerges. For example, the c3d model is a pre-trained cnn using the Sports-1M Dataset {{cite:0670e8d}}. We believe that using pre-trained deep models in practice means intrinsically to use a cross-dataset approach and, if the same classes that are used to train the deep models were also used to test the zsl methods, the disjunction between seen and unseen classes would not be respected because the deep model acquires the knowledge from classes that should be unseen. A similar analysis was presented by {{cite:e3f22f4}}, but in the context of cross-dataset studies. They argued that when external datasets are involved, one has to ensure that the terms of zsl are still met and the seen and unseen categories are disjoint. It is not sufficient to remove only identical classes because there are similar classes, such as Basketball Shooting (UCF101) {{formula:e3bec21b-f9e2-40ef-9ee9-148c5dbeb6e5}}  Basketball or Basketball 3{{formula:3bb94c92-ffa3-4889-b588-873e526b00a4}} 3 or wheelchair basketball (Sports-1M). nblueA protocol to remove semantically similar classes from source category (seen) using the cosine similarity measure and a threshold parameter was defined by {{cite:e3f22f4}} and this analysis was extended by {{cite:ac0f1da}}. However, when pre-trained deep models are used, it is necessary to remove the similar classes from the target and not from the source. For example, in {{cite:4d7d418}}, we need to compare the classes between UCF101 and Sports-1M (used for training C3D model). It is observed that they share 23 identical classes and 17 similar classesManual checks.. Since that work uses the same 30 splits employed by {{cite:269a5bb}}, these shared classes were not removed from the target before the experiment, so the restriction of zsl is not preserved. To keep the zsl disjunction between the training and testing sets, it is necessary to use only unknown classes in the testing time, excluding all classes that were used for training the deep model. In this case, the UCF101 dataset would have 61 possible classes for testing. nblueThis approach has been implemented in the work developed by {{cite:76963ec}}. This new restriction means that it may be impracticable to use the UCF101 or HMDB51 dataset when pre-trained deep models, when c3d or i3d are used. nblueIn fact, as shown in Table {{table:213352e1-91d7-4811-bc8c-5b8fc5e50111}} , more recent datasets, such as Kinetics 600, Kinetics 700 or ARID {{cite:3c098d5}}, have not been explored in zsar.


Scene Graph Generation by Iterative Message Passing

Abstract: Understanding a visual scene goes beyond recognizing individual objects in isolation. Relationships between objects also constitute rich semantic information about the scene. In this work, we explicitly model the objects and their relationships using scene graphs, a visually-grounded graphical structure of an image. We propose a novel end-to-end model that generates such structured scene representation from an input image. Our key insight is that the graph generation problem can be formulated as message passing between the primal node graph and its dual edge graph. Our joint inference model can take advantage of contextual cues to make better predictions on objects and their relationships. The experiments show that our model significantly outperforms previous methods on the Visual Genome dataset as well as support relation inference in NYU Depth V2 dataset.

Visual Translation Embedding Network for Visual Relation Detection

Abstract: Visual relations, such as person ride bike and bike next to car, offer a comprehensive scene understanding of an image, and have already shown their great utility in connecting computer vision and natural language. However, due to the challenging combinatorial complexity of modeling subject-predicate-object relation triplets, very little work has been done to localize and predict visual relations. Inspired by the recent advances in relational representation learning of knowledge bases and convolutional object detection networks, we propose a Visual Translation Embedding network (VTransE) for visual relation detection. VTransE places objects in a low-dimensional relation space where a relation can be modeled as a simple vector translation, i.e., subject + predicate ≈ object. We propose a novel feature extraction layer that enables object-relation knowledge transfer in a fully-convolutional fashion that supports training and inference in a single forward/backward pass. To the best of our knowledge, VTransE is the first end-toend relation detection network. We demonstrate the effectiveness of VTransE over other state-of-the-art methods on two large-scale datasets: Visual Relationship and Visual Genome. Note that even though VTransE is a purely visual model, it is still competitive to the Lu’s multi-modal model with language priors [27].

Detecting Visual Relationships with Deep Relational Networks

Abstract: Relationships among objects play a crucial role in image understanding. Despite the great success of deep learning techniques in recognizing individual objects, reasoning about the relationships among objects remains a challenging task. Previous methods often treat this as a classification problem, considering each type of relationship (e.g. ride) or each distinct visual phrase (e.g. person-ride-horse) as a category. Such approaches are faced with significant difficulties caused by the high diversity of visual appearance for each kind of relationships or the large number of distinct visual phrases. We propose an integrated framework to tackle this problem. At the heart of this framework is the Deep Relational Network, a novel formulation designed specifically for exploiting the statistical dependencies between objects and their relationships. On two large data sets, the proposed method achieves substantial improvement over state-of-the-art.

Scene Graph Generation from Objects, Phrases and Region Captions

Abstract: Object detection, scene graph generation and region captioning, which are three scene understanding tasks at different semantic levels, are tied together: scene graphs are generated on top of objects detected in an image with their pairwise relationship predicted, while region captioning gives a language description of the objects, their attributes, relations and other context information. In this work, to leverage the mutual connections across semantic levels, we propose a novel neural network model, termed as Multi-level Scene Description Network (denoted as MSDN), to solve the three vision tasks jointly in an end-to-end manner. Object, phrase, and caption regions are first aligned with a dynamic graph based on their spatial and semantic connections. Then a feature refining structure is used to pass messages across the three levels of semantic tasks through the graph. We benchmark the learned model on three tasks, and show the joint learning across three tasks with our proposed method can bring mutual improvements over previous models. Particularly, on the scene graph generation task, our proposed method outperforms the stateof- art method with more than 3% margin. Code has been made publicly available.

Large-Scale Visual Relationship Understanding

Abstract: Large scale visual understanding is challenging, as it requires a model to handle the widely-spread and imbalanced distribution of 〈subject, relation, object〉 triples. In real-world scenarios with large numbers of objects and relations, some are seen very commonly while others are barely seen. We develop a new relationship detection model that embeds objects and relations into two vector spaces where both discriminative capability and semantic affinity are preserved. We learn a visual and a semantic module that map features from the two modalities into a shared space, where matched pairs of features have to discriminate against those unmatched, but also maintain close distances to semantically similar ones. Benefiting from that, our model can achieve superior performance even when the visual entity categories scale up to more than 80,000, with extremely skewed class distribution. We demonstrate the efficacy of our model on a large and imbalanced benchmark based of Visual Genome that comprises 53,000+ objects and 29,000+ relations, a scale at which no previous work has been evaluated at. We show superiority of our model over competitive baselines on the original Visual Genome dataset with 80,000+ categories. We also show state-of-the-art performance on the VRD dataset and the scene graph dataset which is a subset of Visual Genome with 200 categories.

The statistics of common VG versions.

Visual Genome {{cite:09dc8bf}} has the maximum number of relation triplets with the most diverse object categories and relation labels up to now. Unlike VRD that is constructed by computer vision experts, VG is annotated by crowd workers and thus a substantial fraction of the object annotations have poor quality and overlapping bounding boxes and/or ambiguous object names. As an attempt to eliminate the noise, prior works have explored semi-automatic ways (e.g., class merging and filtering) to clean up object and relation annotations and constructed their own VG versions. Of these, VG200{{cite:2c8a075}}, VG150{{cite:7f5b792}}, VG-MSDN{{cite:7e26828}} and sVG{{cite:104fedd}} have released their cleansed annotations and are the most frequently used. Other works{{cite:b055236}}, {{cite:302a01f}}, {{cite:d112d7f}}, {{cite:460b405}}, {{cite:1855ee7}}, {{cite:5574682}}, {{cite:9c2419e}}, {{cite:9c4e505}}, {{cite:de5aa17}} use a paper-specific and nonpublicly available split, disabling direct future comparisons with their experiments. Moreover, {{cite:9fd712a}} presents experiments on a large-scale version of VG, named VG80K, and {{cite:c0adeac}} proposes a new split that has not been benchmarked yet. Sun{{cite:4365dd0}} constructed two datasets for hierarchical visual relationship detection (HVRD) based on VRD dataset and VG dataset, named H-VRD and H-VG, by expanding their flat relationship category spaces to hierarchical ones, respectively. The statistics of these datasets are summarized in Table  {{table:d53a06e6-c58e-4a9f-973f-03f91ce1628d}}.


UTM-Chain: Blockchain-Based Secure Unmanned Traffic Management for Internet of Drones

Abstract: Unmanned aerial systems (UAVs) are dramatically evolving and promoting several civil applications. However, they are still prone to many security issues that threaten public safety. Security becomes even more challenging when they are connected to the Internet as their data stream is exposed to attacks. Unmanned traffic management (UTM) represents one of the most important topics for small unmanned aerial systems for beyond-line-of-sight operations in controlled low-altitude airspace. However, without securing the flight path exchanges between drones and ground stations or control centers, serious security threats may lead to disastrous situations. For example, a predefined flight path could be easily altered to make the drone perform illegal operations. Motivated by these facts, this paper discusses the security issues for UTM’s components and addresses the security requirements for such systems. Moreover, we propose UTM-Chain, a lightweight blockchain-based security solution using hyperledger fabric for UTM of low-altitude UAVs which fits the computational and storage resources limitations of UAVs. Moreover, UTM-Chain provides secure and unalterable traffic data between the UAVs and their ground control stations. The performance of the proposed system related to transaction latency and resource utilization is analyzed by using cAdvisor. Finally, the analysis of security aspects demonstrates that the proposed UTM-Chain scheme is feasible and extensible for the secure sharing of UAV data.

Blockchain-based Privacy Preservation for 5G-enabled Drone Communications

Abstract: 5G-enabled drones have potential applications in a variety of both military and civilian settings (e.g., monitoring and tracking of individuals in demonstrations and/or enforcing of social / physical distancing during pandemics such as COVID-19). Such applications generally involve the collection and dissemination of (massive) data from the drones to remote data centres for storage and analysis, for example via 5G networks. Consequently, there are security and privacy considerations underpinning 5G-enabled drone communications. We posit the potential of leveraging blockchain to facilitate privacy preservation, and therefore in this article we will review existing blockchain-based solutions after introducing the architecture for 5G-enabled drone communications and blockchain. We will also review existing legislation and data privacy regulations that need to be considered in the design of blockchain-based solutions, as well as identifying potential challenges and open issues which will hopefully inform future research agenda.

A Drone-Aided Blockchain-Based Smart Vehicular Network

Abstract: The staggering growth of the number of vehicles worldwide has become a critical challenge resulting in tragic incidents, environment pollution, congestion, etc. Therefore, one of the promising approaches is to design a smart vehicular system as it is beneficial to drive safely. Present vehicular system lacks data reliability, security, and easy deployment. Motivated by these issues, this paper addresses a drone-enabled intelligent vehicular system, which is secure, easy to deploy and reliable in quality. Nevertheless, an increase in the number of operating drones in the communication networks makes them more vulnerable towards the cyber-attacks, which can completely sabotage the communication infrastructure. To tackle these problems, we propose a blockchain-based registration and authentication system for the entities such as drones, smart vehicles (SVs) and roadside units (RSUs). This paper is mainly focused on the blockchain-based secure system design and the optimal placement of drones to improve the spectral efficiency of the overall network. In particular, we investigate the association of RSUs with the drones by considering multiple communication-related factors such as available bandwidth, maximum number of links a drone can support, and backhaul limitations. We show that the proposed model can easily be overlaid on the current vehicular network reaping benefits of secure and reliable communications.

Design Guidelines for Blockchain-Assisted 5G-UAV Networks

Abstract: Fifth generation (5G) wireless networks are designed to meet various end-user quality of service (QoS) requirements through high data rates (typically of gigabits per second) and low latencies. Coupled with fog and mobile edge computing, 5G can achieve high data rates, enabling complex autonomous smart city services such as the large deployment of self-driving vehicles and large-scale artificial-intelligence-enabled industrial manufacturing. However, to meet the exponentially growing number of connected IoT devices and irregular data and service requests in both low- and high-density locations, the process of enacting traditional cells supported through fixed and costly base stations requires rethought to enable on-demand mobile access points in the form of unmanned aerial vehicles (UAV) for diversified smart city scenarios. This article envisions a 5G network environment that is supported by blockchain-enabled UAVs to meet dynamic user demands with network access supply. The solution enables decentralized service delivery (drones as a service) and routing to and from end users in a reliable and secure manner. Both public and private blockchains are deployed within the UAVs, supported by fog and cloud computing devices and data centers to provide a wide range of complex authenticated service and data availability. Particular attention is paid to comparing data delivery success rates and message exchange in the proposed solution against traditional UAV-supported cellular networks. Challenges and future research are also discussed with highlights on emerging technologies such as federated learning.

Latest proposals on blockchain-enabled UAV networks

State-of-the-art on blockchain-based UAV networks

The realm of blockchain technology has sufficiently matured to solve emerging issues related to UAV traffic networks. In this section, we briefly review the state-of-the-art in UAV networks that deploy different versions of blockchain to exploit multiple advantages such as immutability, transparency, traceability, and auditability. Unlike other application-oriented reviews {{cite:f622e8c}}, our review spins three axes based on the ultimate motivation of each work, namely: safety, security, and reliability. We dedicate one subsection for each category to highlight the recent major contributions (from 2018 to 2021). Our review hierarchy is illustrated in Figure {{figure:cdee13e0-a865-4dc6-8d9b-7546294004fc}} . Besides, the focus of the reviewed papers is summarized in Table {{table:827f5eda-9978-4cef-a238-a443f3583c02}} .


SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <1MB model size

Abstract:
Recent research on deep neural networks has focused primarily on improving accuracy. For a given accuracy level, it is typically possible to identify multiple DNN architectures that achieve that accuracy level. With equivalent accuracy, smaller DNN architectures offer at least three advantages: (1) Smaller DNNs require less communication across servers during distributed training. (2) Smaller DNNs require less bandwidth to export a new model from the cloud to an autonomous car. (3) Smaller DNNs are more feasible to deploy on FPGAs and other hardware with limited memory. To provide all of these advantages, we propose a small DNN architecture called SqueezeNet. SqueezeNet achieves AlexNet-level accuracy on ImageNet with 50x fewer parameters. Additionally, with model compression techniques we are able to compress SqueezeNet to less than 0.5MB (510x smaller than AlexNet).
The SqueezeNet architecture is available for download here: this https URL

MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications

Abstract: We present a class of efficient models called MobileNets for mobile and embedded vision applications. MobileNets are based on a streamlined architecture that uses depth-wise separable convolutions to build light weight deep neural networks. We introduce two simple global hyper-parameters that efficiently trade off between latency and accuracy. These hyper-parameters allow the model builder to choose the right sized model for their application based on the constraints of the problem. We present extensive experiments on resource and accuracy tradeoffs and show strong performance compared to other popular models on ImageNet classification. We then demonstrate the effectiveness of MobileNets across a wide range of applications and use cases including object detection, finegrain classification, face attributes and large scale geo-localization.

MobileNetV2: Inverted Residuals and Linear Bottlenecks

Abstract: In this paper we describe a new mobile architecture, MobileNetV2, that improves the state of the art performance of mobile models on multiple tasks and benchmarks as well as across a spectrum of different model sizes. We also describe efficient ways of applying these mobile models to object detection in a novel framework we call SSDLite. Additionally, we demonstrate how to build mobile semantic segmentation models through a reduced form of DeepLabv3 which we call Mobile DeepLabv3. is based on an inverted residual structure where the shortcut connections are between the thin bottleneck layers. The intermediate expansion layer uses lightweight depthwise convolutions to filter features as a source of non-linearity. Additionally, we find that it is important to remove non-linearities in the narrow layers in order to maintain representational power. We demonstrate that this improves performance and provide an intuition that led to this design. Finally, our approach allows decoupling of the input/output domains from the expressiveness of the transformation, which provides a convenient framework for further analysis. We measure our performance on ImageNet [1] classification, COCO object detection [2], VOC image segmentation [3]. We evaluate the trade-offs between accuracy, and number of operations measured by multiply-adds (MAdd), as well as actual latency, and the number of parameters.

Searching for MobileNetV3

Abstract: We present the next generation of MobileNets based on a combination of complementary search techniques as well as a novel architecture design. MobileNetV3 is tuned to mobile phone CPUs through a combination of hardware-aware network architecture search (NAS) complemented by the NetAdapt algorithm and then subsequently improved through novel architecture advances. This paper starts the exploration of how automated search algorithms and network design can work together to harness complementary approaches improving the overall state of the art. Through this process we create two new MobileNet models for release: MobileNetV3-Large and MobileNetV3-Small which are targeted for high and low resource use cases. These models are then adapted and applied to the tasks of object detection and semantic segmentation. For the task of semantic segmentation (or any dense pixel prediction), we propose a new efficient segmentation decoder Lite Reduced Atrous Spatial Pyramid Pooling (LR-ASPP). We achieve new state of the art results for mobile classification, detection and segmentation. MobileNetV3-Large is 3.2% more accurate on ImageNet classification while reducing latency by 20% compared to MobileNetV2. MobileNetV3-Small is 6.6% more accurate compared to a MobileNetV2 model with comparable latency. MobileNetV3-Large detection is over 25% faster at roughly the same accuracy as MobileNetV2 on COCO detection. MobileNetV3-Large LR-ASPP is 34% faster than MobileNetV2 R-ASPP at similar accuracy for Cityscapes segmentation.

Searching for MobileNetV3

ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices

Abstract: We introduce an extremely computation-efficient CNN architecture named ShuffleNet, which is designed specially for mobile devices with very limited computing power (e.g., 10-150 MFLOPs). The new architecture utilizes two new operations, pointwise group convolution and channel shuffle, to greatly reduce computation cost while maintaining accuracy. Experiments on ImageNet classification and MS COCO object detection demonstrate the superior performance of ShuffleNet over other structures, e.g. lower top-1 error (absolute 7.8%) than recent MobileNet [12] on ImageNet classification task, under the computation budget of 40 MFLOPs. On an ARM-based mobile device, ShuffleNet achieves ~13Ã— actual speedup over AlexNet while maintaining comparable accuracy.

EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks

Abstract:
Convolutional Neural Networks (ConvNets) are commonly developed at a fixed resource budget, and then scaled up for better accuracy if more resources are available. In this paper, we systematically study model scaling and identify that carefully balancing network depth, width, and resolution can lead to better performance. Based on this observation, we propose a new scaling method that uniformly scales all dimensions of depth/width/resolution using a simple yet highly effective compound coefficient. We demonstrate the effectiveness of this method on scaling up MobileNets and ResNet.
To go even further, we use neural architecture search to design a new baseline network and scale it up to obtain a family of models, called EfficientNets, which achieve much better accuracy and efficiency than previous ConvNets. In particular, our EfficientNet-B7 achieves state-of-the-art 84.4% top-1 / 97.1% top-5 accuracy on ImageNet, while being 8.4x smaller and 6.1x faster on inference than the best existing ConvNet. Our EfficientNets also transfer well and achieve state-of-the-art accuracy on CIFAR-100 (91.7%), Flowers (98.8%), and 3 other transfer learning datasets, with an order of magnitude fewer parameters. Source code is at this https URL.

Comparison of hand-crafted models. Each model has a scaling factor to scale up the number of channels. Here we only report the results of the scaling factor=1.

Discussion: Table {{table:99b9ef84-3c84-4e6b-86dd-7ce3b48f72d1}}  summarizes the models discussed in this section. Significant progresses have been made in lightweight DNN models for classification task, and classification DNN models are also the key component for other CV tasks, like segmentation, detection, etc.
The manual design of lightweight DNN models is considerably dependent on experts' knowledge and also needs a time-consuming hyperparameter exploration.
As automated machine learning (AutoMLhttps://https://www.automl.org/) {{cite:9da73a5}} techniques, like NAS, hyperparameter optimization, emerge to aid in designing DNN models, designers can put their focus on designing effective and efficient DNN modules or operations. Then the new operations can be used as the fundamental elements to generate new models for edge systems.
For instance, the latest MobileNetV3 exploits NAS with the novel architectures from MobileNetV2 and MnasNet {{cite:b5637b1}} to find accurate and efficient DNN model for mobile setting.
We think this is becoming a mainstream trend to adopt NAS with novel lightweight operators to design edge DNN model.
However, existing methods overlook the impact of deployed hardware, on which some operators cannot be supported or effectively executed.
Thus, for the future DNN classification models, especially for edge systems, it is important to design models in a hardware-aware fashion.


Satirical News Detection and Analysis using Attention Mechanism and Linguistic Features

Abstract: Satirical news is considered to be entertainment, but it is potentially deceptive and harmful. Despite the embedded genre in the article, not everyone can recognize the satirical cues and therefore believe the news as true news. We observe that satirical cues are often reflected in certain paragraphs rather than the whole document. Existing works only consider document-level features to detect the satire, which could be limited. We consider paragraph-level linguistic features to unveil the satire by incorporating neural network and attention mechanism. We investigate the difference between paragraph-level features and document-level features, and analyze them on a large satirical news dataset. The evaluation shows that the proposed model detects satirical news effectively and reveals what features are important at which level.

Weight Poisoning Attacks on Pre-trained Models

Abstract: Recently, NLP has seen a surge in the usage of large pre-trained models. Users download weights of models pre-trained on large datasets, then fine-tune the weights on a task of their choice. This raises the question of whether downloading untrusted pre-trained weights can pose a security threat. In this paper, we show that it is possible to construct ``weight poisoning'' attacks where pre-trained weights are injected with vulnerabilities that expose ``backdoors'' after fine-tuning, enabling the attacker to manipulate the model prediction simply by injecting an arbitrary keyword. We show that by applying a regularization method, which we call RIPPLe, and an initialization procedure, which we call Embedding Surgery, such attacks are possible even with limited knowledge of the dataset and fine-tuning procedure. Our experiments on sentiment classification, toxicity detection, and spam detection show that this attack is widely applicable and poses a serious threat. Finally, we outline practical defenses against such attacks. Code to reproduce our experiments is available at https://github.com/neulab/RIPPLe.

GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding

Abstract: For natural language understanding (NLU) technology to be maximally useful, both practically and as a scientific object of study, it must be general: it must be able to process language in a way that is not exclusively tailored to any one specific task or dataset. In pursuit of this objective, we introduce the General Language Understanding Evaluation benchmark (GLUE), a tool for evaluating and analyzing the performance of models across a diverse range of existing NLU tasks. GLUE is model-agnostic, but it incentivizes sharing knowledge across tasks because certain tasks have very limited training data. We further provide a hand-crafted diagnostic test suite that enables detailed linguistic analysis of NLU models. We evaluate baselines based on current methods for multi-task and transfer learning and find that they do not immediately give substantial improvements over the aggregate performance of training a separate model per task, indicating room for improvement in developing general and robust NLU systems.

Dataset details. #C means the number of classes.

We evaluate our methods on 6 datasets. Details are shown in Table {{table:d6fc8459-4175-4fc1-90eb-25cf6c798a2a}} .
We use RoBERTa-large as the backbone pre-trained language model.


Towards Automatic Learning of Procedures From Web Instructional Videos

Abstract: The potential for agents, whether embodied or software, to learn by observing other agents performing procedures involving objects and actions is rich. Current research on automatic procedure learning heavily relies on action labels or video subtitles, even during the evaluation phase, which makes them infeasible in real-world scenarios. This leads to our question: can the human-consensus structure of a procedure be learned from a large set of long, unconstrained videos (e.g., instructional videos from YouTube) with only visual evidence? To answer this question, we introduce the problem of procedure segmentation---to segment a video procedure into category-independent procedure segments. Given that no large-scale dataset is available for this problem, we collect a large-scale procedure segmentation dataset with procedure segments temporally localized and described; we use cooking videos and name the dataset YouCook2. We propose a segment-level recurrent network for generating procedure segments by modeling the dependencies across segments. The generated segments can be used as pre-processing for other tasks, such as dense video captioning and event parsing. We show in our experiments that the proposed model outperforms competitive baselines in procedure segmentation.

Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval

Abstract: Our objective in this work is video-text retrieval – in particular a joint embedding that enables efficient text-to-video retrieval. The challenges in this area include the design of the visual architecture and the nature of the training data, in that the available large scale video-text training datasets, such as HowTo100M, are noisy and hence competitive performance is achieved only at scale through large amounts of compute.We address both these challenges in this paper. We propose an end-to-end trainable model that is designed to take advantage of both large-scale image and video captioning datasets. Our model is an adaptation and extension of the recent ViT and Timesformer architectures, and consists of attention in both space and time. The model is flexible and can be trained on both image and video text datasets, either independently or in conjunction. It is trained with a curriculum learning schedule that begins by treating images as ‘frozen’ snapshots of video, and then gradually learns to attend to increasing temporal context when trained on video datasets. We also provide a new video-text pretraining dataset WebVid-2M, comprised of over two million videos with weak captions scraped from the internet. Despite training on datasets that are an order of magnitude smaller, we show that this approach yields state-of-the-art results on standard downstream video-retrieval benchmarks including MSR-VTT, MSVD, DiDeMo and LSMDC.

The multimodal datasets

Practical guide for data. Multimodal datasets play a crucial role in advancing research on vision and language tasks. These datasets combine different modalities, such as images, text, videos, and audio, providing rich and diverse sources of information for various applications. We categorize the multimodal datasets into different types and present a curated selection of representative datasets for each category, as shown in Table {{table:4bbaf161-aeae-4402-b517-dc7eaf5c10bd}} . For future research, we can use these datasets to conduct experiments to test the model's effectiveness.


WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learning

Abstract: The milestone improvements brought about by deep representation learning and pre-training techniques have led to large performance gains across downstream NLP, IR and Vision tasks. Multimodal modeling techniques aim to leverage large high-quality visio-linguistic datasets for learning complementary information across image and text modalities. In this paper, we introduce the Wikipedia-based Image Text (WIT) Dataset to better facilitate multimodal, multilingual learning. WIT is composed of a curated set of 37.5 million entity rich image-text examples with 11.5 million unique images across 108 Wikipedia languages. Its size enables WIT to be used as a pretraining dataset for multimodal models, as we show when applied to downstream tasks such as image-text retrieval. WIT has four main and unique advantages. First, WIT is the largest multimodal dataset by the number of image-text examples by 3x (at the time of writing). Second, WIT is massively multilingual (first of its kind) with coverage over 100+ languages (each of which has at least 12K examples) and provides cross-lingual texts for many images. Third, WIT represents a more diverse set of concepts and real world entities relative to what previous datasets cover. Lastly, WIT provides a very challenging real-world test set, as we empirically illustrate using an image-text retrieval task as an example. WIT Dataset is available for download and use via a Creative Commons license here: https://github.com/google-research-datasets/wit.

Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts

Abstract: The availability of large-scale image captioning and visual question answering datasets has contributed significantly to recent successes in vision-and-language pretraining. However, these datasets are often collected with overrestrictive requirements inherited from their original target tasks (e.g., image caption generation), which limit the resulting dataset scale and diversity. We take a step further in pushing the limits of vision-and-language pretraining data by relaxing the data collection pipeline used in Conceptual Captions 3M (CC3M) [54] and introduce the Conceptual 12M (CC12M), a dataset with 12 million image-text pairs specifically meant to be used for visionand-language pre-training. We perform an analysis of this dataset and benchmark its effectiveness against CC3M on multiple downstream tasks with an emphasis on long-tail visual recognition. Our results clearly illustrate the benefit of scaling up pre-training data for vision-and-language tasks, as indicated by the new state-of-the-art results on both the nocaps and Conceptual Captions benchmarks.1

Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering

Abstract: Problems at the intersection of vision and language are of significant importance both as challenging research questions and for the rich set of applications they enable. However, inherent structure in our world and bias in our language tend to be a simpler signal for learning than visual modalities, resulting in models that ignore visual information, leading to an inflated sense of their capability. We propose to counter these language priors for the task of Visual Question Answering (VQA) and make vision (the V in VQA) matter! Specifically, we balance the popular VQA dataset by collecting complementary images such that every question in our balanced dataset is associated with not just a single image, but rather a pair of similar images that result in two different answers to the question. Our dataset is by construction more balanced than the original VQA dataset and has approximately twice the number of image-question pairs. Our complete balanced dataset is publicly available at www.visualqa.org as part of the 2nd iteration of the Visual Question Answering Dataset and Challenge (VQA v2.0). We further benchmark a number of state-of-art VQA models on our balanced dataset. All models perform significantly worse on our balanced dataset, suggesting that these models have indeed learned to exploit language priors. This finding provides the first concrete empirical evidence for what seems to be a qualitative sense among practitioners. Finally, our data collection protocol for identifying complementary images enables us to develop a novel interpretable model, which in addition to providing an answer to the given (image, question) pair, also provides a counter-example based explanation. Specifically, it identifies an image that is similar to the original image, but it believes has a different answer to the same question. This can help in building trust for machines among their users.

Statistics of the knowledge sources used.

We use the following four sources of knowledge in our experiments:
Wikipedia-Image-Text (WIT) {{cite:5757840}} consists of the images in Wikipedia, as well as their alt-text captions and contextualized text passages.
Conceptual (CC12M) {{cite:225b9b7}} contains web images paired with alt-text captions. It includes many long-tail entities.
VQA-v2 {{cite:73ba28a}} is a visual question answering dataset. We merge all question-answer pairs per image into a single passage.
WikiData {{cite:58ba08c}} is a structural knowledge graph encoding relations between Wikipedia entities. We linearize all relational triplets per entity into a textual passage following the procedure of {{cite:e8554bb}}.
We have listed the statistical details of these knowledge sources in Table {{table:e84611f1-e3ea-4660-8b1d-e7bae7393366}} .