Scikit-learn: Machine Learning in Python

Abstract: Scikit-learn is a Python module integrating a wide range of state-of-the-art machine learning algorithms for medium-scale supervised and unsupervised problems. This package focuses on bringing machine learning to non-specialists using a general-purpose high-level language. Emphasis is put on ease of use, performance, documentation, and API consistency. It has minimal dependencies and is distributed under the simplified BSD license, encouraging its use in both academic and commercial settings. Source code, binaries, and documentation can be downloaded from http://scikit-learn.sourceforge.net.

XGBoost: A Scalable Tree Boosting System

Abstract: Tree boosting is a highly effective and widely used machine learning method. In this paper, we describe a scalable end-to-end tree boosting system called XGBoost, which is used widely by data scientists to achieve state-of-the-art results on many machine learning challenges. We propose a novel sparsity-aware algorithm for sparse data and weighted quantile sketch for approximate tree learning. More importantly, we provide insights on cache access patterns, data compression and sharding to build a scalable tree boosting system. By combining these insights, XGBoost scales beyond billions of examples using far fewer resources than existing systems.

Surprise: A Python library for recommender systems

Abstract: Recommender systems aim at providing users with a list of recommendations of items that a service offers. For example, a video streaming service will typically rely on a recommender system to propose a personalized list of movies or series to each of its users. A typical problem in recommendation is that of rating prediction: given an incomplete dataset of useritem interations which take the form of numerical ratings (e.g. on a scale from 1 to 5), the goal is to predict the missing ratings for all remaining user-item pairs.

LensKit for Python: Next-Generation Software for Recommender System Experiments

Abstract: LensKit is an open-source toolkit for building, researching, and learning about recommender systems. First released in 2010 as a Java framework, it has supported diverse published research, small-scale production deployments, and education in both MOOC and traditional classroom settings. In this paper, I present the next generation of the LensKit project, re-envisioning the original tool's objectives as flexible Python package for supporting recommender systems research and development. LensKit for Python (LKPY) enables researchers and students to build robust, flexible, and reproducible experiments that make use of the large and growing PyData and Scientific Python ecosystem, including scikit-learn, TensorFlow, and PyTorch. To that end, it provides classical collaborative filtering implementations, recommender system evaluation metrics, data preparation routines, and tools for efficiently batch running recommendation algorithms, all usable in any combination with each other or with other Python software. This paper describes the design goals, use cases, and capabilities of LKPY, contextualized in a reflection on the successes and failures of the original LensKit for Java software.

An overview of the evaluated algorithms and their category. We evaluated the algorithms as implemented in their respective library. Importantly, it is shown whether the algorithm can incorporate side information and how their hyperparameters were tuned.

We evaluated 19 algorithms from nine libraries (Table {{table:75af9c80-a239-46db-8217-f0c3dc5e67e8}} ) on six feature sets generated from the Movielens-100K {{cite:5ddae5a}} data set (Table {{table:661bab0a-53b6-40f7-a321-c20a8b0225b4}} ).

To have an equal ground for algorithm comparisons, we applied some constraints. We performed all experiments on implementations in publicly available libraries to increase accessibility and reproducibility. The Movielens-100K {{cite:5ddae5a}} data set contains explicit ratings as integers ranging from one to five. Therefore, we only chose algorithms that can take explicit ratings as input and predict ratings in that same format. We evaluated the algorithms using five-fold cross-validation. Since one of the research questions is about a comparison of the performance of algorithms against each other, we performed hyperparameter tuning on algorithms that do not default to a tuned parameter setup for the Movielens-100K {{cite:5ddae5a}} data set. Depending on the algorithm, we tuned the hyperparameters either manually, with a random search, or by using SMAC3 {{cite:cdc0d62}}, which is an all-purpose hyperparameter optimization tool. We set the time budget of AutoML tools to one hour for each fold. Table {{table:75af9c80-a239-46db-8217-f0c3dc5e67e8}}  lists all libraries and algorithms and their categories, if they use side information or not, and how their hyperparameters were tuned.


Large-Scale Evolution of Image Classifiers

Abstract: Neural networks have proven effective at solving difficult problems but designing their architectures can be challenging, even for image classification problems alone. Our goal is to minimize human participation, so we employ evolutionary algorithms to discover such networks automatically. Despite significant computational requirements, we show that it is now possible to evolve models with accuracies within the range of those published in the last year. Specifically, we employ simple evolutionary techniques at unprecedented scales to discover models for the CIFAR-10 and CIFAR-100 datasets, starting from trivial initial conditions and reaching accuracies of 94.6% (95.6% for ensemble) and 77.0%, respectively. To do this, we use novel and intuitive mutation operators that navigate large search spaces; we stress that no human participation is required once evolution starts and that the output is a fully-trained model. Throughout this work, we place special emphasis on the repeatability of results, the variability in the outcomes and the computational requirements.

Genetic CNN

Abstract: The deep Convolutional Neural Network (CNN) is the state-of-the-art solution for large-scale visual recognition. Following basic principles such as increasing the depth and constructing highway connections, researchers have manually designed a lot of fixed network structures and verified their effectiveness. In this paper, we discuss the possibility of learning deep network structures automatically. Note that the number of possible network structures increases exponentially with the number of layers in the network, which inspires us to adopt the genetic algorithm to efficiently traverse this large search space. We first propose an encoding method to represent each network structure in a fixed-length binary string, and initialize the genetic algorithm by generating a set of randomized individuals. In each generation, we define standard genetic operations, e.g., selection, mutation and crossover, to eliminate weak individuals and then generate more competitive ones. The competitiveness of each individual is defined as its recognition accuracy, which is obtained via training the network from scratch and evaluating it on a validation set. We run the genetic process on two small datasets, i.e., MNIST and CIFAR10, demonstrating its ability to evolve and find high-quality structures which are little studied before. These structures are also transferrable to the large-scale ILSVRC2012 dataset.

A Genetic Programming Approach to Designing Convolutional Neural Network Architectures

Abstract: The convolutional neural network (CNN), which is one of the deep learning models, has seen much success in a variety of computer vision tasks. However, designing CNN architectures still requires expert knowledge and a lot of trial and error. In this paper, we attempt to automatically construct CNN architectures for an image classification task based on Cartesian genetic programming (CGP). In our method, we adopt highly functional modules, such as convolutional blocks and tensor concatenation, as the node functions in CGP. The CNN structure and connectivity represented by the CGP encoding method are optimized to maximize the validation accuracy. To evaluate the proposed method, we constructed a CNN architecture for the image classification task with the CIFAR-10 dataset. The experimental result shows that the proposed method can be used to automatically find the competitive CNN architecture compared with state-of-the-art models.

Hierarchical Representations for Efficient Architecture Search

Abstract: We explore efficient neural architecture search methods and show that a simple yet powerful evolutionary algorithm can discover new architectures with excellent performance. Our approach combines a novel hierarchical genetic representation scheme that imitates the modularized design pattern commonly adopted by human experts, and an expressive search space that supports complex topologies. Our algorithm efficiently discovers architectures that outperform a large number of manually designed models for image classification, obtaining top-1 error of 3.6% on CIFAR-10 and 20.3% when transferred to ImageNet, which is competitive with the best existing neural architecture search approaches. We also present results using random search, achieving 0.3% less top-1 accuracy on CIFAR-10 and 0.1% less on ImageNet whilst reducing the search time from 36 hours down to 1 hour.

Simple And Efficient Architecture Search for Convolutional Neural Networks

Abstract: Neural networks have recently had a lot of success for many tasks. However, neural network architectures that perform well are still typically designed manually by experts in a cumbersome trial-and-error process. We propose a new method to automatically search for well-performing CNN architectures based on a simple hill climbing procedure whose operators apply network morphisms, followed by short optimization runs by cosine annealing. Surprisingly, this simple method yields competitive results, despite only requiring resources in the same order of magnitude as training a single network. E.g., on CIFAR-10, our method designs and trains networks with an error rate below 6% in only 12 hours on a single GPU; training for one day reduces this error further, to almost 5%.

High-level details of various evolutionary algorithms for neural architecture search.

In the remaining section we discuss the popular choices for search space, mutation operators and selection functions that have been utilized for neural architecture search.
We note that EA-based neural architecture search methods include a set of highly diverse approaches which have benefited from the varied choices of encoding the search space along with the choices of mutation operators and selection functions.
In the context of this work we describe six notable works in EA-based neural architecture search.
A broad overview of these approaches is provided in Table {{table:3eba4711-4464-44e1-8ba4-a9b7a7b2580a}} .


Genomic Profiling of the Steroidogenic Acute Regulatory Protein in Breast Cancer: In Silico Assessments and a Mechanistic Perspective

Abstract: Cancer is a multifactorial condition with aberrant growth of cells. A substantial number of cancers, breast in particular, are hormone sensitive and evolve due to malfunction in the steroidogenic machinery. Breast cancer, one of the most prevalent form of cancers in women, is primarily stimulated by estrogens. Steroid hormones are made from cholesterol, and regulation of steroid/estrogen biosynthesis is essentially influenced by the steroidogenic acute regulatory (StAR) protein. Although the impact of StAR in breast cancer remains a mystery, we recently reported that StAR protein is abundantly expressed in hormone sensitive breast cancer, but not in its non-cancerous counterpart. Herein, we analyzed genomic profiles, hormone receptor expression, mutation, and survival for StAR and steroidogenic enzyme genes in a variety of hormone sensitive cancers. These profiles were specifically assessed in breast cancer, exploiting The Cancer Genome Atlas (TCGA) datasets. Whereas StAR and key steroidogenic enzyme genes evaluated (CYP11A1, HSD3B, CYP17A1, CYP19A1, and HSD17B) were altered to varying levels in these hormone responsive cancers, amplification of the StAR gene was correlated with poor overall survival of patients afflicted with breast cancer. Amplification of the StAR gene and its correlation to survival was also verified in a number of breast cancer studies. Additionally, TCGA breast cancer tumors associated with aberrant high expression of StAR mRNA were found to be an unfavorable risk factor for survival of patients with breast cancer. Further analyses of tumors, nodal status, and metastases of breast cancer tumors expressing StAR mRNA displayed cancer deaths in stage specific manners. The majority of these tumors were found to express estrogen and progesterone receptors, signifying a link between StAR and luminal subtype breast cancer. Collectively, analyses of genomic and molecular profiles of key steroidogenic factors provide novel insights that StAR plays an important role in the biologic behavior and/or pathogenesis of hormone sensitive breast cancer.

Exact Penalization at D-Stationary Points of Cardinality- or Rank-Constrained Problem

Abstract: This paper studies the properties of d-stationary points of the trimmed lasso (Luo et al., 2013, Huang et al., 2015, and Gotoh et al., 2018) and the composite optimization problem with the truncated nuclear norm (Gao and Sun, 2010, and Zhang et al., 2012), which are known as tight relaxations of nonconvex optimization problems that have either cardinality or rank constraints, respectively. First, we extend the trimmed lasso for broader applications and conduct a unified analysis of the property of the generalized trimmed lasso. Next, the equivalence between local optimality and d-stationarity of the generalized trimmed lasso is shown under a suitable assumption. More generally, the equivalence is shown for problems that minimize a function defined by the pointwise minimum of finitely many convex functions. Then, we present new results of the exact penalty at d-stationary points of the generalized trimmed lasso and the problem with the truncated nuclear norm penalty under mild assumptions. Our exact penalty results are not only new, but also intuitive, so that we can see what properties of the problems play a role for establishing the exact penalization. Lastly, it is shown that existing algorithms that yield a d-stationary point always converge to a solution that satisfies the cardinality or rank constraint.

Groups discovered by CompFS on the METABRIC dataset.

We found supporting evidence in the scientific literature for all but 1 of the genes discovered by CompFS (Table {{table:32e3b3a9-38f5-4cef-934a-a2cdb8d13fe9}} ). In addition, within each group, we found further evidence of the interactions between genes, demonstrating the ability for CompFS to learn informative groups of features.
For example, in Group 1, CXCR1 and PEN-2 (the protein encoded by PSENEN) are known to interact {{cite:4e96ed8}}. In Group 2, BMP6 encodes a member of the TGF-{{formula:3b43ac21-0677-409c-9e01-4528a1a97f39}}  superfamily of proteins, and TGF-{{formula:81c814dc-d7a4-41b9-9248-4766b22489a1}}  triggers activation of SMAD3 {{cite:0661cca}}. In the same group, MAPK1 activity is dependent on the activity of PRKCQ in breast cancer cells {{cite:96df9d5}}, while MAPK1 is also known to interact with MAPT {{cite:8c3def1}}, SMAD3 {{cite:7d662be}}, and BMP6 {{cite:0170cd6}}.
Additional supporting evidence can be found in Appendix .

We first provide validation for the individual features discovered by CompFS in Table {{table:32e3b3a9-38f5-4cef-934a-a2cdb8d13fe9}} . We found supporting evidence for 24 of the 25 features discovered by CompFS.


PlantDoc: A Dataset for Visual Plant Disease Detection

Abstract: India loses 35% of the annual crop yield due to plant diseases. Early detection of plant diseases remains difficult due to the lack of lab infrastructure and expertise. In this paper, we explore the possibility of computer vision approaches for scalable and early plant disease detection. The lack of availability of sufficiently large-scale non-lab data set remains a major challenge for enabling vision based plant disease detection. Against this background, we present PlantDoc: a dataset for visual plant disease detection. Our dataset contains 2,598 data points in total across 13 plant species and up to 17 classes of diseases, involving approximately 300 human hours of effort in annotating internet scraped images. To show the efficacy of our dataset, we learn 3 models for the task of plant disease classification. Our results show that modelling using our dataset can increase the classification accuracy by up to 31%. We believe that our dataset can help reduce the entry barrier of computer vision techniques in plant disease detection.

Automatic and Reliable Leaf Disease Detection Using Deep Learning Techniques

Abstract: Plants are a major source of food for the world population. Plant diseases contribute to production loss, which can be tackled with continuous monitoring. Manual plant disease monitoring is both laborious and error-prone. Early detection of plant diseases using computer vision and artificial intelligence (AI) can help to reduce the adverse effects of diseases and also overcome the shortcomings of continuous human monitoring. In this work, we propose the use of a deep learning architecture based on a recent convolutional neural network called EfficientNet on 18,161 plain and segmented tomato leaf images to classify tomato diseases. The performance of two segmentation models i.e., U-net and Modified U-net, for the segmentation of leaves is reported. The comparative performance of the models for binary classification (healthy and unhealthy leaves), six-class classification (healthy and various groups of diseased leaves), and ten-class classification (healthy and various types of unhealthy leaves) are also reported. The modified U-net segmentation model showed accuracy, IoU, and Dice score of 98.66%, 98.5%, and 98.73%, respectively, for the segmentation of leaf images. EfficientNet-B7 showed superior performance for the binary classification and six-class classification using segmented images with an accuracy of 99.95% and 99.12%, respectively. Finally, EfficientNet-B4 achieved an accuracy of 99.89% for ten-class classification using segmented images. It can be concluded that all the architectures performed better in classifying the diseases when trained with deeper networks on segmented images. The performance of each of the experimental studies reported in this work outperforms the existing literature.

Summary of deep learning approaches.

Deep learning is a rising branch of machine learning which consists of different architectures and associated learning algorithms. For leaf disease classification, most deep learning models and algorithms are based on neural networks with many number of hidden layers. We categorise deep learning approaches for this task into deep neural networks, convolutional neural networks for image classification, and convolutional neural networks for object detection& classification. Table {{table:ce27bd55-1333-480f-a125-5dcc9c9df76d}}  provides a summary of recent Deep Learning approaches for leaf disease classification.


Derivative of continuous action function

Variance-based Actor Critic for robustness

Asynchronous Methods for Deep Reinforcement Learning

Abstract: We propose a conceptually simple and lightweight framework for deep reinforcement learning that uses asynchronous gradient descent for optimization of deep neural network controllers. We present asynchronous variants of four standard reinforcement learning algorithms and show that parallel actor-learners have a stabilizing effect on training allowing all four methods to successfully train neural network controllers. The best performing method, an asynchronous variant of actor-critic, surpasses the current state-of-the-art on the Atari domain while training for half the time on a single multi-core CPU instead of a GPU. Furthermore, we show that asynchronous actor-critic succeeds on a wide variety of continuous motor control problems as well as on a new task of navigating random 3D mazes using a visual input.

Continuous control with deep reinforcement learning

Abstract: We adapt the ideas underlying the success of Deep Q-Learning to the continuous action domain. We present an actor-critic, model-free algorithm based on the deterministic policy gradient that can operate over continuous action spaces. Using the same learning algorithm, network architecture and hyper-parameters, our algorithm robustly solves more than 20 simulated physics tasks, including classic problems such as cartpole swing-up, dexterous manipulation, legged locomotion and car driving. Our algorithm is able to find policies whose performance is competitive with those found by a planning algorithm with full access to the dynamics of the domain and its derivatives. We further demonstrate that for many of the tasks the algorithm can learn policies end-to-end: directly from raw pixel inputs.

Trust Region Policy Optimization

Abstract: We describe an iterative procedure for optimizing policies, with guaranteed monotonic improvement. By making several approximations to the theoretically-justified procedure, we develop a practical algorithm, called Trust Region Policy Optimization (TRPO). This algorithm is similar to natural policy gradient methods and is effective for optimizing large nonlinear policies such as neural networks. Our experiments demonstrate its robust performance on a wide variety of tasks: learning simulated robotic swimming, hopping, and walking gaits; and playing Atari games using images of the screen as input. Despite its approximations that deviate from the theory, TRPO tends to give monotonic improvement, with little tuning of hyperparameters.

Proximal Policy Optimization Algorithms

Abstract: We propose a new family of policy gradient methods for reinforcement learning, which alternate between sampling data through interaction with the environment, and optimizing a "surrogate" objective function using stochastic gradient ascent. Whereas standard policy gradient methods perform one gradient update per data sample, we propose a novel objective function that enables multiple epochs of minibatch updates. The new methods, which we call proximal policy optimization (PPO), have some of the benefits of trust region policy optimization (TRPO), but they are much simpler to implement, more general, and have better sample complexity (empirically). Our experiments test PPO on a collection of benchmark tasks, including simulated robotic locomotion and Atari game playing, and we show that PPO outperforms other online policy gradient methods, and overall strikes a favorable balance between sample complexity, simplicity, and wall-time.

Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor

Abstract: Model-free deep reinforcement learning (RL) algorithms have been demonstrated on a range of challenging decision making and control tasks. However, these methods typically suffer from two major challenges: very high sample complexity and brittle convergence properties, which necessitate meticulous hyperparameter tuning. Both of these challenges severely limit the applicability of such methods to complex, real-world domains. In this paper, we propose soft actor-critic, an off-policy actor-critic deep RL algorithm based on the maximum entropy reinforcement learning framework. In this framework, the actor aims to maximize expected reward while also maximizing entropy. That is, to succeed at the task while acting as randomly as possible. Prior deep RL methods based on this framework have been formulated as Q-learning methods. By combining off-policy updates with a stable stochastic actor-critic formulation, our method achieves state-of-the-art performance on a range of continuous control benchmark tasks, outperforming prior on-policy and off-policy methods. Furthermore, we demonstrate that, in contrast to other off-policy algorithms, our approach is very stable, achieving very similar performance across different random seeds.

Policy-Based Algorithms:REINFORCE, Asynchronous Advantage Actor Critic, Deep DeterministicPolicy Gradient, Trust Region Policy Optimization, Proximal PolicyOptimization, Soft Actor Critic

Value-based reinforcement learning works well with applications such
as games, with discrete action spaces.
The next chapter, Chap. , discusses a different
approach: deep policy-based reinforcement learning
(Table {{table:3d6eef29-85dd-420e-b68b-a929adc0a399}} ). In addition to discrete spaces, this approach is also
suited for continuous actions spaces, such as robot arm
movement, and simulated articulated locomotion. We see how a simulated
half-cheetah teaches itself to run.

Policy-based methods are a popular approach in model-free deep reinforcement
learning. Many algorithms have been developed that perform
well. Table {{table:3d6eef29-85dd-420e-b68b-a929adc0a399}}  lists some of the better known algorithms that will be
covered in this chapter.

Inspired by breakthroughs in supervised image recognition, deep
learning was also applied to Q-learning, causing the Atari breakthrough for
deep reinforcement learning. The basis of the success of deep learning
in reinforcement learning are methods to break correlations and improve
convergence (replay buffer and a separate target
network). The DQN algorithm {{cite:794ba97}} has become quite well known. Policy-based and actor critic approaches
work well with deep learning, and are also applicable to continuous
action spaces. Many model-free actor critic variants have been
developed {{cite:ed3ec26}}, {{cite:17b00f5}}, {{cite:82635ed}}, {{cite:8a4c450}},
they are often tested on simulated robot applications. Algorithms often
reach good quality optima, but model-free algorithms have a high
sample complexity. Tables {{table:80350558-2648-485a-b725-feda1ba902f1}}  and {{table:3d6eef29-85dd-420e-b68b-a929adc0a399}}  list these algorithms.


The Singing Voice Conversion Challenge 2023

Abstract: We present the latest iteration of the voice conversion challenge (VCC) series, a bi-annual scientific event aiming to compare and understand different voice conversion (VC) systems based on a common dataset. This year we shifted our focus to singing voice conversion (SVC), thus named the challenge the Singing Voice Conversion Challenge (SVCC). A new database was constructed for two tasks, namely in-domain and cross-domain SVC. The challenge was run for two months, and in total we received 26 submissions, including 2 baselines. Through a large-scale crowd-sourced listening test, we observed that for both tasks, although human-level naturalness was achieved by the top system, no team was able to obtain a similarity score as high as the target speakers. Also, as expected, cross-domain SVC is harder than in-domain SVC, especially in the similarity aspect. We also investigated whether existing objective measurements were able to predict perceptual performance, and found that only few of them could reach a significant correlation.

Multi-Singer: Fast Multi-Singer Singing Voice Vocoder With A Large-Scale Corpus

Abstract: High-fidelity multi-singer singing voice synthesis is challenging for neural vocoder due to the singing voice data shortage, limited singer generalization, and large computational cost. Existing open corpora could not meet requirements for high-fidelity singing voice synthesis because of the scale and quality weaknesses. Previous vocoders have difficulty in multi-singer modeling, and a distinct degradation emerges when conducting unseen singer singing voice generation. To accelerate singing voice researches in the community, we release a large-scale, multi-singer Chinese singing voice dataset OpenSinger. To tackle the difficulty in unseen singer modeling, we propose Multi-Singer, a fast multi-singer vocoder with generative adversarial networks. Specifically, 1) Multi-Singer uses a multi-band generator to speed up both training and inference procedure. 2) to capture and rebuild singer identity from the acoustic feature (i.e., mel-spectrogram), Multi-Singer adopts a singer conditional discriminator and conditional adversarial training objective. 3) to supervise the reconstruction of singer identity in the spectrum envelopes in frequency domain, we propose an auxiliary singer perceptual loss. The joint training approach effectively works in GANs for multi-singer voices modeling. Experimental results verify the effectiveness of OpenSinger and show that Multi-Singer improves unseen singer singing voices modeling in both speed and quality over previous methods. The further experiment proves that combined with FastSpeech 2 as the acoustic model, Multi-Singer achieves strong robustness in the multi-singer singing voice synthesis pipeline.

Learning the Beauty in Songs: Neural Singing Voice Beautifier

Abstract: We are interested in a novel task, singing voice beautification (SVB). Given the singing voice of an amateur singer, SVB aims to improve the intonation and vocal tone of the voice, while keeping the content and vocal timbre. Current automatic pitch correction techniques are immature, and most of them are restricted to intonation but ignore the overall aesthetic quality. Hence, we introduce Neural Singing Voice Beautifier (NSVB), the first generative model to solve the SVB task, which adopts a conditional variational autoencoder as the backbone and learns the latent representations of vocal tone. In NSVB, we propose a novel time-warping approach for pitch correction: Shape-Aware Dynamic Time Warping (SADTW), which ameliorates the robustness of existing time-warping approaches, to synchronize the amateur recording with the template pitch curve. Furthermore, we propose a latent-mapping algorithm in the latent space to convert the amateur vocal tone to the professional one. To achieve this, we also propose a new dataset containing parallel singing recordings of both amateur and professional versions. Extensive experiments on both Chinese and English songs demonstrate the effectiveness of our methods in terms of both objective and subjective metrics. Audio samples are available at https://neuralsvb.github.io. Codes: https://github.com/MoonInTheRiver/NeuralSVB.

DiffSinger: Singing Voice Synthesis via Shallow Diffusion Mechanism

Abstract:
Singing voice synthesis (SVS) systems are built to synthesize high-quality and expressive singing voice, in which the acoustic model generates the acoustic features (e.g., mel-spectrogram) given a music score. Previous singing acoustic models adopt a simple loss (e.g., L1 and L2) or generative adversarial network (GAN) to reconstruct the acoustic features, while they suffer from over-smoothing and unstable training issues respectively, which hinder the naturalness of synthesized singing.
In this work, we propose DiffSinger, an acoustic model for SVS based on the diffusion probabilistic model. DiffSinger is a parameterized Markov chain that iteratively converts the noise into mel-spectrogram conditioned on the music score. By implicitly optimizing variational bound, DiffSinger can be stably trained and generate realistic outputs.
To further improve the voice quality and speed up inference, we introduce a shallow diffusion mechanism to make better use of the prior knowledge learned by the simple loss. Specifically, DiffSinger starts generation at a shallow step smaller than the total number of diffusion steps, according to the intersection of the diffusion trajectories of the ground-truth mel-spectrogram and the one predicted by a simple mel-spectrogram decoder. Besides, we propose boundary prediction methods to locate the intersection and determine the shallow step adaptively.
The evaluations conducted on a Chinese singing dataset demonstrate that DiffSinger outperforms state-of-the-art SVS work. Extensional experiments also prove the generalization of our methods on text-to-speech task (DiffSpeech). Audio samples: https://diffsinger.github.io. Codes: https://github.com/MoonInTheRiver/DiffSinger.

Opencpop: A High-Quality Open Source Chinese Popular Song Corpus for Singing Voice Synthesis

Abstract: This paper introduces Opencpop, a publicly available high-quality Mandarin singing corpus designed for singing voice synthesis (SVS). The corpus consists of 100 popular Mandarin songs performed by a female professional singer. Audio files are recorded with studio quality at a sampling rate of 44,100 Hz and the corresponding lyrics and musical scores are provided. All singing recordings have been phonetically annotated with phoneme boundaries and syllable (note) boundaries. To demonstrate the reliability of the released data and to provide a baseline for future research, we built baseline deep neural network-based SVS models and evaluated them with both objective metrics and subjective mean opinion score (MOS) measure. Experimental results show that the best SVS model trained on our database achieves 3.70 MOS, indicating the reliability of the provided corpus. Opencpop is released to the open-source community WeNet, and the corpus, as well as synthesized demos, can be found on the project homepage.

PJS: phoneme-balanced Japanese singing-voice corpus

Abstract: This paper presents a free Japanese singing-voice corpus that can be used for highly applicable singing-voice synthesis research. A singing-voice corpus helps develop singing-voice synthesis, but existing corpora have two critical problems: data imbalance (i.e., singing-voice corpora do not guarantee phoneme balance, unlike speaking-voice corpora) and copyright issues (i.e., cannot legally share data). To avoid these problems, we constructed a phoneme-balanced Japanese singing-voice (PJS) corpus that guarantees phoneme balance and is licensed with CC BY-SA 4.0, and we composed melodies using a phoneme-balanced speaking-voice corpus. Furthermore, to temporally align phoneme sequences with speech feature sequences, we compare three alignment methods: Viterbi alignment of hidden Markov models, dynamic time warping using a synthesized voice, and statistical voice conversion. Experimental results demonstrate that 1) our corpus contains more unique monophones and diphones than an existing corpus, and 2) the voice-conversion-based method provides the most accurate alignment.

Statistics of the fine-tuned datasets for our singing voice vocoder. VCTK and SVCC (cross-domain) are two speech datasets, while the others are singing voice datasets.

For WeNet, we use the official models pretrained by 10k hours Wenetspeech and 10k hours Gigaspeech to extract content features. For Whisper, we use the multilingual MEDIUM model. For ContentVec, we use the official 500-CLASS model preatrained by 1k hours Librispeech. For BigVGAN vocoder, we adopt the offical 24KHZ_100BAND model that pretrained by 585 hours LibriTTS {{cite:0006f91}}. We fine-tune it with the five training datasets and additional four datasets, in total 83.1 hours of speech and 128.3 hours of singing voice data (Table {{table:4ae57685-fe93-4d84-8a0c-836e7de968f5}} ). The sampling rate is 24kHz, the mel bands are 100, and the hop size is 256.

Effect of Singing Voice Data for Vocoder (EQ2)

To investigate the similarity or distinction between singing voice and speech, we conduct a preliminary data analysis from the aspects of F0, energy, and speaker timbre. Specifically, we analyze the differences between singing voice and speech in the range of F0 and energy, and the distribution of the Long-Term Average Spectrum (LTAS). Note that LTAS represents the formant and resonance features of the human vocal tract, which can reveal the timbre characteristics {{cite:27a2d88}}. We use the ten datasets of Table {{table:4ae57685-fe93-4d84-8a0c-836e7de968f5}}  for analysis. In Figure {{figure:cb18e15b-581c-4e5c-9b70-84acfcbd68c0}} , we observe that there are larger and wider ranges of F0 and energy in singing voices. Besides, singing timbres distribute distinctly from speaking ones and are more diverse. Given these specific features of the singing voice, it is necessary to explore whether fine-tuning a pretrained speech vocoder with singing voice data can be helpful.


Pneumonia images from downsampled version of CheXpert

Contains fungal pneumonia, bacterial pneumonia and non-COVID viral pneumonia (12,17,12 during our experiments)

CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison

Abstract: Large, labeled datasets have driven deep learning methods to achieve expert-level performance on a variety of medical imaging tasks. We present CheXpert, a large dataset that contains 224,316 chest radiographs of 65,240 patients. We design a labeler to automatically detect the presence of 14 observations in radiology reports, capturing uncertainties inherent in radiograph interpretation. We investigate different approaches to using the uncertainty labels for training convolutional neural networks that output the probability of these observations given the available frontal and lateral radiographs. On a validation set of 200 chest radiographic studies which were manually annotated by 3 board-certified radiologists, we find that different uncertainty approaches are useful for different pathologies. We then evaluate our best model on a test set composed of 500 chest radiographic studies annotated by a consensus of 5 board-certified radiologists, and compare the performance of our model to that of 3 additional radiologists in the detection of 5 selected pathologies. On Cardiomegaly, Edema, and Pleural Effusion, the model ROC and PR curves lie above all 3 radiologist operating points. We release the dataset to the public as a standard benchmark to evaluate performance of chest radiograph interpretation models. The dataset is freely available at https://stanfordmlgroup.github.io/competitions/chexpert .

COVID-19 Image Data Collection

Abstract: This paper describes the initial COVID-19 open image data collection. It was created by assembling medical images from websites and publications and currently contains 123 frontal view X-rays.

An overview of COVID -ve penumonia datasets used for our analysis.

Analysis of Sources for COVID Negative Pneumonia Dataset

COVID-Net {{cite:74f7450}} has been widely cited for COVID-19 X-Ray classification. The arXiv version appeared in late March. Much of the literature followed the dataset used in COVID-Net. COVID-Net used Cohen's repository {{cite:243c54c}}, {{cite:c4650c9}} for COVID +ve samples. Negative samples are from the pneumonia dataset by Kermany et al. (popularly known as Paul Timothy Mooney's Kaggle Pneumonia dataset) {{cite:f1ff7ba}}. It is important to note that in the current version of that paper (version 4 at the time of writing), the authors have completely changed the dataset they used originally. They have now compiled the dataset from a variety of sources. A recent review paper {{cite:1df816d}} surveys highly cited papers on CXR based COVID-19 diagnosis. Three out of four techniques mentioned in {{cite:1df816d}}, viz. {{cite:19f05dc}}, {{cite:83d897c}}, {{cite:74f7450}}, use Kermany's dataset for COVID -ve samples. Since these three papers are highly cited, many researchers have used the same datasets to carry out their experiments. For our study, we ran baselines on the same dataset (Cohen's dataset for COVID +ve and Kermany's dataset for COVID -ve) to diagnose COVID positivity from CXR images. We found that Mobilenet v2 {{cite:36f11d6}} pretrained on ImageNet {{cite:10449f7}} yielded nearly 100% accuracy on this dataset. Closer examination using t-SNE reveals that samples in the Kermany's dataset are significantly different from the samples in the Cohen's dataset, and hence it is almost trivial to determine which source an image belongs to, by visual inspection alone. Fig. {{figure:df0914cf-a764-434e-a47e-3ffe098001a1}}  contains a t-SNE plot {{cite:a1d74f7}} of samples from Kermany's and Cohen's datasets. It is evident that the two datasets form distinct clusters and are easily seperable. Therefore, training for COVID positivity using their combination would be unwise. We highlight this issue by training a classifier on this dataset, and testing it on samples from other sources. This analysis raises the question of selecting an appropriate dataset for evaluating approaches to CXR based COVID-19 diagnosis. We therefore compiled pneumonic CXR images of COVID negative subjects from a diversity of sources, viz. CheXpert {{cite:2e0762f}}, Kermany {{cite:f1ff7ba}}, Cohen {{cite:243c54c}}, {{cite:c4650c9}}, NIH {{cite:e2fc2e2}} and Open-i {{cite:1089dec}}. Throughout this paper, we focus on the task of differentiating COVID +ve CXRs from pneumonic COVID -ve CXRs.
Table {{table:58405ebd-84c4-4212-a56b-90dee87bb605}}  provides an overview.


Benchmark for license plate character segmentation

Abstract: Abstract. Automatic license plate recognition (ALPR) has been the focus of many researches in the past years. In general, ALPR is divided into the following problems: detection of on-track vehicles, license plate detection, segmentation of license plate characters, and optical character recognition (OCR). Even though commercial solutions are available for controlled acquisition conditions, e.g., the entrance of a parking lot, ALPR is still an open problem when dealing with data acquired from uncontrolled environments, such as roads and highways when relying only on imaging sensors. Due to the multiple orientations and scales of the license plates captured by the camera, a very challenging task of the ALPR is the license plate character segmentation (LPCS) step, because its effectiveness is required to be (near) optimal to achieve a high recognition rate by the OCR. To tackle the LPCS problem, this work proposes a benchmark composed of a dataset designed to focus specifically on the character segmentation step of the ALPR within an evaluation protocol. Furthermore, we propose the Jaccard-centroid coefficient, an evaluation measure more suitable than the Jaccard coefficient regarding the location of the bounding box within the ground-truth annotation. The dataset is composed of 2000 Brazilian license plates consisting of 14000 alphanumeric symbols and their corresponding bounding box annotations. We also present a straightforward approach to perform LPCS efficiently. Finally, we provide an experimental evaluation for the dataset based on five LPCS approaches and demonstrate the importance of character segmentation for achieving an accurate OCR.

A Robust Real-Time Automatic License Plate Recognition Based on the YOLO Detector

Abstract: Automatic License Plate Recognition (ALPR) has been a frequent topic of research due to many practical applications. However, many of the current solutions are still not robust in real-world situations, commonly depending on many constraints. This paper presents a robust and efficient ALPR system based on the state-of-the-art YOLO object detector. The Convolutional Neural Networks (CNNs) are trained and fine-tuned for each ALPR stage so that they are robust under different conditions (e.g., variations in camera, lighting, and background). Specially for character segmentation and recognition, we design a two-stage approach employing simple data augmentation tricks such as inverted License Plates (LPs) and flipped characters. The resulting ALPR approach achieved impressive results in two datasets. First, in the SSIG dataset, composed of 2,000 frames from 101 vehicle videos, our system achieved a recognition rate of 93.53% and 47 Frames Per Second (FPS), performing better than both Sighthound and OpenALPR commercial systems (89.80% and 93.03%, respectively) and considerably outperforming previous results (81.80%). Second, targeting a more realistic scenario, we introduce a larger public dataset, called UFPR-ALPR dataset, designed to ALPR. This dataset contains 150 videos and 4,500 frames captured when both camera and vehicles are moving and also contains different types of vehicles (cars, motorcycles, buses and trucks). In our proposed dataset, the trial versions of commercial systems achieved recognition rates below 70%. On the other hand, our system performed better, with recognition rate of 78.33% and 35 FPS.

CNN for License Plate Motion Deblurring

Abstract: In this work we explore the previously proposed approach of direct blind deconvolution and denoising with convolutional neural networks in a situation where the blur kernels are partially constrained. We focus on blurred images from a real-life traffic surveillance system, on which we, for the first time, demonstrate that neural networks trained on artificial data provide superior reconstruction quality on real images compared to traditional blind deconvolution methods. The training data is easy to obtain by blurring sharp photos from a target system with a very rough approximation of the expected blur kernels, thereby allowing custom CNNs to be trained for a specific application (image content and blur range). Additionally, we evaluate the behavior and limits of the CNNs with respect to blur direction range and length.

Iranis: A Large-scale Dataset of Iranian Vehicles License Plate Characters

Abstract: Providing huge amounts of data is a fundamental demand when dealing with Deep Neural Networks (DNNs). Employing these algorithms to solve computer vision problems resulted in the advent of various image datasets to feed the most common visual imagery deep structures, known as Convolutional Neural Networks (CNNs). In this regard, some datasets can be found that contain hundreds or even thousands of images for license plate detection and optical character recognition purposes. However, no publicly available image dataset provides such data for the recognition of Farsi characters used in car license plates. The gap has to be filled due to the numerous advantages of developing accurate deep learning-based systems for law enforcement and surveillance purposes. This paper introduces a large-scale dataset that includes images of numbers and characters used in Iranian car license plates. The dataset, named Iranis, contains more than 83,000 images of Farsi numbers and letters collected from real-world license plate images captured by various cameras. The variety of instances in terms of camera shooting angle, illumination, resolution, and contrast make the dataset a proper choice for training DNNs. Dataset images are manually annotated for object detection and image classification. Finally, and to build a baseline for Farsi character recognition, the paper provides a performance analysis using a YOLO v.3 object detector.

Overview over the most common data sets related to ALPR

Table {{table:138f20a6-57dc-4e6a-84d6-0f33eaedd55a}}  gives an overview over the most common data sets related to ALPR. Most of these data sets holding annotated license plate data are too small to train large NNs, as this requires several hundreds of thousand training samples. The largest data sets listed in Table {{table:138f20a6-57dc-4e6a-84d6-0f33eaedd55a}}  hold between 140,000 and 250,000 samples of Czech or Chinese license plates, respectively. Moreover, common data sets hold mostly license plates of good visual quality, as the camera acquiring the license plate samples is oriented directly towards the license plates. For forensic ALPR applications this does not match the common use case, as here often license plates in the background of an image are of special interest. Those samples are often very small, suffer from bad illumination and show an overall bad visual quality. Therefore, existing license plate data sets often lead to insufficient results in forensic ALPR applications.

Compared to the synthetic and partly-real data set, the real data set seems rather small. However, when compared to other state-of-the-art databases as given in Table {{table:138f20a6-57dc-4e6a-84d6-0f33eaedd55a}}  it can be observed that its size is similar or even larger than other common data sets. For example the size of the commonly used AOLPE or PKU Vehicle data set is less than half of our real data set. The ground-truth annotation is identically structured for all three types of data. It does not contain any blanks. The region identifier is separated from the individual identifier by a dash representing the larger margin between the first and second text block.

The common approach to alleviate this problem is to train the OCR application from scratch by using a specific data set of license plates from the region of interest that includes similar artifacts. Due to restrictions regarding data protection and the elaborate labeling process, data sets holding real world training data are very limited. The size of the real data set produced with our proposed rendering-based data generation is comparable to common real-world license plate data sets, as denoted in Table {{table:138f20a6-57dc-4e6a-84d6-0f33eaedd55a}} . Using our generated real data set, we trained the Tesseract OCR application in its version 4.1.1 for three epochs. Thereby, a CER of {{formula:6b57ecb9-8b11-42ee-a4df-eb08ec9ecdeb}}  and a MR of {{formula:09784297-70df-4801-9f27-c678f2958cda}}  is obtained on the independent real-world test set. The very high CER and MR indicate that the recognition scenario is too challenging for an algorithm trained directly on the real data set. Moreover, this shows that the scenario of forensic license plate recognition is more challenging than license plate recognition for civil applications such as access control where the considered samples hold notably less artifacts. The results suggests that the real data set is too small to train the algorithm on it solely, as none of the license plates is recognized correctly and the large majority of characters is recognized falsely, as well.


Use TEEs for confidential CPU operations and masked data for confidential GPU operations

Apply differential privacy to randomize intermediate gradients

Applies differential privacy to hide sensitive pixels in images

Mix-up images from the training set and public domain with random selection and weighing to hide the content of the sensitive image.

side channel attacks are an unaddressed concern

Aim to share trained models, and thus does not preserve model confidentiality; result in significant drop in model quality; vulnerable to generative adversarial network attacks.

Does not hide the global visual characteristic content of images, vulnerable to visual re-identification attacks

Vulnerable to ciphertext-only (Level 1 adversarial knowledge) image re-construction attacks; May expose trained model to wide variety of model-based and membership attacks for the inside-dataset setting.

More efficient than cryptographic protocols, but GPU operations on masked data are still expensive

Fast training; No changes to the training architecture.

Slalom: Fast, Verifiable and Private Execution of Neural Networks in Trusted Hardware

Abstract: As Machine Learning (ML) gets applied to security-critical or sensitive domains, there is a growing need for integrity and privacy for outsourced ML computations. A pragmatic solution comes from Trusted Execution Environments (TEEs), which use hardware and software protections to isolate sensitive computations from the untrusted software stack. However, these isolation guarantees come at a price in performance, compared to untrusted alternatives. This paper initiates the study of high performance execution of Deep Neural Networks (DNNs) in TEEs by efficiently partitioning DNN computations between trusted and untrusted devices. Building upon an efficient outsourcing scheme for matrix multiplication, we propose Slalom, a framework that securely delegates execution of all linear layers in a DNN from a TEE (e.g., Intel SGX or Sanctum) to a faster, yet untrusted, co-located processor. We evaluate Slalom by running DNNs in an Intel SGX enclave, which selectively delegates work to an untrusted GPU. For canonical DNNs (VGG16, MobileNet and ResNet variants) we obtain 6x to 20x increases in throughput for verifiable inference, and 4x to 11x for verifiable and private inference.

Deep Learning with Differential Privacy

Abstract: Machine learning techniques based on neural networks are achieving remarkable results in a wide variety of domains. Often, the training of models requires large, representative datasets, which may be crowdsourced and contain sensitive information. The models should not expose private information in these datasets. Addressing this goal, we develop new algorithmic techniques for learning and a refined analysis of privacy costs within the framework of differential privacy. Our implementation and experiments demonstrate that we can train deep neural networks with non-convex objectives, under a modest privacy budget, and at a manageable cost in software complexity, training efficiency, and model quality.

PrivyNet: A Flexible Framework for Privacy-Preserving Deep Neural Network Training

Abstract: Massive data exist among user local platforms that usually cannot support deep neural network (DNN) training due to computation and storage resource constraints. Cloud-based training schemes provide beneficial services but suffer from potential privacy risks due to excessive user data collection. To enable cloud-based DNN training while protecting the data privacy simultaneously, we propose to leverage the intermediate representations of the data, which is achieved by splitting the DNNs and deploying them separately onto local platforms and the cloud. The local neural network (NN) is used to generate the feature representations. To avoid local training and protect data privacy, the local NN is derived from pre-trained NNs. The cloud NN is then trained based on the extracted intermediate representations for the target learning task. We validate the idea of DNN splitting by characterizing the dependency of privacy loss and classification accuracy on the local NN topology for a convolutional NN (CNN) based image classification task. Based on the characterization, we further propose PrivyNet to determine the local NN topology, which optimizes the accuracy of the target learning task under the constraints on privacy loss, local computation, and storage. The efficiency and effectiveness of PrivyNet are demonstrated with the CIFAR-10 dataset.

Visual Interpretability for Deep Learning: a Survey

Abstract: This paper reviews recent studies in understanding neural-network representations and learning neural networks with interpretable/disentangled middle-layer representations. Although deep neural networks have exhibited superior performance in various tasks, the interpretability is always the Achilles' heel of deep neural networks. At present, deep neural networks obtain high discrimination power at the cost of low interpretability of their black-box representations. We believe that high model interpretability may help people to break several bottlenecks of deep learning, e.g., learning from very few annotations, learning via human-computer communications at the semantic level, and semantically debugging network representations. We focus on convolutional neural networks (CNNs), and we revisit the visualization of CNN representations, methods of diagnosing representations of pre-trained CNNs, approaches for disentangling pre-trained CNN representations, learning of CNNs with disentangled representations, and middle-to-end learning based on model interpretability. Finally, we discuss prospective trends in explainable artificial intelligence.

Related work on training phase protection.

Thus, on the one end, existing disguising mechanisms are too weak to protect almost nothing. On the other end, if an encryption mechanism, e.g., homomorphic encryption, or a complex cryptographic protocol, is applied, such linking or reconstruction would be impossible. However, the current cryptographic schemes incur extremely high costs in almost all aspects of computation, communication, and storage. Thus, they are impractical for resource-intensive tasks like training a DNN model. Hardware-assisted approaches, such as TEEs, are still under investigation to ensure the expected security properties. Along with all these possible approaches, we aim to explore and develop new image-disguising methods to achieve good balances among costs, data utility, and security guarantees. Table {{table:8fa7468d-7416-4066-a1ed-b9196415caa0}}  summarizes the current work in protecting data and models in the training phase.


Bayesian networks {{cite:e69ded2}}, {{cite:7c4026b}}Maximum Entropy {{cite:4546ea1}}, {{cite:947ab67}}

k-NN {{cite:e2adf50}} Neural networks {{cite:e2adf50}}

Multi-dimensional classification of biomedical text: Toward automated, practical provision of high-utility text to diverse users

Abstract: Motivation: Much current research in biomedical text mining is concerned with serving biologists by extracting certain information from scientific text. We note that there is no ‘average biologist’ client; different users have distinct needs. For instance, as noted in past evaluation efforts (BioCreative, TREC, KDD) database curators are often interested in sentences showing experimental evidence and methods. Conversely, lab scientists searching for known information about a protein may seek facts, typically stated with high confidence. Text-mining systems can target specific end-users and become more effective, if the system can first identify text regions rich in the type of scientific content that is of interest to the user, retrieve documents that have many such regions, and focus on fact extraction from these regions. Here, we study the ability to characterize and classify such text automatically. We have recently introduced a multi-dimensional categorization and annotation scheme, developed to be applicable to a wide variety of biomedical documents and scientific statements, while intended to support specific biomedical retrieval and extraction tasks. Results: The annotation scheme was applied to a large corpus in a controlled effort by eight independent annotators, where three individual annotators independently tagged each sentence. We then trained and tested machine learning classifiers to automatically categorize sentence fragments based on the annotation. We discuss here the issues involved in this task, and present an overview of the results. The latter strongly suggest that automatic annotation along most of the dimensions is highly feasible, and that this new framework for scientific sentence categorization is applicable in practice. Contact: shatkay@cs.queensu.ca

Label Distribution Learning

Abstract: Although multi-label learning can deal with many problems with label ambiguity, it does not fit some real applications well where the overall distribution of the importance of the labels matters. This paper proposes a novel learning paradigm named label distribution learning (LDL) for such kind of applications. The label distribution covers a certain number of labels, representing the degree to which each label describes the instance. LDL is a more general learning framework which includes both single-label and multi-label learning as its special cases. This paper proposes six LDL algorithms in three ways: problem transformation, algorithm adaptation, and specialized algorithm design. In order to compare their performance, six evaluation measures are suggested for LDL algorithms, and the first batch of real-world label distribution datasets are prepared. Experimental results on the ten real-world datasets show clear advantage of the specialized algorithms, which indicates the importance of special design for the characteristics of the LDL problem.

Summary table of presented methods according to their type of approach.

Table {{table:b12a648d-a0da-47e6-b807-d3b6a647e200}}  gathers all the methods described previously to tackle nonstandard supervised tasks.


MSCOCO, SBU, LN, CC, CC12, VG, WIT, RC, YFCC100M

FLAVA: A Foundational Language And Vision Alignment Model

Abstract: State-of-the-art vision and vision-and-language models rely on large-scale visio-linguistic pretraining for obtaining good performance on a variety of downstream tasks. Generally, such models are often either cross-modal (contrastive) or multi-modal (with earlier fusion) but not both; and they often only target specific modalities or tasks. A promising direction would be to use a single holistic universal model, as a “foundation”, that targets all modalities at once-a true vision and language foundation model should be good at vision tasks, language tasks, and cross- and multi-modal vision and language tasks. We introduce FLAVA as such a model and demonstrate impressive performance on a wide range of 35 tasks spanning these target modalities.

ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision

Abstract: Vision-and-Language Pre-training (VLP) has improved performance on various joint vision-and-language downstream tasks. Current approaches to VLP heavily rely on image feature extraction processes, most of which involve region supervision (e.g., object detection) and the convolutional architecture (e.g., ResNet). Although disregarded in the literature, we find it problematic in terms of both (1) efficiency/speed, that simply extracting input features requires much more computation than the multimodal interaction steps; and (2) expressive power, as it is upper bounded to the expressive power of the visual embedder and its predefined visual vocabulary. In this paper, we present a minimal VLP model, Vision-and-Language Transformer (ViLT), monolithic in the sense that the processing of visual inputs is drastically simplified to just the same convolution-free manner that we process textual inputs. We show that ViLT is up to tens of times faster than previous VLP models, yet with competitive or better downstream task performance. Our code and pre-trained weights are available at https://github.com/dandelin/vilt.

Bridge-Tower: Building Bridges Between Encoders in Vision-Language Representation Learning

Abstract: Vision-Language (VL) models with the Two-Tower architecture have dominated visual-language representation learning in recent years. Current VL models either use lightweight uni-modal encoders and learn to extract, align and fuse both modalities simultaneously in a deep cross-modal encoder, or feed the last-layer uni-modal representations from the deep pre-trained uni-modal encoders into the top cross-modal encoder. Both approaches potentially restrict vision-language representation learning and limit model performance. In this paper, we propose BridgeTower, which introduces multiple bridge layers that build a connection between the top layers of uni-modal encoders and each layer of the cross-modal encoder. This enables effective bottom-up cross-modal alignment and fusion between visual and textual representations of different semantic levels of pre-trained uni-modal encoders in the cross-modal encoder. Pre-trained with only 4M images, BridgeTower achieves state-of-the-art performance on various downstream vision-language tasks. In particular, on the VQAv2 test-std set, BridgeTower achieves an accuracy of 78.73%, outperforming the previous state-of-the-art model METER by 1.09% with the same pre-training data and almost negligible additional parameters and computational costs. Notably, when further scaling the model, BridgeTower achieves an accuracy of 81.15%, surpassing models that are pre-trained on orders-of-magnitude larger datasets. Code and checkpoints are available at https://github.com/microsoft/BridgeTower.

The pre-training datasets include MSCOCO {{cite:899a474}}, SBU Captions, Localized Narratives (LN), Visual Genome (VG) {{cite:1dbede0}}, Wikipedia Image Text (WIT) {{cite:6e11126}}, Conceptual Captions (CC) {{cite:650616c}}, Conceptual Captions 12M (CC12) {{cite:d5c004c}}, Red Caps (RC) {{cite:73b023a}}, YFCC100M {{cite:0602045}}, and LAION-400M {{cite:52d42c5}}.

A summary of the model details can be found in Table {{table:d979fa4d-0ede-4aac-99e4-1e25cbf67c51}} . The highest performing model is BridgeTower but it also had the largest number of parameters and the slowest. Additionally, BridgeTower utilizes a pre-trained CLIP visual encoder, improving upon CLIPs performance. ViLT, BridgeTower and FLAVA all require image-text pairs, making a greater number of comparisons difficult, especially for downstream tasks like image classification on ImageNet where there are 1000 classes. However, because FLAVA merges dual-stream encoder output prior to cross-encoding, it is easier to extract feature embeddings prior to the cross-encoding for a greater number of comparisons. This however does not utilize its full potential for performance. Figure {{figure:7e6c9167-44be-41ba-9bed-434e23b25a73}}  shows examples of how this image-text pair input is a strength for performance in these kinds of tasks. The bottom shows ViLT and how its visual attention changes based on its input while the top shows CLIP which has consistent attention no matter the text, visual input.


Zoom Better to See Clearer: Human and Object Parsing with Hierarchical Auto-Zoom Net

Abstract: Parsing articulated objects, e.g. humans and animals, into semantic parts (e.g. body, head and arms, etc.) from natural images is a challenging and fundamental problem for computer vision. A big difficulty is the large variability of scale and location for objects and their corresponding parts. Even limited mistakes in estimating scale and location will degrade the parsing output and cause errors in boundary details. To tackle these difficulties, we propose a "Hierarchical Auto-Zoom Net" (HAZN) for object part parsing which adapts to the local scales of objects and parts. HAZN is a sequence of two "Auto-Zoom Net" (AZNs), each employing fully convolutional networks that perform two tasks: (1) predict the locations and scales of object instances (the first AZN) or their parts (the second AZN); (2) estimate the part scores for predicted object instance or part regions. Our model can adaptively "zoom" (resize) predicted image regions into their proper scales to refine the parsing. We conduct extensive experiments over the PASCAL part datasets on humans, horses, and cows. For humans, our approach significantly outperforms the state-of-the-arts by 5% mIOU and is especially better at segmenting small instances and small parts. We obtain similar improvements for parsing cows and horses over alternative methods. In summary, our strategy of first zooming into objects and then zooming into parts is very effective. It also enables us to process different regions of the image at different scales adaptively so that, for example, we do not need to waste computational resources scaling the entire image.

Look into Person: Self-supervised Structure-sensitive Learning and A New Benchmark for Human Parsing

Abstract: Human parsing has recently attracted a lot of research interests due to its huge application potentials. However existing datasets have limited number of images and annotations, and lack the variety of human appearances and the coverage of challenging cases in unconstrained environment. In this paper, we introduce a new benchmark "Look into Person (LIP)" that makes a significant advance in terms of scalability, diversity and difficulty, a contribution that we feel is crucial for future developments in human-centric analysis. This comprehensive dataset contains over 50,000 elaborately annotated images with 19 semantic part labels, which are captured from a wider range of viewpoints, occlusions and background complexity. Given these rich annotations we perform detailed analyses of the leading human parsing approaches, gaining insights into the success and failures of these methods. Furthermore, in contrast to the existing efforts on improving the feature discriminative capability, we solve human parsing by exploring a novel self-supervised structure-sensitive learning approach, which imposes human pose structures into parsing results without resorting to extra supervision (i.e., no need for specifically labeling human joints in model training). Our self-supervised learning framework can be injected into any advanced neural networks to help incorporate rich high-level knowledge regarding human joints from a global perspective and improve the parsing results. Extensive evaluations on our LIP and the public PASCAL-Person-Part dataset demonstrate the superiority of our method.

Interpretable Structure-Evolving LSTM

Abstract: This paper develops a general framework for learning interpretable data representation via Long Short-Term Memory (LSTM) recurrent neural networks over hierarchal graph structures. Instead of learning LSTM models over the pre-fixed structures, we propose to further learn the intermediate interpretable multi-level graph structures in a progressive and stochastic way from data during the LSTM network optimization. We thus call this model the structure-evolving LSTM. In particular, starting with an initial element-level graph representation where each node is a small data element, the structure-evolving LSTM gradually evolves the multi-level graph representations by stochastically merging the graph nodes with high compatibilities along the stacked LSTM layers. In each LSTM layer, we estimate the compatibility of two connected nodes from their corresponding LSTM gate outputs, which is used to generate a merging probability. The candidate graph structures are accordingly generated where the nodes are grouped into cliques with their merging probabilities. We then produce the new graph structure with a Metropolis-Hasting algorithm, which alleviates the risk of getting stuck in local optimums by stochastic sampling with an acceptance probability. Once a graph structure is accepted, a higher-level graph is then constructed by taking the partitioned cliques as its nodes. During the evolving process, representation becomes more abstracted in higher-levels where redundant information is filtered out, allowing more efficient propagation of long-range data dependencies. We evaluate the effectiveness of structure-evolving LSTM in the application of semantic object parsing and demonstrate its advantage over state-of-the-art LSTM models on standard benchmarks.

Joint Multi-Person Pose Estimation and Semantic Part Segmentation

Abstract: Human pose estimation and semantic part segmentation are two complementary tasks in computer vision. In this paper, we propose to solve the two tasks jointly for natural multi-person images, in which the estimated pose provides object-level shape prior to regularize part segments while the part-level segments constrain the variation of pose locations. Specifically, we first train two fully convolutional neural networks (FCNs), namely Pose FCN and Part FCN, to provide initial estimation of pose joint potential and semantic part potential. Then, to refine pose joint location, the two types of potentials are fused with a fully-connected conditional random field (FCRF), where a novel segment-joint smoothness term is used to encourage semantic and spatial consistency between parts and joints. To refine part segments, the refined pose and the original part potential are integrated through a Part FCN, where the skeleton feature from pose serves as additional regularization cues for part segments. Finally, to reduce the complexity of the FCRF, we induce human detection boxes and infer the graph inside each box, making the inference forty times faster. Since there's no dataset that contains both part segments and pose labels, we extend the PASCAL VOC part dataset with human pose joints and perform extensive experiments to compare our method against several most recent strategies. We show that on this dataset our algorithm surpasses competing methods by a large margin in both tasks.

Holistic, Instance-Level Human Parsing

Abstract: Object parsing -- the task of decomposing an object into its semantic parts -- has traditionally been formulated as a category-level segmentation problem. Consequently, when there are multiple objects in an image, current methods cannot count the number of objects in the scene, nor can they determine which part belongs to which object. We address this problem by segmenting the parts of objects at an instance-level, such that each pixel in the image is assigned a part label, as well as the identity of the object it belongs to. Moreover, we show how this approach benefits us in obtaining segmentations at coarser granularities as well. Our proposed network is trained end-to-end given detections, and begins with a category-level segmentation module. Thereafter, a differentiable Conditional Random Field, defined over a variable number of instances for every input image, reasons about the identity of each part by associating it with a human detection. In contrast to other approaches, our method can handle the varying number of people in each image and our holistic network produces state-of-the-art results in instance-level part and human segmentation, together with competitive results in category-level part segmentation, all achieved by a single forward-pass through our neural network.

Weakly and Semi Supervised Human Body Part Parsing via Pose-Guided Knowledge Transfer

Abstract: Human body part parsing, or human semantic part segmentation, is fundamental to many computer vision tasks. In conventional semantic segmentation methods, the ground truth segmentations are provided, and fully convolutional networks (FCN) are trained in an end-to-end scheme. Although these methods have demonstrated impressive results, their performance highly depends on the quantity and quality of training data. In this paper, we present a novel method to generate synthetic human part segmentation data using easily-obtained human keypoint annotations. Our key idea is to exploit the anatomical similarity among human to transfer the parsing results of a person to another person with similar pose. Using these estimated results as additional training data, our semi-supervised model outperforms its strong-supervised counterpart by 6 mIOU on the PASCAL-Person-Part dataset [6], and we achieve state-of-the-art human parsing results. Our approach is general and can be readily extended to other object/animal parsing task assuming that their anatomical similarity can be annotated by keypoints. The proposed model and accompanying source code will be made publicly available.

SPGNet: Semantic Prediction Guidance for Scene Parsing

Abstract: Multi-scale context module and single-stage encoder-decoder structure are commonly employed for semantic segmentation. The multi-scale context module refers to the operations to aggregate feature responses from a large spatial extent, while the single-stage encoder-decoder structure encodes the high-level semantic information in the encoder path and recovers the boundary information in the decoder path. In contrast, multi-stage encoder-decoder networks have been widely used in human pose estimation and show superior performance than their single-stage counterpart. However, few efforts have been attempted to bring this effective design to semantic segmentation. In this work, we propose a Semantic Prediction Guidance (SPG) module which learns to re-weight the local features through the guidance from pixel-wise semantic prediction. We find that by carefully re-weighting features across stages, a two-stage encoder-decoder network coupled with our proposed SPG module can significantly outperform its one-stage counterpart with similar parameters and computations. Finally, we report experimental results on the semantic segmentation benchmark Cityscapes, in which our SPGNet attains 81.1% on the test set using only 'fine' annotations.

RefineNet: Multi-path Refinement Networks for High-Resolution Semantic Segmentation

Abstract: Recently, very deep convolutional neural networks (CNNs) have shown outstanding performance in object recognition and have also been the first choice for dense classification problems such as semantic segmentation. However, repeated subsampling operations like pooling or convolution striding in deep CNNs lead to a significant decrease in the initial image resolution. Here, we present RefineNet, a generic multi-path refinement network that explicitly exploits all the information available along the down-sampling process to enable high-resolution prediction using long-range residual connections. In this way, the deeper layers that capture high-level semantic features can be directly refined using fine-grained features from earlier convolutions. The individual components of RefineNet employ residual connections following the identity mapping mindset, which allows for effective end-to-end training. Further, we introduce chained residual pooling, which captures rich background context in an efficient manner. We carry out comprehensive experiments and set new state-of-the-art results on seven public datasets. In particular, we achieve an intersection-over-union score of 83.4 on the challenging PASCAL VOC 2012 dataset, which is the best reported result to date.

Learning Compositional Neural Information Fusion for Human Parsing

Abstract: This work proposes to combine neural networks with the compositional hierarchy of human bodies for efficient and complete human parsing. We formulate the approach as a neural information fusion framework. Our model assembles the information from three inference processes over the hierarchy: direct inference (directly predicting each part of a human body using image information), bottom-up inference (assembling knowledge from constituent parts), and top-down inference (leveraging context from parent nodes). The bottom-up and top-down inferences explicitly model the compositional and decompositional relations in human bodies, respectively. In addition, the fusion of multi-source information is conditioned on the inputs, i.e., by estimating and considering the confidence of the sources. The whole model is end-to-end differentiable, explicitly modeling information flows and structures. Our approach is extensively evaluated on four popular datasets, outperforming the state-of-the-arts in all cases, with a fast processing speed of 23fps. Our code and results have been released to help ease future research in this direction.

Graphonomy: Universal Human Parsing via Graph Transfer Learning

Abstract: Prior highly-tuned human parsing models tend to fit towards each dataset in a specific domain or with discrepant label granularity, and can hardly be adapted to other human parsing tasks without extensive re-training. In this paper, we aim to learn a single universal human parsing model that can tackle all kinds of human parsing needs by unifying label annotations from different domains or at various levels of granularity. This poses many fundamental learning challenges, e.g. discovering underlying semantic structures among different label granularity, performing proper transfer learning across different image domains, and identifying and utilizing label redundancies across related tasks. To address these challenges, we propose a new universal human parsing agent, named "Graphonomy", which incorporates hierarchical graph transfer learning upon the conventional parsing network to encode the underlying label semantic structures and propagate relevant semantic information. In particular, Graphonomy first learns and propagates compact high-level graph representation among the labels within one dataset via Intra-Graph Reasoning, and then transfers semantic information across multiple datasets via Inter-Graph Transfer. Various graph transfer dependencies (\eg, similarity, linguistic knowledge) between different datasets are analyzed and encoded to enhance graph transfer capability. By distilling universal semantic graph representation to each specific task, Graphonomy is able to predict all levels of parsing labels in one system without piling up the complexity. Experimental results show Graphonomy effectively achieves the state-of-the-art results on three human parsing benchmarks as well as advantageous universal human parsing performance.

Searching for Efficient Multi-Scale Architectures for Dense Image Prediction

Abstract: The design of neural network architectures is an important component for achieving state-of-the-art performance with machine learning systems across a broad array of tasks. Much work has endeavored to design and build architectures automatically through clever construction of a search space paired with simple learning algorithms. Recent progress has demonstrated that such meta-learning methods may exceed scalable human-invented architectures on image classification tasks. An open question is the degree to which such methods may generalize to new domains. In this work we explore the construction of meta-learning techniques for dense image prediction focused on the tasks of scene parsing, person-part segmentation, and semantic image segmentation. Constructing viable search spaces in this domain is challenging because of the multi-scale representation of visual information and the necessity to operate on high resolution imagery. Based on a survey of techniques in dense image prediction, we construct a recursive search space and demonstrate that even with efficient random search, we can identify architectures that outperform human-invented architectures and achieve state-of-the-art performance on three dense prediction tasks including 82.7% on Cityscapes (street scene parsing), 71.3% on PASCAL-Person-Part (person-part segmentation), and 87.9% on PASCAL VOC 2012 (semantic image segmentation). Additionally, the resulting architecture is more computationally efficient, requiring half the parameters and half the computational cost as previous state of the art systems.

Quantitative results on PASCAL-Person-Part test in terms of mean pixel Intersection-over-Union (mIoU) (%)

In this section, we outline the performance of several of the previous methods on popular segmentation benchmarks. Since some models are not reported their performance on standard datasets, it is hard to make a full comparison. The following tables summarize the performances of several of the prominent models on different datasets. Table {{table:f7de7341-a0e6-4f62-938b-a63c45ab7878}}  focuses on the PASCAL-Person-Part test set. Since the introduction of FCN-based models, the performance has been greatly improved. Table {{table:631a904f-0766-4e4b-82b1-4c78fb58d104}}  focuses on the LIP validation. On this dataset, the latest model improved by 32.01% in terms of mIoU over the original model. Table {{table:d936b018-501f-4a00-aeff-680c54e314e1}}  focuses on the CIHP test set. This dataset is more challenging than the PASCAL-Person-Part dataset, both two datasets are multi-person. Table {{table:ab1df25e-0982-4318-abfe-1fb26582083c}}  focuses on the ATR test set. The latest model achieves 87.16 in terms of F-1. Finally, Table {{table:af4d1e0a-dcc5-4a1d-9ad2-5141d7748284}}  summarizes the performance of several models for the Fashion Clothing dataset.


Exploring Plausible Patches Using Source Code Embeddings in JavaScript

Abstract: Despite the immense popularity of the Automated Program Repair (APR) field, the question of patch validation is still open. Most of the present-day approaches follow the so-called Generate-and-Validate approach, where first a candidate solution is being generated and after validated against an oracle. The latter, however, might not give a reliable result, because of the imperfections in such oracles; one of which is usually the test suite. Although (re-) running the test suite is right under one's nose, in real life applications the problem of over- and underfitting often occurs, resulting in inadequate patches. Efforts that have been made to tackle with this problem include patch filtering, test suite expansion, careful patch producing and many more. Most approaches to date use post-filtering relying either on test execution traces or make use of some similarity concept measured on the generated patches. Our goal is to investigate the nature of these similarity-based approaches. To do so, we trained a Doc2Vec model on an open-source JavaScript project and generated 465 patches for 10 bugs in it. These plausible patches alongside with the developer fix are then ranked based on their similarity to the original program. We analyzed these similarity lists and found that plain document embeddings may lead to misclassification - it fails to capture nuanced code semantics. Nevertheless, in some cases it also provided useful information, thus helping to better understand the area of Automated Program Repair.

Predicting Patch Correctness Based on the Similarity of Failing Test Cases

Abstract: Towards predicting patch correctness in APR, we propose a simple, but novel hypothesis on how the link between the patch behaviour and failing test specifications can be drawn: similar failing test cases should require similar patches. We then propose BATS, an unsupervised learning-based system to predict patch correctness by checking patch Behaviour Against failing Test Specification. BATS exploits deep representation learning models for code and patches: for a given failing test case, the yielded embedding is used to compute similarity metrics in the search for historical similar test cases in order to identify the associated applied patches, which are then used as a proxy for assessing generated patch correctness. Experimentally, we first validate our hypothesis by assessing whether ground-truth developer patches cluster together in the same way that their associated failing test cases are clustered. Then, after collecting a large dataset of 1278 plausible patches (written by developers or generated by some 32 APR tools), we use BATS to predict correctness: BATS achieves an AUC between 0.557 to 0.718 and a recall between 0.562 and 0.854 in identifying correct patches. Compared against previous work, we demonstrate that our approach outperforms state-of-the-art performance in patch correctness prediction, without the need for large labeled patch datasets in contrast with prior machine learning-based approaches. While BATS is constrained by the availability of similar test cases, we show that it can still be complementary to existing approaches: used in conjunction with a recent approach implementing supervised learning, BATS improves the overall recall in detecting correct patches. We finally show that BATS can be complementary to the state-of-the-art PATCH-SIM dynamic approach of identifying the correct patches for APR tools.

A summary and comparison of learning-based APCA studies

Table {{table:08c0e487-4df8-467e-b952-8c43cead7732}}  presents a summary of existing learning-based techniques to predict patch correctness automatically in the literature.
The first and second columns list the APCA technique and the time of publication.
The third column lists the targeted programming languages.
The fourth column lists the features adopted by the APCA technique.
The remaining columns list the employed datasets and the public repositories.

Now, we discuss and summarize these individual approaches as follows.

[colback=gray!15, colframe=black, boxsep=-0.15cm, middle=-0.15cm,breakable]
✎ Summary
{{formula:311a13ab-625d-4b26-a3b3-71161b45d568}}

The overfitting issue has become a key focus in the field of APR, which has led to the emergence and rapid development of recent APCA techniques.
DL techniques have been gradually used to predict the correctness of patches by learning features from historical corpora.
Compared to traditional dynamic and static APCA, learning-based APCA has shown impressive performance in prediction accuracy and recall.
We provide a summary of the existing learning-based APCA techniques in Table {{table:08c0e487-4df8-467e-b952-8c43cead7732}} .
In the literature, most existing APCA techniques employ a two-component pipeline, i.e., the feature extractor and the classifier.
The former extracts the features from the source code of patches, e.g., hand-crafted features, static representation features and dynamic execution features, while the latter trains a classifier to perform binary prediction, e.g., Random Forest and Decision tree.
Despite increasing research efforts being put into this phase and encouraging progress being made, the problem of patch overfitting still hinders the application and deployment of repair techniques in practice.
Therefore, the APR community needs more advanced APCA techniques to improve the correctness of returned patches, e.g., patch-aware feature extraction and more powerful pre-trained models.

{{formula:6f93f060-04c4-479a-846c-803b7228c8c7}}


AID: A Benchmark Dataset for Performance Evaluation of Aerial Scene Classification

Abstract: Aerial scene classification, which aims to automatically label an aerial image with a specific semantic category, is a fundamental problem for understanding high-resolution remote sensing imagery. In recent years, it has become an active task in remote sensing area and numerous algorithms have been proposed for this task, including many machine learning and data-driven approaches. However, the existing datasets for aerial scene classification like UC-Merced dataset and WHU-RS19 are with relatively small sizes, and the results on them are already saturated. This largely limits the development of scene classification algorithms. This paper describes the Aerial Image Dataset (AID): a large-scale dataset for aerial scene classification. The goal of AID is to advance the state-of-the-arts in scene classification of remote sensing images. For creating AID, we collect and annotate more than ten thousands aerial scene images. In addition, a comprehensive review of the existing aerial scene classification techniques as well as recent widely-used deep learning methods is given. Finally, we provide a performance analysis of typical aerial scene classification and deep learning approaches on AID, which can be served as the baseline results on this benchmark.

Describing Textures in the Wild

Abstract: Patterns and textures are defining characteristics of many natural objects: a shirt can be striped, the wings of a butterfly can be veined, and the skin of an animal can be scaly. Aiming at supporting this analytical dimension in image understanding, we address the challenging problem of describing textures with semantic attributes. We identify a rich vocabulary of forty-seven texture terms and use them to describe a large dataset of patterns collected in the wild.The resulting Describable Textures Dataset (DTD) is the basis to seek for the best texture representation for recognizing describable texture attributes in images. We port from object recognition to texture recognition the Improved Fisher Vector (IFV) and show that, surprisingly, it outperforms specialized texture descriptors not only on our problem, but also in established material recognition datasets. We also show that the describable attributes are excellent texture descriptors, transferring between datasets and tasks; in particular, combined with IFV, they significantly outperform the state-of-the-art by more than 8 percent on both FMD and KTHTIPS-2b benchmarks. We also demonstrate that they produce intuitive descriptions of materials and Internet images.

Hard Sample Aware Noise Robust Learning for Histopathology Image Classification

Abstract: Deep learning-based histopathology image classification is a key technique to help physicians in improving the accuracy and promptness of cancer diagnosis. However, the noisy labels are often inevitable in the complex manual annotation process, and thus mislead the training of the classification model. In this work, we introduce a novel hard sample aware noise robust learning method for histopathology image classification. To distinguish the informative hard samples from the harmful noisy ones, we build an easy/hard/noisy (EHN) detection model by using the sample training history. Then we integrate the EHN into a self-training architecture to lower the noise rate through gradually label correction. With the obtained almost clean dataset, we further propose a noise suppressing and hard enhancing (NSHE) scheme to train the noise robust model. Compared with the previous works, our method can save more clean samples and can be directly applied to the real-world noisy dataset scenario without using a clean subset. Experimental results demonstrate that the proposed scheme outperforms the current state-of-the-art methods in both the synthetic and real-world noisy datasets. The source code and data are available at https://github.com/bupt-ai-cz/HSA-NRL/.

The image datasets used in our experiments.

For clarity, we summarize the statistics of the eight image dataset in Table {{table:4985b1f3-b3ef-4529-b2b8-451271956d93}}  and illustrate some sample images of them in Fig. {{figure:ecb60d1c-d2bf-41f7-9459-c4b37f7c114c}} .


SQuAD: 100,000+ Questions for Machine Comprehension of Text

Abstract:
We present the Stanford Question Answering Dataset (SQuAD), a new reading comprehension dataset consisting of 100,000+ questions posed by crowdworkers on a set of Wikipedia articles, where the answer to each question is a segment of text from the corresponding reading passage. We analyze the dataset to understand the types of reasoning required to answer the questions, leaning heavily on dependency and constituency trees. We build a strong logistic regression model, which achieves an F1 score of 51.0%, a significant improvement over a simple baseline (20%). However, human performance (86.8%) is much higher, indicating that the dataset presents a good challenge problem for future research.
The dataset is freely available at this https URL

Know What You Don’t Know: Unanswerable Questions for SQuAD

Abstract: Extractive reading comprehension systems can often locate the correct answer to a question in a context document, but they also tend to make unreliable guesses on questions for which the correct answer is not stated in the context. Existing datasets either focus exclusively on answerable questions, or use automatically generated unanswerable questions that are easy to identify. To address these weaknesses, we present SQuADRUn, a new dataset that combines the existing Stanford Question Answering Dataset (SQuAD) with over 50,000 unanswerable questions written adversarially by crowdworkers to look similar to answerable ones. To do well on SQuADRUn, systems must not only answer questions when possible, but also determine when no answer is supported by the paragraph and abstain from answering. SQuADRUn is a challenging natural language understanding task for existing models: a strong neural system that gets 86% F1 on SQuAD achieves only 66% F1 on SQuADRUn. We release SQuADRUn to the community as the successor to SQuAD.

NewsQA: A Machine Comprehension Dataset

Abstract: We present NewsQA, a challenging machine comprehension dataset of over 100,000 human-generated question-answer pairs. Crowdworkers supply questions and answers based on a set of over 10,000 news articles from CNN, with answers consisting of spans of text in the articles. We collect this dataset through a four-stage process designed to solicit exploratory questions that require reasoning. Analysis confirms that NewsQA demands abilities beyond simple word matching and recognizing textual entailment. We measure human performance on the dataset and compare it to several strong neural models. The performance gap between humans and machines (13.3% F1) indicates that significant progress can be made on NewsQA through future research. The dataset is freely available online.

Quiz-Style Question Generation for News Stories

Abstract: A large majority of American adults get at least some of their news from the Internet. Even though many online news products have the goal of informing their users about the news, they lack scalable and reliable tools for measuring how well they are achieving this goal, and therefore have to resort to noisy proxy metrics (e.g., click-through rates or reading time) to track their performance. As a first step towards measuring news informedness at a scale, we study the problem of quiz-style multiple-choice question generation, which may be used to survey users about their knowledge of recent news. In particular, we formulate the problem as two sequence-to-sequence tasks: question-answer generation (QAG) and distractor, or incorrect answer, generation (DG). We introduce NewsQuizQA, the first dataset intended for quiz-style question-answer generation, containing 20K human written question-answer pairs from 5K news article summaries. Using this dataset, we propose a series of novel techniques for applying large pre-trained Transformer encoder-decoder models, namely PEGASUS and T5, to the tasks of question-answer generation and distractor generation. We show that our models outperform strong baselines using both automated metrics and human raters. We provide a case study of running weekly quizzes on real-world users via the Google Surveys platform over the course of two months. We found that users generally found the automatically generated questions to be educational and enjoyable. Finally, to serve the research community, we are releasing the NewsQuizQA dataset.

Comparison of related datasets. Note that there are more synchronic datasets that are not listed here {{cite:b50fb98}} (roughly about 30 common QA datasets based on our investigation).

Thus, the goal of this work is to create a large-scale QA dataset over long-term historical document collections, that can promote the development of ODQA systems on such valuable resources. The characteristics of our QA dataset can be compared with those of the most related datasets in Tab. {{table:15cb8c80-95d4-4621-921d-b69f9f2e4d7f}} .