C-RNN-GAN: Continuous recurrent neural networks with adversarial training

Abstract: Generative adversarial networks have been proposed as a way of efficiently training deep generative neural networks. We propose a generative adversarial model that works on continuous sequential data, and apply it by training it on a collection of classical music. We conclude that it generates music that sounds better and better as the model is trained, report statistics on generated music, and let the reader judge the quality by downloading the generated songs.

Real-valued (Medical) Time Series Generation with Recurrent Conditional GANs

Abstract: Generative Adversarial Networks (GANs) have shown remarkable success as a framework for training models to produce realistic-looking data. In this work, we propose a Recurrent GAN (RGAN) and Recurrent Conditional GAN (RCGAN) to produce realistic real-valued multi-dimensional time series, with an emphasis on their application to medical data. RGANs make use of recurrent neural networks (RNNs) in the generator and the discriminator. In the case of RCGANs, both of these RNNs are conditioned on auxiliary information. We demonstrate our models in a set of toy datasets, where we show visually and quantitatively (using sample likelihood and maximum mean discrepancy) that they can successfully generate realistic time-series. We also describe novel evaluation methods for GANs, where we generate a synthetic labelled training dataset, and evaluate on a real test set the performance of a model trained on the synthetic data, and vice-versa. We illustrate with these metrics that RCGANs can generate time-series data useful for supervised training, with only minor degradation in performance on real test data. This is demonstrated on digit classification from ‘serialised’ MNIST and by training an early warning system on a medical dataset of 17,000 patients from an intensive care unit. We further discuss and analyse the privacy concerns that may arise when using RCGANs to generate realistic synthetic medical time series data, and demonstrate results from differentially private training of the RCGAN.

T-CGAN: Conditional Generative Adversarial Network for Data Augmentation in Noisy Time Series with Irregular Sampling

Abstract: In this paper we propose a data augmentation method for time series with irregular sampling, Time-Conditional Generative Adversarial Network (T-CGAN). Our approach is based on Conditional Generative Adversarial Networks (CGAN), where the generative step is implemented by a deconvolutional NN and the discriminative step by a convolutional NN. Both the generator and the discriminator are conditioned on the sampling timestamps, to learn the hidden relationship between data and timestamps, and consequently to generate new time series. We evaluate our model with synthetic and real-world datasets. For the synthetic data, we compare the performance of a classifier trained with T-CGAN-generated data, against the performance of the same classifier trained on the original data. Results show that classifiers trained on T-CGAN-generated data perform the same as classifiers trained on real data, even with very short time series and small training sets. For the real world datasets, we compare our method with other techniques of data augmentation for time series, such as time slicing and time warping, over a classification problem with unbalanced datasets. Results show that our method always outperforms the other approaches, both in case of regularly sampled and irregularly sampled time series. We achieve particularly good performance in case with a small training set and short, noisy, irregularly-sampled time series.

Adversarial Audio Synthesis

Abstract: Audio signals are sampled at high temporal resolutions, and learning to synthesize audio requires capturing structure across a range of timescales. Generative adversarial networks (GANs) have seen wide success at generating images that are both locally and globally coherent, but they have seen little application to audio generation. In this paper we introduce WaveGAN, a first attempt at applying GANs to unsupervised synthesis of raw-waveform audio. WaveGAN is capable of synthesizing one second slices of audio waveforms with global coherence, suitable for sound effect generation. Our experiments demonstrate that, without labels, WaveGAN learns to produce intelligible words when trained on a small-vocabulary speech dataset, and can also synthesize audio from other domains such as drums, bird vocalizations, and piano. We compare WaveGAN to a method which applies GANs designed for image generation on image-like audio feature representations, finding both approaches to be promising.

Conditional GAN for timeseries generation

Abstract: It is abundantly clear that time dependent data is a vital source of information in the world. The challenge has been for applications in machine learning to gain access to a considerable amount of quality data needed for algorithm development and analysis. Modeling synthetic data using a Generative Adversarial Network (GAN) has been at the heart of providing a viable solution. Our work focuses on one dimensional times series and explores the few shot approach, which is the ability of an algorithm to perform well with limited data. This work attempts to ease the frustration by proposing a new architecture, Time Series GAN (TSGAN), to model realistic time series data. We evaluate TSGAN on 70 data sets from a benchmark time series database. Our results demonstrate that TSGAN performs better than the competition both quantitatively using the Frechet Inception Score (FID) metric, and qualitatively when classification is used as the evaluation criteria.

Quant GANs: deep generation of financial time series

Abstract: Modeling financial time series by stochastic processes is a challenging task and a central area of research in financial mathematics. As an alternative, we introduce Quant GANs, a data-driven model which is inspired by the recent success of generative adversarial networks (GANs). Quant GANs consist of a generator and discriminator function, which utilize temporal convolutional networks (TCNs) and thereby achieve to capture long-range dependencies such as the presence of volatility clusters. The generator function is explicitly constructed such that the induced stochastic process allows a transition to its risk-neutral distribution. Our numerical results highlight that distributional properties for small and large lags are in an excellent agreement and dependence properties such as volatility clusters, leverage effects, and serial autocorrelations can be generated by the generator function of Quant GANs, demonstrably in high fidelity.

COT-GAN: Generating Sequential Data via Causal Optimal Transport

Abstract: We introduce COT-GAN, an adversarial algorithm to train implicit generative models optimized for producing sequential data. The loss function of this algorithm is formulated using ideas from Causal Optimal Transport (COT), which combines classic optimal transport methods with an additional temporal causality constraint. Remarkably, we find that this causality condition provides a natural framework to parameterize the cost function that is learned by the discriminator as a robust (worst-case) distance, and an ideal mechanism for learning time dependent data distributions. Following Genevay et al.\ (2018), we also include an entropic penalization term which allows for the use of the Sinkhorn algorithm when computing the optimal transport cost. Our experiments show effectiveness and stability of COT-GAN when generating both low- and high-dimensional time series data. The success of the algorithm also relies on a new, improved version of the Sinkhorn divergence which demonstrates less bias in learning.

Sig-Wasserstein GANs for Time Series Generation

Abstract: Synthetic data is an emerging technology that can significantly accelerate the development and deployment of AI machine learning pipelines. In this work, we develop high-fidelity time-series generators, the SigWGAN, by combining continuous-time stochastic models with the newly proposed signature $W_1$ metric. The former are the Logsig-RNN models based on the stochastic differential equations, whereas the latter originates from the universal and principled mathematical features to characterize the measure induced by time series. SigWGAN allows turning computationally challenging GAN min-max problem into supervised learning while generating high fidelity samples. We validate the proposed model on both synthetic data generated by popular quantitative risk models and empirical financial data. Codes are available at https://github.com/SigCGANs/Sig-Wasserstein-GANs.git.

Time-series Generation by Contrastive Imitation

Abstract: Consider learning a generative model for time-series data. The sequential setting poses a unique challenge: Not only should the generator capture the conditional dynamics of (stepwise) transitions, but its open-loop rollouts should also preserve the joint distribution of (multi-step) trajectories. On one hand, autoregressive models trained by MLE allow learning and computing explicit transition distributions, but suffer from compounding error during rollouts. On the other hand, adversarial models based on GAN training alleviate such exposure bias, but transitions are implicit and hard to assess. In this work, we study a generative framework that seeks to combine the strengths of both: Motivated by a moment-matching objective to mitigate compounding error, we optimize a local (but forward-looking) transition policy, where the reinforcement signal is provided by a global (but stepwise-decomposable) energy model trained by contrastive estimation. At training, the two components are learned cooperatively, avoiding the instabilities typical of adversarial objectives. At inference, the learned policy serves as the generator for iterative sampling, and the learned energy serves as a trajectory-level measure for evaluating sample quality. By expressly training a policy to imitate sequential behavior of time-series features in a dataset, this approach embodies "generation by imitation". Theoretically, we illustrate the correctness of this formulation and the consistency of the algorithm. Empirically, we evaluate its ability to generate predictively useful samples from real-world datasets, verifying that it performs at the standard of existing benchmarks.

Towards Generating Real-World Time Series Data

Abstract: Time series data generation has drawn increasing attention in recent years. Several generative adversarial network (GAN) based methods have been proposed to tackle the problem usually with the assumption that the targeted time series data are well-formatted and complete. However, real-world time series (RTS) data are far away from this utopia, e.g., long sequences with variable lengths and informative missing data raise intractable challenges for designing powerful generation algorithms. In this paper, we propose a novel generative framework for RTS data - RTSGAN to tackle the aforementioned challenges. RTSGAN first learns an encoder-decoder module which provides a mapping between a time series instance and a fixed-dimension latent vector and then learns a generation module to generate vectors in the same latent space. By combining the generator and the decoder, RTSGAN is able to generate RTS which respect the original feature distributions and the temporal dynamics. To generate time series with missing values, we further equip RTSGAN with an observation embedding layer and a decide-and-generate decoder to better utilize the informative missing patterns. Experiments on the four RTS datasets show that the proposed framework outperforms the previous generation methods in terms of synthetic data utility for downstream classification and prediction tasks.

PSA-GAN: Progressive Self Attention GANs for Synthetic Time Series

Abstract: Realistic synthetic time series data of sufficient length enables practical applications in time series modeling tasks, such as forecasting, but remains a challenge. In this paper we present PSA-GAN, a generative adversarial network (GAN) that generates long time series samples of high quality using progressive growing of GANs and self-attention. We show that PSA-GAN can be used to reduce the error in two downstream forecasting tasks over baselines that only use real data. We also introduce a Frechet-Inception Distance-like score, Context-FID, assessing the quality of synthetic time series samples. In our downstream tasks, we find that the lowest scoring models correspond to the best-performing ones. Therefore, Context-FID could be a useful tool to develop time series GAN models.

Conditional Loss and Deep Euler Scheme for Time Series Generation

Abstract: We introduce three new generative models for time series that are based on Euler discretization of Stochastic Differential Equations (SDEs) and Wasserstein metrics. Two of these methods rely on the adaptation of generative adversarial networks (GANs) to time series. The third algorithm, called Conditional Euler Generator (CEGEN), minimizes a dedicated distance between the transition probability distributions over all time steps. In the context of Itô processes, we provide theoretical guarantees that minimizing this criterion implies accurate estimations of the drift and volatility parameters. Empirically, CEGEN outperforms state-of-the-art and GANs on both marginal and temporal dynamic metrics. Besides, correlation structures are accurately identified in high dimension. When few real data points are available, we verify the effectiveness of CEGEN when combined with transfer learning methods on model-based simulations. Finally, we illustrate the robustness of our methods on various real-world data sets.

Time-series Transformer Generative Adversarial Networks

Abstract: Many real-world tasks are plagued by limitations on data: in some instances very little data is available and in others, data is protected by privacy enforcing regulations (e.g. GDPR). We consider limitations posed specifically on time-series data and present a model that can generate synthetic time-series which can be used in place of real data. A model that generates synthetic time-series data has two objectives: 1) to capture the stepwise conditional distribution of real sequences, and 2) to faithfully model the joint distribution of entire real sequences. Autoregressive models trained via maximum likelihood estimation can be used in a system where previous predictions are fed back in and used to predict future ones; in such models, errors can accrue over time. Furthermore, a plausible initial value is required making MLE based models not really generative. Many downstream tasks learn to model conditional distributions of the time-series, hence, synthetic data drawn from a generative model must satisfy 1) in addition to performing 2). We present TsT-GAN, a framework that capitalises on the Transformer architecture to satisfy the desiderata and compare its performance against five state-of-the-art models on five datasets and show that TsT-GAN achieves higher predictive performance on all datasets.

Generating multivariate time series with COmmon Source CoordInated GAN (COSCI-GAN)

Abstract: Generating multivariate time series is a promising approach for sharing sensitive data in many medical, financial, and IoT applications. A common type of multivariate time series originates from a single source such as the biometric measurements from a medical patient. This leads to complex dynamical patterns between individual time series that are hard to learn by typical generation models such as GANs. There is valuable information in those patterns that machine learning models can use to better classify, predict or perform other downstream tasks. We propose a novel framework that takes time series' common origin into account and favors channel/feature relationships preservation. The two key points of our method are: 1) the individual time series are generated from a common point in latent space and 2) a central discriminator favors the preservation of inter-channel/feature dynamics. We demonstrate empirically that our method helps preserve channel/feature correlations and that our synthetic data performs very well in downstream tasks with medical and financial data.

TimeVAE: A Variational Auto-Encoder for Multivariate Time Series Generation

Abstract: Recent work in synthetic data generation in the time-series domain has focused on the use of Generative Adversarial Networks. We propose a novel architecture for synthetically generating time-series data with the use of Variational Auto-Encoders (VAEs). The proposed architecture has several distinct properties: interpretability, ability to encode domain knowledge, and reduced training times. We evaluate data generation quality by similarity and predictability against four multivariate datasets. We experiment with varying sizes of training data to measure the impact of data availability on generation quality for our VAE method as well as several state-of-the-art data generation methods. Our results on similarity tests show that the VAE approach is able to accurately represent the temporal attributes of the original data. On next-step prediction tasks using generated data, the proposed VAE architecture consistently meets or exceeds performance of state-of-the-art data generation methods. While noise reduction may cause the generated data to deviate from original data, we demonstrate the resulting de-noised data can significantly improve performance for next-step prediction using generated data. Finally, the proposed architecture can incorporate domain-specific time-patterns such as polynomial trends and seasonalities to provide interpretable outputs. Such interpretability can be highly advantageous in applications requiring transparency of model outputs or where users desire to inject prior knowledge of time-series patterns into the generative model.

Causal Recurrent Variational Autoencoder for Medical Time Series Generation

Abstract: We propose causal recurrent variational autoencoder (CR-VAE), a novel generative model that is able to learn a Granger causal graph from a multivariate time series x and incorporates the underlying causal mechanism into its data generation process. Distinct to the classical recurrent VAEs, our CR-VAE uses a multi-head decoder, in which the $p$-th head is responsible for generating the $p$-th dimension of $\mathbf{x}$ (i.e., $\mathbf{x}^p$). By imposing a sparsity-inducing penalty on the weights (of the decoder) and encouraging specific sets of weights to be zero, our CR-VAE learns a sparse adjacency matrix that encodes causal relations between all pairs of variables. Thanks to this causal matrix, our decoder strictly obeys the underlying principles of Granger causality, thereby making the data generating process transparent. We develop a two-stage approach to train the overall objective. Empirically, we evaluate the behavior of our model in synthetic data and two real-world human brain datasets involving, respectively, the electroencephalography (EEG) signals and the functional magnetic resonance imaging (fMRI) data. Our model consistently outperforms state-of-the-art time series generative models both qualitatively and quantitatively. Moreover, it also discovers a faithful causal graph with similar or improved accuracy over existing Granger causality-based causal inference methods. Code of CR-VAE is publicly available at https://github.com/hongmingli1995/CR-VAE.

Vector Quantized Time Series Generation with a Bidirectional Prior Model

Abstract: Time series generation (TSG) studies have mainly focused on the use of Generative Adversarial Networks (GANs) combined with recurrent neural network (RNN) variants. However, the fundamental limitations and challenges of training GANs still remain. In addition, the RNN-family typically has difficulties with temporal consistency between distant timesteps. Motivated by the successes in the image generation (IMG) domain, we propose TimeVQVAE, the first work, to our knowledge, that uses vector quantization (VQ) techniques to address the TSG problem. Moreover, the priors of the discrete latent spaces are learned with bidirectional transformer models that can better capture global temporal consistency. We also propose VQ modeling in a time-frequency domain, separated into low-frequency (LF) and high-frequency (HF). This allows us to retain important characteristics of the time series and, in turn, generate new synthetic signals that are of better quality, with sharper changes in modularity, than its competing TSG methods. Our experimental evaluation is conducted on all datasets from the UCR archive, using well-established metrics in the IMG literature, such as Fr\'echet inception distance and inception scores. Our implementation on GitHub: \url{https://github.com/ML4ITS/TimeVQVAE}.

Neural Ordinary Differential Equations

Abstract: We introduce a new family of deep neural network models. Instead of specifying a discrete sequence of hidden layers, we parameterize the derivative of the hidden state using a neural network. The output of the network is computed using a black-box differential equation solver. These continuous-depth models have constant memory cost, adapt their evaluation strategy to each input, and can explicitly trade numerical precision for speed. We demonstrate these properties in continuous-depth residual networks and continuous-time latent variable models. We also construct continuous normalizing flows, a generative model that can train by maximum likelihood, without partitioning or ordering the data dimensions. For training, we show how to scalably backpropagate through any ODE solver, without access to its internal operations. This allows end-to-end training of ODEs within larger models.

Neural SDEs as Infinite-Dimensional GANs

Abstract: Stochastic differential equations (SDEs) are a staple of mathematical modelling of temporal dynamics. However, a fundamental limitation has been that such models have typically been relatively inflexible, which recent work introducing Neural SDEs has sought to solve. Here, we show that the current classical approach to fitting SDEs may be approached as a special case of (Wasserstein) GANs, and in doing so the neural and classical regimes may be brought together. The input noise is Brownian motion, the output samples are time-evolving paths produced by a numerical solver, and by parameterising a discriminator as a Neural Controlled Differential Equation (CDE), we obtain Neural SDEs as (in modern machine learning parlance) continuous-time generative time series models. Unlike previous work on this problem, this is a direct extension of the classical approach without reference to either prespecified statistics or density functions. Arbitrary drift and diffusions are admissible, so as the Wasserstein loss has a unique global minima, in the infinite data limit any SDE may be learnt. Example code has been made available as part of the \texttt{torchsde} repository.

GT-GAN: General Purpose Time Series Synthesis with Generative Adversarial Networks

Abstract: Time series synthesis is an important research topic in the field of deep learning, which can be used for data augmentation. Time series data types can be broadly classified into regular or irregular. However, there are no existing generative models that show good performance for both types without any model changes. Therefore, we present a general purpose model capable of synthesizing regular and irregular time series data. To our knowledge, we are the first designing a general purpose time series synthesis model, which is one of the most challenging settings for time series synthesis. To this end, we design a generative adversarial network-based method, where many related techniques are carefully integrated into a single framework, ranging from neural ordinary/controlled differential equations to continuous time-flow processes. Our method outperforms all existing methods.

Deep Latent State Space Models for Time-Series Generation

Abstract: Methods based on ordinary differential equations (ODEs) are widely used to build generative models of time-series. In addition to high computational overhead due to explicitly computing hidden states recurrence, existing ODE-based models fall short in learning sequence data with sharp transitions - common in many real-world systems - due to numerical challenges during optimization. In this work, we propose LS4, a generative model for sequences with latent variables evolving according to a state space ODE to increase modeling capacity. Inspired by recent deep state space models (S4), we achieve speedups by leveraging a convolutional representation of LS4 which bypasses the explicit evaluation of hidden states. We show that LS4 significantly outperforms previous continuous-time generative models in terms of marginal distribution, classification, and prediction scores on real-world datasets in the Monash Forecasting Repository, and is capable of modeling highly stochastic data with sharp temporal transitions. LS4 sets state-of-the-art for continuous-time latent generative models, with significant improvement of mean squared error and tighter variational lower bounds on irregularly-sampled datasets, while also being x100 faster than other baselines on long sequences.

Modeling Continuous Stochastic Processes with Dynamic Normalizing Flows

Abstract: Normalizing flows transform a simple base distribution into a complex target distribution and have proved to be powerful models for data generation and density estimation. In this work, we propose a novel type of normalizing flow driven by a differential deformation of the Wiener process. As a result, we obtain a rich time series model whose observable process inherits many of the appealing properties of its base process, such as efficient computation of likelihoods and marginals. Furthermore, our continuous treatment provides a natural framework for irregular time series with an independent arrival process, including straightforward interpolation. We illustrate the desirable properties of the proposed model on popular stochastic processes and demonstrate its superior flexibility to variational RNN and latent ODE baselines in a series of experiments on synthetic and real-world data.

Regular Time-series Generation using SGM

Abstract: Score-based generative models (SGMs) are generative models that are in the spotlight these days. Time-series frequently occurs in our daily life, e.g., stock data, climate data, and so on. Especially, time-series forecasting and classification are popular research topics in the field of machine learning. SGMs are also known for outperforming other generative models. As a result, we apply SGMs to synthesize time-series data by learning conditional score functions. We propose a conditional score network for the time-series generation domain. Furthermore, we also derive the loss function between the score matching and the denoising score matching in the time-series generation domain. Finally, we achieve state-of-the-art results on real-world datasets in terms of sampling diversity and quality.

Summary of popular TSG methods with their backbone models and specialties (TS: Time Series).

C1: We provide a new taxonomy of various TSG methods based on three foundational generative models.
Regarding L1, we establish a taxonomy for systematic comparison and categorization (§).
Specifically, we categorize TSG methods based on three foundational generative models, offering a new perspective for understanding existing research.
On this basis, we compare and analyze the model they used, make connections if different methods use similar techniques, and elaborate upon the inheritance and improvement of relative methods (Figure {{figure:1eb861dc-b259-400e-8286-ca6b22c6ca10}}  and Table {{table:da7c0f08-9de6-41da-a34e-f48e3c5835cd}} ).

We detail ten representative TSG methods (A1–A10) grounded in the three foundational generative models.
Table {{table:da7c0f08-9de6-41da-a34e-f48e3c5835cd}}  provides a summary of these methods, including their backbone models.


PAC-Bayesian Inequalities for Martingales

Abstract: We present a set of high-probability inequalities that control the concentration of weighted averages of multiple (possibly uncountably many) simultaneously evolving and interdependent martingales. Our results extend the PAC-Bayesian (probably approximately correct) analysis in learning theory from the i.i.d. setting to martingales opening the way for its application to importance weighted sampling, reinforcement learning, and other interactive learning domains, as well as many other domains in probability theory and statistics, where martingales are encountered. We also present a comparison inequality that bounds the expectation of a convex function of a martingale difference sequence shifted to the [0, 1] interval by the expectation of the same function of independent Bernoulli random variables. This inequality is applied to derive a tighter analog of Hoeffding-Azuma's inequality.

PAC-Bayes Iterated Logarithm Bounds for Martingale Mixtures

Abstract: We give tight concentration bounds for mixtures of martingales that are simultaneously uniform over (a) mixture distributions, in a PAC-Bayes sense; and (b) all finite times. These bounds are proved in terms of the martingale variance, extending classical Bernstein inequalities, and sharpening and simplifying prior work.

On the properties of variational approximations of Gibbs posteriors

Abstract: The PAC-Bayesian approach is a powerful set of techniques to derive non- asymptotic risk bounds for random estimators. The corresponding optimal distribution of estimators, usually called the Gibbs posterior, is unfortunately intractable. One may sample from it using Markov chain Monte Carlo, but this is often too slow for big datasets. We consider instead variational approximations of the Gibbs posterior, which are fast to compute. We undertake a general study of the properties of such approximations. Our main finding is that such a variational approximation has often the same rate of convergence as the original PAC-Bayesian procedure it approximates. We specialise our results to several learning tasks (classification, ranking, matrix completion),discuss how to implement a variational approximation in each case, and illustrate the good properties of said approximation on real datasets.

PAC-Bayes Unleashed: Generalisation Bounds with Unbounded Losses

Abstract: We present new PAC-Bayesian generalisation bounds for learning problems with unbounded loss functions. This extends the relevance and applicability of the PAC-Bayes learning framework, where most of the existing literature focuses on supervised learning problems with a bounded loss function (typically assumed to take values in the interval [0;1]). In order to relax this classical assumption, we propose to allow the range of the loss to depend on each predictor. This relaxation is captured by our new notion of HYPothesis-dependent rangE (HYPE). Based on this, we derive a novel PAC-Bayesian generalisation bound for unbounded loss functions, and we instantiate it on a linear regression problem. To make our theory usable by the largest audience possible, we include discussions on actual computation, practicality and limitations of our assumptions.

A Note on the PAC Bayesian Theorem

Abstract: We prove general exponential moment inequalities for averages of [0,1]-valued iid random variables and use them to tighten the PAC Bayesian Theorem. The logarithmic dependence on the sample count in the enumerator of the PAC Bayesian bound is halved.

A Strongly Quasiconvex PAC-Bayesian Bound

Abstract: We propose a new PAC-Bayesian bound and a way of constructing a hypothesis space, so that the bound is convex in the posterior distribution and also convex in a trade-off parameter between empirical performance of the posterior distribution and its complexity. The complexity is measured by the Kullback-Leibler divergence to a prior. We derive an alternating procedure for minimizing the bound. We show that the bound can be rewritten as a one-dimensional function of the trade-off parameter and provide sufficient conditions under which the function has a single global minimum. When the conditions are satisfied the alternating minimization is guaranteed to converge to the global minimum of the bound. We provide experimental results demonstrating that rigorous minimization of the bound is competitive with cross-validation in tuning the trade-off between complexity and empirical performance. In all our experiments the trade-off turned to be quasiconvex even when the sufficient conditions were violated.

Integral Probability Metrics PAC-Bayes Bounds

Abstract: We present a PAC-Bayes-style generalization bound which enables the replacement of the KL-divergence with a variety of Integral Probability Metrics (IPM). We provide instances of this bound with the IPM being the total variation metric and the Wasserstein distance. A notable feature of the obtained bounds is that they naturally interpolate between classical uniform convergence bounds in the worst case (when the prior and posterior are far away from each other), and improved bounds in favorable cases (when the posterior and prior are close). This illustrates the possibility of reinforcing classical generalization bounds with algorithm- and data-dependent components, thus making them more suitable to analyze algorithms that use a large hypothesis space.

Some existing results which are either strengthened by or consequences of our unified framework.The first column refers to the type of underlying process used to construct the bound.The final column points to which corollary implies the existing result (either directly or as a consequence of selecting certain parameters; the precise relationship will be described in the text).The above results are mostly corollaries of Theorem  (a PAC-Bayes framework with the KL divergence), but several rely on Theorem  (a framework for general {{formula:832a608c-d11e-42be-a9cc-0697b854a255}} -divergences).The PAC-Bayes literature is large and we cannot include all previous results and their relationships, but we hope this gives the reader an idea of the scope of our approach.All existing results, save for those of {{cite:4726495}} and {{cite:cab7d63}}, are fixed-time bounds.We do not provide numbers in the second and third rows because the bounds were not explicitly written out in {{cite:c7b9e6e}}, {{cite:48737fb}}. See {{cite:ac9b6dd}} for a summary.

Recently, concurrent to our own work, {{cite:4726495}} derived a few anytime-valid PAC-Bayes bounds. They also employ supermartingales and Ville's inequality, two ingredients which are also central to our approach. Our general framework will encompass their results, recovering their theorems as special cases of our own. More importantly however, our unified framework will strengthen or generalize a much broader slew of existing PAC-Bayes bounds. See Table {{table:efc8af59-03ba-4dd6-9aed-fb083f2c8d6d}}  for a summary of these generalizations.

In this work, we identify a general martingale-like structure at the heart of many existing PAC-Bayes bounds. This structure takes the form of either a nonnegative supermartingale or a nonnegative reverse submartingale. Such an identification enables us to (i) give a general framework for seeking new bounds, and (ii) give time-uniform extensions of many existing PAC-Bayes bounds. Our time-uniform extensions do not lose any tightness compared with their fixed-time counterparts. Additionally, we are able to relax many of the assumptions in traditional bounds, such as i.i.d data and stationary loss functions. A summary of our generalizations is given in Table {{table:efc8af59-03ba-4dd6-9aed-fb083f2c8d6d}} .

The machinery of nonnegative supermartingales (and their mixtures) in addition to Ville's inequality is sufficient to give time-uniform PAC-Bayes bounds in a wide variety of situations. Section  is dedicated to this task.
See the first half of Table {{table:efc8af59-03ba-4dd6-9aed-fb083f2c8d6d}}  for those bounds which are recovered using this technique.
However, to recover time-uniform versions of other well-known PAC-Bayes bounds, we must rely on reverse-time martingales. We introduce these next.

A General Recipe for Stochastic Processes

We now present results for nonnegative processes upper bounded by either a supermartingale or a reverse submartingale. We will consider processes {{formula:1259997b-2a8d-42d7-b7a0-89f15a5e0a23}}  which are functions of a parameter {{formula:feb2403a-4d1c-46d4-86b1-2959fbeb9f88}} . While the following theorem does not appear to be in the form of a traditional PAC-Bayes bound, a variety of typical bounds can be recovered by considering particular processes {{formula:8e4f731c-bb45-4134-933e-9efb5faae445}}  (Table {{table:efc8af59-03ba-4dd6-9aed-fb083f2c8d6d}} ).
Many such fruitful processes will be presented throughout the remainder of this manuscript.

We have demonstrated that underlying many PAC-Bayes bounds is a (typically implicit) supermartingale or reverse submartingale structure.
Such structure, when coupled with the method of mixtures and Ville's inequalities, provides a general method of deriving new bounds and serves to illuminate the connection between existing bounds (Table {{table:efc8af59-03ba-4dd6-9aed-fb083f2c8d6d}} ).
For instance, we are able to generate PAC-Bayes bounds for sub-{{formula:8ddd7ab9-55aa-4789-aea0-7bd719d3718a}}  processes {{cite:f308e4f}}, a broad class of stochastic processes which itself encapsulates a large swath of existing concentration inequalities.
More generally, as soon as one identifies a nonnegative supermartingale or reverse submartingale with bounded initial value, our framework supplies a time-uniform PAC-Bayes bound. We hope this serves to both ease the search for future bounds and to provide a more unified view of the existing literature.


Black-Box Generation of Adversarial Text Sequences to Evade Deep Learning Classifiers

Abstract: Although various techniques have been proposed to generate adversarial samples for white-box attacks on text, little attention has been paid to a black-box attack, which is a more realistic scenario. In this paper, we present a novel algorithm, DeepWordBug, to effectively generate small text perturbations in a black-box setting that forces a deep-learning classifier to misclassify a text input. We develop novel scoring strategies to find the most important words to modify such that the deep classifier makes a wrong prediction. Simple character-level transformations are applied to the highest-ranked words in order to minimize the edit distance of the perturbation. We evaluated DeepWordBug on two real-world text datasets: Enron spam emails and IMDB movie reviews. Our experimental results indicate that DeepWordBug can reduce the classification accuracy from 99% to 40% on Enron and from 87% to 26% on IMDB. Our results strongly demonstrate that the generated adversarial sequences from a deep-learning model can similarly evade other deep models.

Interpretable Adversarial Perturbation in Input Embedding Space for Text

Abstract: Following great success in the image processing field, the idea of adversarial training has been applied to tasks in the natural language processing (NLP) field. One promising approach directly applies adversarial training developed in the image processing field to the input word embedding space instead of the discrete input space of texts. However, this approach abandons such interpretability as generating adversarial texts to significantly improve the performance of NLP tasks. This paper restores interpretability to such methods by restricting the directions of perturbations toward the existing words in the input embedding space. As a result, we can straightforwardly reconstruct each input with perturbations to an actual text by considering the perturbations to be the replacement of words in the sentence while maintaining or even improving the task performance.

Learning to Discriminate Perturbations for Blocking Adversarial Attacks in Text Classification

Abstract: Adversarial attacks against machine learning models have threatened various real-world applications such as spam filtering and sentiment analysis. In this paper, we propose a novel framework, learning to DIScriminate Perturbations (DISP), to identify and adjust malicious perturbations, thereby blocking adversarial attacks for text classification models. To identify adversarial attacks, a perturbation discriminator validates how likely a token in the text is perturbed and provides a set of potential perturbations. For each potential perturbation, an embedding estimator learns to restore the embedding of the original word based on the context and a replacement token is chosen based on approximate kNN search. DISP can block adversarial attacks for any NLP model without modifying the model structure or training procedure. Extensive experiments on two benchmark datasets demonstrate that DISP significantly outperforms baseline methods in blocking adversarial attacks for text classification. In addition, in-depth analysis shows the robustness of DISP across different situations.

Word modifications of the main word-level adversarial attacks. Sub.: Substitution, Rand.: random attack randomly samples a word to replace the target word. Embed: it replaces the word with a word among the top-10 nearest words in the embedding space.

Compared to character-level attacks, a word-level attack is naturally more imperceptible for humans and more difficult for machine learning algorithms to defend. In Table {{table:ae1f252b-b92b-4997-af62-0a8aaf352cbc}} , we summarize several word-level perturbations proposed in several research works. In Table {{table:539cae99-d72d-45b5-a281-7d6ae64345b4}} , we report several word-level attack contributions and categorize them based on model accessibility, attack type, targeted model, application, or task. In a broader sense, summarize the main research contributions made at word-level adversarial attack in Fig. {{figure:1ec2aa65-8b55-4a35-901f-ebc44cc66eda}} . These can be divided into four main categories; (i) optimizing adversarial attacks,
(ii) identifying vulnerability in new applications, (iii) attack analysis and understanding, and (iv) investigating new attack aspects. Below we discuss each of them in detail.


Introduction to the CoNLL-2003 Shared Task: Language-Independent Named Entity Recognition

Abstract: We describe the CoNLL-2003 shared task: language-independent named entity recognition. We give background information on the data sets (English and German) and the evaluation method, present a general overview of the systems that have taken part in the task and discuss their performance.

SQuAD: 100,000+ Questions for Machine Comprehension of Text

Abstract:
We present the Stanford Question Answering Dataset (SQuAD), a new reading comprehension dataset consisting of 100,000+ questions posed by crowdworkers on a set of Wikipedia articles, where the answer to each question is a segment of text from the corresponding reading passage. We analyze the dataset to understand the types of reasoning required to answer the questions, leaning heavily on dependency and constituency trees. We build a strong logistic regression model, which achieves an F1 score of 51.0%, a significant improvement over a simple baseline (20%). However, human performance (86.8%) is much higher, indicating that the dataset presents a good challenge problem for future research.
The dataset is freely available at this https URL

Summary of the models analyzed in our study

A final perspective from which we analyze the carbon emissions of ML models is by comparing the amount of carbon emitted by models to their performance on benchmark tasks such as image classification, machine translation and question answering.
We compare the emissions of the models from our sample and their performance on four tasks: image recognition on ImageNet {{cite:fd456ef}} (35 models), machine translation for English-French and English-German on the 2014 WMT Translation tasks {{cite:9dcfedc}} (30 models), question answering on the SQuAD 1.1 dataset {{cite:13ecc4c}} (10 models), and named entity recognition on the CoNLL 2003 dataset {{cite:95549e5}} (11 models) We also had data from a fifth task, object detection, which is represented in Table {{table:bda68913-51c7-412c-a8ea-103fda8a11b2}}  and Figure {{figure:fa40c6ce-9c96-4302-9a43-159e55f3fa0a}} , but we did not have enough distinct data points to enable a meaningful comparison.. Our goal with this analysis is to validate whether, generally speaking, the more carbon-intensive models from our sample achieved better performance on common benchmarks compared to the models with less incurred emissions.


This dataset used STC dataset {{cite:f9624a5}}, and performed automatic annotation by using the best performing classifier trained on NLPCC 2013 http://tcci.ccf.org.cn/conference/2013/ and NLPCC 2014 http://tcci.ccf.org.cn/conference/2014/ datasets. This dataset contains 217,905 conversations where each response annotated by 6 emotion labels.

This dataset is built by using crowd-sourcing scenario using ParlAI which involves 810 different participants. This dataset contains 24,850 conversations/prompts where each conversation annotated by 32 emotion labels.

This dataset gathered from Twitter based on customer care conversation in 62 brands across different industries. 500 conversation was chosen randomly and annotated by 8 major tones using CrowdFlower https://www.figure-eight.com/ platform

This dataset (called “DailyDialog”) is crawled from various websites which serve for English learner to practice English dialog in daily life. DailyDialog contains 13,118 dialogs, and was manually annotated with communication intention and emotion information. They use six primary basic emotion (Anger, Disgust, Fear, Happiness, Sadness, Surprise) to label the conversations.

Emotional Chatting Machine: Emotional Conversation Generation with Internal and External Memory

Abstract: Perception and expression of emotion are key factors to the success of dialogue systems or conversational agents. However, this problem has not been studied in large-scale conversation generation so far. In this paper, we propose Emotional Chatting Machine (ECM) that can generate appropriate responses not only in content (relevant and grammatical) but also in emotion (emotionally consistent). To the best of our knowledge, this is the first work that addresses the emotion factor in large-scale conversation generation. ECM addresses the factor using three new mechanisms that respectively (1) models the high-level abstraction of emotion expressions by embedding emotion categories, (2) captures the change of implicit internal emotion states, and (3) uses explicit emotion expressions with an external emotion vocabulary. Experiments show that the proposed model can generate responses appropriate not only in content but also in emotion.

I Know the Feeling: Learning to Converse with Empathy

Abstract:
Beyond understanding what is being discussed, human communication requires an
awareness of what someone is feeling. One challenge for dialogue agents is
being able to recognize feelings in the conversation partner and reply
accordingly, a key communicative skill that is trivial for humans. Research in
this area is made difficult by the paucity of large-scale publicly available
datasets both for emotion and relevant dialogues. This work proposes a new task
for empathetic dialogue generation and EmpatheticDialogues, a dataset of 25k
conversations grounded in emotional contexts to facilitate training and
evaluating dialogue systems. Our experiments indicate that models explicitly
leveraging emotion predictions from previous utterances are perceived to be
more empathetic by human evaluators, while improving on other metrics as well
(e.g. perceived relevance of responses, BLEU scores).

Touch Your Heart: A Tone-aware Chatbot for Customer Care on Social Media

Abstract: Chatbot has become an important solution to rapidly increasing customer care demands on social media in recent years. However, current work on chatbot for customer care ignores a key to impact user experience - tones. In this work, we create a novel tone-aware chatbot that generates toned responses to user requests on social media. We first conduct a formative research, in which the effects of tones are studied. Significant and various influences of different tones on user experience are uncovered in the study. With the knowledge of effects of tones, we design a deep learning based chatbot that takes tone information into account. We train our system on over 1.5 million real customer care conversations collected from Twitter. The evaluation reveals that our tone-aware chatbot generates as appropriate responses to user requests as human agents. More importantly, our chatbot is perceived to be even more empathetic than human agents.

DailyDialog: A Manually Labelled Multi-turn Dialogue Dataset

Abstract: We develop a high-quality multi-turn dialog dataset, DailyDialog, which is intriguing in several aspects. The language is human-written and less noisy. The dialogues in the dataset reflect our daily communication way and cover various topics about our daily life. We also manually label the developed dataset with communication intention and emotion information. Then, we evaluate existing approaches on DailyDialog dataset and hope it benefit the research field of dialog systems. The dataset is available on http://yanran.li/dailydialog

Summarization of dataset available for emotionally-aware chatbot.

In this section, we try to investigate the available resources in building EAC. As other artificial intelligent agents, building chatbot also needs a dataset to be learned, to be able to produce a meaningful conversation as a human-like agent. Therefore, some studies propose dataset which contains textual conversation annotated by different emotion categories. Table {{table:f5c75e99-40ff-4e18-aba1-cf306285511c}}  summarizes the available dataset that found in recent years. We categorize the dataset based on the language, source of the data, and further description which contains some information such as annotation approach, size of instances, and emotion labels. All datasets were proposed during 2017 and 2018, started by dataset provided by NLPCC 2017 Shared Task on Emotion Generation Challenge organizers. This dataset gathered from Sina Weibo social media https://www.weibo.com/login.php, so it consists of social conversation in Chinese. Based on our study, all of the datasets that we discover are only available in two languages, English and Chinese. However, the source of these datasets is very diverse such as social media (Twitter, Sina Weibo, and Facebook Message), online content, and human writing through crowdsourcing scenario. Our investigation found that every dataset use different set of emotion label depends on its focus and objective in building the chatbot.


Sequential Recommendation with Graph Neural Networks

Abstract: Sequential recommendation aims to leverage users' historical behaviors to predict their next interaction. Existing works have not yet addressed two main challenges in sequential recommendation. First, user behaviors in their rich historical sequences are often implicit and noisy preference signals, they cannot sufficiently reflect users' actual preferences. In addition, users' dynamic preferences often change rapidly over time, and hence it is difficult to capture user patterns in their historical sequences. In this work, we propose a graph neural network model called SURGE (short for SeqUential Recommendation with Graph neural nEtworks) to address these two issues. Specifically, SURGE integrates different types of preferences in long-term user behaviors into clusters in the graph by re-constructing loose item sequences into tight item-item interest graphs based on metric learning. This helps explicitly distinguish users' core interests, by forming dense clusters in the interest graph. Then, we perform cluster-aware and query-aware graph convolutional propagation and graph pooling on the constructed graph. It dynamically fuses and extracts users' current activated core interests from noisy user behavior sequences. We conduct extensive experiments on both public and proprietary industrial datasets. Experimental results demonstrate significant performance gains of our proposed method compared to state-of-the-art methods. Further studies on sequence length confirm that our method can model long behavioral sequences effectively and efficiently.

Knowledge-Enhanced Graph Neural Networks for Sequential Recommendation

Abstract: With the rapid increase in the popularity of big data and internet technology, sequential recommendation has become an important method to help people find items they are potentially interested in. Traditional recommendation methods use only recurrent neural networks (RNNs) to process sequential data. Although effective, the results may be unable to capture both the semantic-based preference and the complex transitions between items adequately. In this paper, we model separated session sequences into session graphs and capture complex transitions using graph neural networks (GNNs). We further link items in interaction sequences with existing external knowledge base (KB) entities and integrate the GNN-based recommender with key-value memory networks (KV-MNs) to incorporate KB knowledge. Specifically, we set a key matrix to many relation embeddings that learned from KB, corresponding to many entity attributes, and set up a set of value matrices storing the semantic-based preferences of different users for the corresponding attribute. By using a hybrid of a GNN and KV-MN, each session is represented as the combination of the current interest (i.e., sequential preference) and the global preference (i.e., semantic-based preference) of that session. Extensive experiments on three public real-world datasets show that our method performs better than baseline algorithms consistently.

Inter-sequence Enhanced Framework for Personalized Sequential Recommendation

Abstract: Modeling the sequential correlation of users' historical interactions is essential in sequential recommendation. However, the majority of the approaches mainly focus on modeling the \emph{intra-sequence} item correlation within each individual sequence but neglect the \emph{inter-sequence} item correlation across different user interaction sequences. Though several studies have been aware of this issue, their method is either simple or implicit. To make better use of such information, we propose an inter-sequence enhanced framework for the Sequential Recommendation (ISSR). In ISSR, both inter-sequence and intra-sequence item correlation are considered. Firstly, we equip graph neural networks in the inter-sequence correlation encoder to capture the high-order item correlation from the user-item bipartite graph and the item-item graph. Then, based on the inter-sequence correlation encoder, we build GRU network and attention network in the intra-sequence correlation encoder to model the item sequential correlation within each individual sequence and temporal dynamics for predicting users' preferences over candidate items. Additionally, we conduct extensive experiments on three real-world datasets. The experimental results demonstrate the superiority of ISSR over many state-of-the-art methods and the effectiveness of the inter-sequence correlation encoder.

Memory Augmented Graph Neural Networks for Sequential Recommendation

Abstract: The chronological order of user-item interactions can reveal time-evolving and sequential user behaviors in many recommender systems. The items that users will interact with may depend on the items accessed in the past. However, the substantial increase of users and items makes sequential recommender systems still face non-trivial challenges: (1) the hardness of modeling the short-term user interests; (2) the difficulty of capturing the long-term user interests; (3) the effective modeling of item co-occurrence patterns. To tackle these challenges, we propose a memory augmented graph neural network (MA-GNN) to capture both the long- and short-term user interests. Specifically, we apply a graph neural network to model the item contextual information within a short-term period and utilize a shared memory network to capture the long-range dependencies between items. In addition to the modeling of user interests, we employ a bilinear function to capture the co-occurrence patterns of related items. We extensively evaluate our model on five real-world datasets, comparing with several state-of-the-art methods and using a variety of performance metrics. The experimental results demonstrate the effectiveness of our model for the task of Top-K sequential recommendation.

Dynamic Graph Neural Networks for Sequential Recommendation

Abstract: Modeling user preference from his historical sequences is one of the core problems of sequential recommendation. Existing methods in this field are widely distributed from conventional methods to deep learning methods. However, most of them only model users' interests within their own sequences and ignore the dynamic collaborative signals among different user sequences, making it insufficient to explore users' preferences. We take inspiration from dynamic graph neural networks to cope with this challenge, modeling the user sequence and dynamic collaborative signals into one framework. We propose a new method named Dynamic Graph Neural Network for Sequential Recommendation (DGSR), which connects different user sequences through a dynamic graph structure, exploring the interactive behavior of users and items with time and order information. Furthermore, we design a Dynamic Graph Recommendation Network to extract user's preferences from the dynamic graph. Consequently, the next-item prediction task in sequential recommendation is converted into a link prediction between the user node and the item node in a dynamic graph. Extensive experiments on three public benchmarks show that DGSR outperforms several state-of-the-art methods. Further studies demonstrate the rationality and effectiveness of modeling user sequences through a dynamic graph.

RetaGNN: Relational Temporal Attentive Graph Neural Networks for Holistic Sequential Recommendation

Abstract: Sequential recommendation (SR) is to accurately recommend a list of items for a user based on her current accessed ones. While new-coming users continuously arrive in the real world, one crucial task is to have inductive SR that can produce embeddings of users and items without re-training. Given user-item interactions can be extremely sparse, another critical task is to have transferable SR that can transfer the knowledge derived from one domain with rich data to another domain. In this work, we aim to present the holistic SR that simultaneously accommodates conventional, inductive, and transferable settings. We propose a novel deep learning-based model, Relational Temporal Attentive Graph Neural Networks (RetaGNN), for holistic SR. The main idea of RetaGNN is three-fold. First, to have inductive and transferable capabilities, we train a relational attentive GNN on the local subgraph extracted from a user-item pair, in which the learnable weight matrices are on various relations among users, items, and attributes, rather than nodes or edges. Second, long-term and short-term temporal patterns of user preferences are encoded by a proposed sequential self-attention mechanism. Third, a relation-aware regularization term is devised for better training of RetaGNN. Experiments conducted on MovieLens, Instagram, and Book-Crossing datasets exhibit that RetaGNN can outperform state-of-the-art methods under conventional, inductive, and transferable settings. The derived attention weights also bring model explainability.

Continuous-Time Sequential Recommendation with Temporal Graph Collaborative Transformer

Abstract: In order to model the evolution of user preference, we should learn user/item embeddings based on time-ordered item purchasing sequences, which is defined as Sequential Recommendation~(SR) problem. Existing methods leverage sequential patterns to model item transitions. However, most of them ignore crucial temporal collaborative signals, which are latent in evolving user-item interactions and coexist with sequential patterns. Therefore, we propose to unify sequential patterns and temporal collaborative signals to improve the quality of recommendation, which is rather challenging. Firstly, it is hard to simultaneously encode sequential patterns and collaborative signals. Secondly, it is non-trivial to express the temporal effects of collaborative signals. Hence, we design a new framework Temporal Graph Sequential Recommender (TGSRec) upon our defined continuous-time bipartite graph. We propose a novel Temporal Collaborative Transformer TCT layer in TGSRec, which advances the self-attention mechanism by adopting a novel collaborative attention. TCT layer can simultaneously capture collaborative signals from both users and items, as well as considering temporal dynamics inside sequential patterns. We propagate the information learned from TCT layer over the temporal graph to unify sequential patterns and temporal collaborative signals. Empirical results on five datasets show that modelname significantly outperforms other baselines, in average up to 22.5% and 22.1% absolute improvements in Recall@10 and MRR, respectively.

Discovering Collaborative Signals for Next POI Recommendation with Iterative Seq2Graph Augmentation

Abstract: Being an indispensable component in location-based social networks, next point-of-interest (POI) recommendation recommends users unexplored POIs based on their recent visiting histories. However, existing work mainly models check-in data as isolated POI sequences, neglecting the crucial collaborative signals from cross-sequence check-in information. Furthermore, the sparse POI-POI transitions restrict the ability of a model to learn effective sequential patterns for recommendation. In this paper, we propose Sequence-to-Graph (Seq2Graph) augmentation for each POI sequence, allowing collaborative signals to be propagated from correlated POIs belonging to other sequences. We then devise a novel Sequence-to-Graph POI Recommender (SGRec), which jointly learns POI embeddings and infers a user's temporal preferences from the graph-augmented POI sequence. To overcome the sparsity of POI-level interactions, we further infuse category-awareness into SGRec with a multi-task learning scheme that captures the denser category-wise transitions. As such, SGRec makes full use of the collaborative signals for learning expressive POI representations, and also comprehensively uncovers multi-level sequential patterns for user preference modelling. Extensive experiments on two real-world datasets demonstrate the superiority of SGRec against state-of-the-art methods in next POI recommendation.

STP-UDGAT: Spatial-Temporal-Preference User Dimensional Graph Attention Network for Next POI Recommendation

Abstract: Next Point-of-Interest (POI) recommendation is a longstanding problem across the domains of Location-Based Social Networks (LBSN) and transportation. Recent Recurrent Neural Network (RNN) based approaches learn POI-POI relationships in a local view based on independent user visit sequences. This limits the model's ability to directly connect and learn across users in a global view to recommend semantically trained POIs. In this work, we propose a Spatial-Temporal-Preference User Dimensional Graph Attention Network (STP-UDGAT), a novel explore-exploit model that concurrently exploits personalized user preferences and explores new POIs in global spatial-temporal-preference (STP) neighbourhoods, while allowing users to selectively learn from other users. In addition, we propose random walks as a masked self-attention option to leverage the STP graphs' structures and find new higher-order POI neighbours during exploration. Experimental results on six real-world datasets show that our model significantly outperforms baseline and state-of-the-art methods.

Details of GNN models for sequential recommendation.

Since GNN has the ability of high-order relationship modeling by aggregating information from neighbor nodes, after fusing multiple sequences into one graph, it can learn representations of both users and items in different sequences, which can't be accomplished by Markov model or recurrent neural network. Wang et al. {{cite:d0828e0}} propose a simple method that directly converts the sequence information into directed edges on the graph and then uses GNN to learn representations. Liu et al.  {{cite:51dd592}} construct a user-item bipartite graph and an item-item graph at the same time, where the edges of the item-item graph indicate co-occurrence in a sequence, with edge weights assigned according to the number of occurrences. The representations learned by GNN are used in the final recommendation through the recurrent neural network. Different from directly converting the temporal sequence into directed edges in the graph, DGSR {{cite:b7d6cf7}} and TGSRec {{cite:56abbb3}} consider the timestamps in the sequence in the process of graph construction. In the graph, each edge represents the interaction between the user and the item, and has the corresponding time attribute. Then perform convolution operations on the temporal graph to learn the representations of users and items. GES-SASRec {{cite:dd3518c}} and SGRec {{cite:a76cdf0}} focus on the learning of item representations. For an item in a sequence, GES-SASRec {{cite:dd3518c}} considers the next item of this item in other sequences, and SGRec {{cite:a76cdf0}} not only considers the next item but also considers the previous one. By aggregating the items before and after the target item in different sequences, the representation of the item is enhanced. GPR {{cite:f9d7192}} and GME {{cite:e83dcbc}} constructs edges between items by considering the frequency of consecutive occurrences or occurrences in the same sequence to enhance the representation. Some works are more complicated. For example, RetaGNN {{cite:e4849d1}} considers the attributes of the items when constructing the graph, while STP-UDGAT {{cite:1b0b41e}} considers the geographic location, timestamp, and frequency in the POI recommendation. Table {{table:3b152e4b-3933-4e89-aee1-1464b5b0aca6}}  summarizes the above works.


Seeing Stars: Exploiting Class Relationships for Sentiment Categorization with Respect to Rating Scales

Abstract: We address the rating-inference problem, wherein rather than simply decide whether a review is "thumbs up" or "thumbs down", as in previous sentiment analysis work, one must determine an author's evaluation with respect to a multi-point scale (e.g., one to five "stars"). This task represents an interesting twist on standard multi-class text categorization because there are several different degrees of similarity between class labels; for example, "three stars" is intuitively closer to "four stars" than to "one star".We first evaluate human performance at the task. Then, we apply a meta-algorithm, based on a metric labeling formulation of the problem, that alters a given n-ary classifier's output in an explicit attempt to ensure that similar items receive similar labels. We show that the meta-algorithm can provide significant improvements over both multi-class and regression versions of SVMs when we employ a novel similarity measure appropriate to the problem.

The Temple University Hospital Seizure Detection Corpus

Abstract: We introduce the TUH EEG Seizure Corpus (TUSZ), which is the largest open source corpus of its type, and represents an accurate characterization of clinical conditions. In this paper, we describe the techniques used to develop TUSZ, evaluate their effectiveness, and present some descriptive statistics on the resulting corpus.

Performance of the bestpipeline found using Lale with hyperopt. In all cases, thehyperopt trials used average 10-fold cross validation on thetraining set. For datasets that have a pre-defined train-testsplit, this table reports the accuracy on the test set, averagedover 3 experiments.

To demonstrate portability, we picked four datasets from different
modalities. For each dataset, we specified a planned pipeline with
operator choices in Lale, and then used hyperopt to pick the best
operators and tune their hyperparameters.
Table {{table:55158bd6-801e-4cd7-a19a-5c6739a5f571}}  summarizes the results.


First large-scale part segmentation dataset

Quadruped animal parsing, reorganized from {{cite:f527aa8}}

Human body parsing, reorganized from {{cite:f527aa8}}

Unsupervised domain adaptation from synthetic vehicles

Detect What You Can: Detecting and Representing Objects Using Holistic Models and Body Parts

Abstract: Detecting objects becomes difficult when we need to deal with large shape deformation, occlusion and low resolution. We propose a novel approach to i) handle large deformations and partial occlusions in animals (as examples of highly deformable objects), ii) describe them in terms of body parts, and iii) detect them when their body parts are hard to detect (e.g., animals depicted at low resolution). We represent the holistic object and body parts separately and use a fully connected model to arrange templates for the holistic object and body parts. Our model automatically decouples the holistic object or body parts from the model when they are hard to detect. This enables us to represent a large number of holistic object and body part combinations to better deal with different "detectability" patterns caused by deformations, occlusion and/or low resolution. We apply our method to the six animal categories in the PASCAL VOC dataset and show that our method significantly improves state-of-the-art (by 4.1% AP) and provides a richer representation for objects. During training we use annotations for body parts (e.g., head, torso, etc.), making use of a new dataset of fully annotated object parts for PASCAL VOC 2010, which provides a mask for each part.

Semantic Part Segmentation using Compositional Model combining Shape and Appearance

Abstract: In this paper, we study the problem of semantic part segmentation for animals. This is more challenging than standard object detection, object segmentation and pose estimation tasks because semantic parts of animals often have similar appearance and highly varying shapes. To tackle these challenges, we build a mixture of compositional models to represent the object boundary and the boundaries of semantic parts. And we incorporate edge, appearance, and semantic part cues into the compositional model. Given part-level segmentation annotation, we develop a novel algorithm to learn a mixture of compositional models under various poses and viewpoints for certain animal classes. Furthermore, a linear complexity algorithm is offered for efficient inference of the compositional model using dynamic programming. We evaluate our method for horse and cow using a newly annotated dataset on Pascal VOC 2010 which has pixelwise part labels. Experimental results demonstrate the effectiveness of our method.

Attention to Scale: Scale-aware Semantic Image Segmentation

Abstract: Incorporating multi-scale features in fully convolutional neural networks (FCNs) has been a key element to achieving state-of-the-art performance on semantic image segmentation. One common way to extract multi-scale features is to feed multiple resized input images to a shared deep network and then merge the resulting features for pixelwise classification. In this work, we propose an attention mechanism that learns to softly weight the multi-scale features at each pixel location. We adapt a state-of-the-art semantic image segmentation model, which we jointly train with multi-scale input images and the attention model. The proposed attention model not only outperforms average- and max-pooling, but allows us to diagnostically visualize the importance of features at different positions and scales. Moreover, we show that adding extra supervision to the output at each scale is essential to achieving excellent performance when merging multi-scale features. We demonstrate the effectiveness of our model with extensive experiments on three challenging datasets, including PASCAL-Person-Part, PASCAL VOC 2012 and a subset of MS-COCO 2014.

Look into Person: Joint Body Parsing & Pose Estimation Network and A New Benchmark

Abstract: Human parsing and pose estimation have recently received considerable interest due to their substantial application potentials. However, the existing datasets have limited numbers of images and annotations and lack a variety of human appearances and coverage of challenging cases in unconstrained environments. In this paper, we introduce a new benchmark named "Look into Person (LIP)" that provides a significant advancement in terms of scalability, diversity, and difficulty, which are crucial for future developments in human-centric analysis. This comprehensive dataset contains over 50,000 elaborately annotated images with 19 semantic part labels and 16 body joints, which are captured from a broad range of viewpoints, occlusions, and background complexities. Using these rich annotations, we perform detailed analyses of the leading human parsing and pose estimation approaches, thereby obtaining insights into the successes and failures of these methods. To further explore and take advantage of the semantic correlation of these two tasks, we propose a novel joint human parsing and pose estimation network to explore efficient context modeling, which can simultaneously predict parsing and pose with extremely high quality. Furthermore, we simplify the network to solve human parsing by exploring a novel self-supervised structure-sensitive learning approach, which imposes human pose structures into the parsing results without resorting to extra supervision. The dataset, code and models are available at http://www.sysu-hcp.net/lip/.

Adaptive Temporal Encoding Network for Video Instance-level Human Parsing

Abstract: Beyond the existing single-person and multiple-person human parsing tasks in static images, this paper makes the first attempt to investigate a more realistic video instance-level human parsing that simultaneously segments out each person instance and parses each instance into more fine-grained parts (e.g., head, leg, dress). We introduce a novel Adaptive Temporal Encoding Network (ATEN) that alternatively performs temporal encoding among key frames and flow-guided feature propagation from other consecutive frames between two key frames. Specifically, ATEN first incorporates a Parsing-RCNN to produce the instance-level parsing result for each key frame, which integrates both the global human parsing and instance-level human segmentation into a unified model. To balance between accuracy and efficiency, the flow-guided feature propagation is used to directly parse consecutive frames according to their identified temporal consistency with key frames. On the other hand, ATEN leverages the convolution gated recurrent units (convGRU) to exploit temporal changes over a series of key frames, which are further used to facilitate the frame-level instance-level parsing. By alternatively performing direct feature propagation between consistent frames and temporal encoding network among key frames, our ATEN achieves a good balance between frame-level accuracy and time efficiency, which is a common crucial problem in video object segmentation research. To demonstrate the superiority of our ATEN, extensive experiments are conducted on the most popular video segmentation benchmark (DAVIS) and a newly collected Video Instance-level Parsing (VIP) dataset, which is the first video instance-level human parsing dataset comprised of 404 sequences and over 20k frames with instance-level and pixel-wise annotations.

Part-aware Panoptic Segmentation

Abstract: In this work, we introduce the new scene understanding task of Part-aware Panoptic Segmentation (PPS), which aims to understand a scene at multiple levels of abstraction, and unifies the tasks of scene parsing and part parsing. For this novel task, we provide consistent annotations on two commonly used datasets: Cityscapes and Pascal VOC. Moreover, we present a single metric to evaluate PPS, called Part-aware Panoptic Quality (PartPQ). For this new task, using the metric and annotations, we set multiple baselines by merging results of existing state-of-the-art methods for panoptic segmentation and part segmentation. Finally, we conduct several experiments that evaluate the importance of the different levels of abstraction in this single task.

Learning Part Segmentation through Unsupervised Domain Adaptation from Synthetic Vehicles

Abstract: Part segmentations provide a rich and detailed part-level description of objects. However, their annotation requires an enormous amount of work, which makes it difficult to apply standard deep learning methods. In this paper, we propose the idea of learning part segmentation through unsupervised domain adaptation (UDA) from synthetic data. We first introduce UDA-Part, a comprehensive part segmentation dataset for vehicles that can serve as an adequate benchmark for UDA (https://qliu24.github.io/udapart). In UDA-Part, we label parts on 3D CAD models which enables us to generate a large set of annotated synthetic images. We also annotate parts on a number of real images to provide a real test set. Secondly, to advance the adaptation of part models trained from the synthetic data to the real images, we introduce a new UDA algorithm that leverages the object's spatial structure to guide the adaptation process. Our experimental results on two real test datasets confirm the superiority of our approach over existing works, and demonstrate the promise of learning part segmentation for general objects from synthetic data. We believe our dataset provides a rich testbed to study UDA for part segmentation and will help to significantly push forward research in this area.

Summarization and comparisons of 12 widely-used part segmentation benchmark datasets. Note that PPS {{cite:05dc461}} re-organizes two new datasets based on existing data annotations.

Human parsing. As in Tab. {{table:8909a245-9337-48a4-bba3-1bee6034decc}} , earlier works first tend to solve the specific categories of part segmentation, i.e., human parsingAlso noted as human part segmentation in some works.. Representative datasets including Fashionista {{cite:1ca041e}} focuses on the human clothes parsing, which segments human objects into typical classes including shorts, shoes, boots and sweaters. However, this dataset contains over 56 categories with a limited number of 685 images, which is not applicable to large machine-learning systems. With the development of deep learning techniques, large datasets are proposed to train and benchmark these deep models, e.g., ATR {{cite:6ce1cc5}} and LIP {{cite:60c19de}}, which consist of over 50,000 images of 20 categories for training and testing. These large benchmarks, as well as the accompanying baseline, have achieved great success in parsing humans into dressing clothes. Nevertheless, decomposing human objects with different clothing parts would lead to semantic inconsistencies on certain occasions. Hence the other line of works proposes to segment human bodies into semantic parts following the morphological rules, which share the same definitions with human poses. For example, Chen et al. {{cite:98913b8}} propose to organize the PASCAL-Person-Part dataset to segment human bodies into 7 semantic parts, including lower/upper-arms, torsos, lower/upper-legs, heads, and backgrounds. Leading by this trend, dozens of works {{cite:a25c347}}, {{cite:ee70a95}}, {{cite:19073fe}}, {{cite:2a4e0f5}}, {{cite:22cd5f4}}, {{cite:60c19de}}, {{cite:ea24bc2}}, {{cite:384ec0c}}, {{cite:97f8a5d}}, {{cite:1ddaa23}}, {{cite:7cdb71b}}, {{cite:3a1968a}}, {{cite:461a189}}, {{cite:4824d62}}, {{cite:5d91d91}}, {{cite:75f78eb}} propose to address this critical issue using deep learning techniques, which build well-established parsing baselines for understanding human structures.


Survey of existing risks and development of new layer framework to enhance security

Survey of benefits of 6G technology for IoT

A survey of IoT security threats based on IoT architecture

In depth analysis of existing risks and framework development for additional study

In depth analysis of current and future benefits of 6G technology

In depth analysis of security threats on each layer of IoT architecture

Focus on previous and existing network technology

Limited research on privacy and security

No discussion on the development of futuristic network technology and the implication of developing networks on IoT security

Cryotographic security methods that can work efficiently on IoT devices and standardised data collection method

Investigation of energy efficiency issues relating to 6g IoT networks

Application of the proposed classification on 6G networks

IoT Privacy and Security: Challenges and Solutions

Abstract: Privacy and security are among the significant challenges of the Internet of Things (IoT). Improper device updates, lack of efficient and robust security protocols, user unawareness, and famous active device monitoring are among the challenges that IoT is facing. In this work, we are exploring the background of IoT systems and security measures, and identifying (a) different security and privacy issues, (b) approaches used to secure the components of IoT-based environments and systems, (c) existing security solutions, and (d) the best privacy models necessary and suitable for different layers of IoT driven applications. In this work, we proposed a new IoT layered model: generic and stretched with the privacy and security components and layers identification. The proposed cloud/edge supported IoT system is implemented and evaluated. The lower layer represented by the IoT nodes generated from the Amazon Web Service (AWS) as Virtual Machines. The middle layer (edge) implemented as a Raspberry Pi 4 hardware kit with support of the Greengrass Edge Environment in AWS. We used the cloud-enabled IoT environment in AWS to implement the top layer (the cloud). The security protocols and critical management sessions were between each of these layers to ensure the privacy of the users’ information. We implemented security certificates to allow data transfer between the layers of the proposed cloud/edge enabled IoT model. Not only is the proposed system model eliminating possible security vulnerabilities, but it also can be used along with the best security techniques to countermeasure the cybersecurity threats facing each one of the layers; cloud, edge, and IoT.

6G Internet of Things: A Comprehensive Survey

Abstract: The sixth generation (6G) wireless communication networks are envisioned to revolutionize customer services and applications via the Internet of Things (IoT) towards a future of fully intelligent and autonomous systems. In this article, we explore the emerging opportunities brought by 6G technologies in IoT networks and applications, by conducting a holistic survey on the convergence of 6G and IoT. We first shed light on some of the most fundamental 6G technologies that are expected to empower future IoT networks, including edge intelligence, reconfigurable intelligent surfaces, space-air-ground-underwater communications, Terahertz communications, massive ultra-reliable and low-latency communications, and blockchain. Particularly, compared to the other related survey papers, we provide an in-depth discussion of the roles of 6G in a wide range of prospective IoT applications via five key domains, namely Healthcare Internet of Things, Vehicular Internet of Things and Autonomous Driving, Unmanned Aerial Vehicles, Satellite Internet of Things, and Industrial Internet of Things. Finally, we highlight interesting research challenges and point out potential directions to spur further research in this promising area.

A Survey of IoT Security Based on a Layered Architecture of Sensing and Data Analysis

Abstract: The Internet of Things (IoT) is leading today’s digital transformation. Relying on a combination of technologies, protocols, and devices such as wireless sensors and newly developed wearable and implanted sensors, IoT is changing every aspect of daily life, especially recent applications in digital healthcare. IoT incorporates various kinds of hardware, communication protocols, and services. This IoT diversity can be viewed as a double-edged sword that provides comfort to users but can lead also to a large number of security threats and attacks. In this survey paper, a new compacted and optimized architecture for IoT is proposed based on five layers. Likewise, we propose a new classification of security threats and attacks based on new IoT architecture. The IoT architecture involves a physical perception layer, a network and protocol layer, a transport layer, an application layer, and a data and cloud services layer. First, the physical sensing layer incorporates the basic hardware used by IoT. Second, we highlight the various network and protocol technologies employed by IoT, and review the security threats and solutions. Transport protocols are exhibited and the security threats against them are discussed while providing common solutions. Then, the application layer involves application protocols and lightweight encryption algorithms for IoT. Finally, in the data and cloud services layer, the main important security features of IoT cloud platforms are addressed, involving confidentiality, integrity, authorization, authentication, and encryption protocols. The paper is concluded by presenting the open research issues and future directions towards securing IoT, including the lack of standardized lightweight encryption algorithms, the use of machine-learning algorithms to enhance security and the related challenges, the use of Blockchain to address security challenges in IoT, and the implications of IoT deployment in 5G and beyond.

Related literature

To answer the research questions outlined above, a comprehensive search process was conducted to capture the relevant literature. In aligning with the inclusion and exclusion criteria outlined in Section REF , literature searches were kept within the period of 2009 until 2022. However, as part of the background discussion of internet privacy, which forms an integral part of IoT privacy, the research included journal articles from 1989. Google Scholar was used to identify thematic trends within articles within the periods identified. From the papers identified, the titles, keywords and abstract were collected for thematic analysis. From this search process, we were able to identify 137 articles for inclusion, of which the most related articles are shown in Table {{table:26136f32-8818-4c20-9dc6-191dd3794695}} . The search terms used in Google Scholar are highlighted as follows:


Electronic pathology textbooksPEIR Digital Library

Medical Segmentation Decathlon{{cite:9028a43}},NIH Chest X-ray{{cite:17e35ba}},CHAOS{{cite:0b7f332}}:- Chest X-rays/CTs- Abdomen CTs/MRIs- Head CTs/MRIs- Neck CTs- Pelvic cavity CTs

- Color- Location- Appearance- Shape- etc.

- Organ- Position- Knowledge Graph- Abnormality- Modality- Plane- Quality- Color- Size- Shape

PathVQA: 30000+ Questions for Medical Visual Question Answering

Abstract: Is it possible to develop an "AI Pathologist" to pass the board-certified examination of the American Board of Pathology? To achieve this goal, the first step is to create a visual question answering (VQA) dataset where the AI agent is presented with a pathology image together with a question and is asked to give the correct answer. Our work makes the first attempt to build such a dataset. Different from creating general-domain VQA datasets where the images are widely accessible and there are many crowdsourcing workers available and capable of generating question-answer pairs, developing a medical VQA dataset is much more challenging. First, due to privacy concerns, pathology images are usually not publicly available. Second, only well-trained pathologists can understand pathology images, but they barely have time to help create datasets for AI research. To address these challenges, we resort to pathology textbooks and online digital libraries. We develop a semi-automated pipeline to extract pathology images and captions from textbooks and generate question-answer pairs from captions using natural language processing. We collect 32,799 open-ended questions from 4,998 pathology images where each question is manually checked to ensure correctness. To our best knowledge, this is the first dataset for pathology VQA. Our dataset will be released publicly to promote research in medical VQA.

Slake: A Semantically-Labeled Knowledge-Enhanced Dataset For Medical Visual Question Answering

Abstract: Medical visual question answering (Med-VQA) has tremendous potential in healthcare. However, the development of this technology is hindered by the lacking of publicly-available and high-quality labeled datasets for training and evaluation. In this paper, we present a large bilingual dataset, SLAKE, with comprehensive semantic labels annotated by experienced physicians and a new structural medical knowledge base for Med-VQA. Besides, SLAKE includes richer modalities and covers more human body parts than the currently available dataset. We show that SLAKE can be used to facilitate the development and evaluation of Med-VQA systems. The dataset can be downloaded from http://www.med-vqa.com/slake.

Overview of the medical VQA datasets and their main characteristics. The VQA 2.0 is a general-domain VQA dataset listed here for comparison. The datasets are presented in chronological order.

To the best of our knowledge, there are 8 public-available medical VQA datasets up to date: VQA-MED-2018 {{cite:2fb1f26}}, VQA-RAD {{cite:eec8757}}, VQA-MED-2019 {{cite:8663dd4}}, RadVisDial {{cite:8fc94d3}}, PathVQA {{cite:0675f71}}, VQA-MED-2020 {{cite:c01f323}}, SLAKE {{cite:a875f4e}}, and VQA-MED-2021 {{cite:33a8141}} (in chronological order). The datasets details are summarized in Table {{table:519075eb-f4ac-4fa4-af6c-0356421253d7}} . In the following paragraphs, we provide an overview of the QA pairs collection.


Cosmos QA: Machine Reading Comprehension with Contextual Commonsense Reasoning

Abstract: Understanding narratives requires reading between the lines, which in turn, requires interpreting the likely causes and effects of events, even when they are not mentioned explicitly. In this paper, we introduce Cosmos QA, a large-scale dataset of 35,600 problems that require commonsense-based reading comprehension, formulated as multiple-choice questions. In stark contrast to most existing reading comprehension datasets where the questions focus on factual and literal understanding of the context paragraph, our dataset focuses on reading between the lines over a diverse collection of people’s everyday narratives, asking such questions as “what might be the possible reason of ...?", or “what would have happened if ..." that require reasoning beyond the exact text spans in the context. To establish baseline performances on Cosmos QA, we experiment with several state-of-the-art neural architectures for reading comprehension, and also propose a new architecture that improves over the competitive baselines. Experimental results demonstrate a significant gap between machine (68.4%) and human performance (94%), pointing to avenues for future research on commonsense machine comprehension. Dataset, code and leaderboard is publicly available at https://wilburone.github.io/cosmos.

DREAM: A Challenge Data Set and Models for Dialogue-Based Reading Comprehension

Abstract: We present DREAM, the first dialogue-based multiple-choice reading comprehension data set. Collected from English as a Foreign Language examinations designed by human experts to evaluate the comprehension level of Chinese learners of English, our data set contains 10,197 multiple-choice questions for 6,444 dialogues. In contrast to existing reading comprehension data sets, DREAM is the first to focus on in-depth multi-turn multi-party dialogue understanding. DREAM is likely to present significant challenges for existing reading comprehension systems: 84% of answers are non-extractive, 85% of questions require reasoning beyond a single sentence, and 34% of questions also involve commonsense knowledge. We apply several popular neural reading comprehension models that primarily exploit surface information within the text and find them to, at best, just barely outperform a rule-based approach. We next investigate the effects of incorporating dialogue structure and different kinds of general world knowledge into both rule-based and (neural and non-neural) machine learning-based reading comprehension models. Experimental results on the DREAM data set show the effectiveness of dialogue structure and general world knowledge. DREAM is available at https://dataset.org/dream/.

MCScript: A Novel Dataset for Assessing Machine Comprehension Using Script Knowledge

Abstract: We introduce a large dataset of narrative texts and questions about these texts, intended to be used in a machine comprehension task that requires reasoning using commonsense knowledge. Our dataset complements similar datasets in that we focus on stories about everyday activities, such as going to the movies or working in the garden, and that the questions require commonsense knowledge, or more specifically, script knowledge, to be answered. We show that our mode of data collection via crowdsourcing results in a substantial amount of such inference questions. The dataset forms the basis of a shared task on commonsense and script knowledge organized at SemEval 2018 and provides challenging test cases for the broader natural language understanding community.

MCScript2.0: A Machine Comprehension Corpus Focused on Script Events and Participants

Abstract: We introduce MCScript2.0, a machine comprehension corpus for the end-to-end evaluation of script knowledge. MCScript2.0 contains approx. 20,000 questions on approx. 3,500 texts, crowdsourced based on a new collection process that results in challenging questions. Half of the questions cannot be answered from the reading texts, but require the use of commonsense and, in particular, script knowledge. We give a thorough analysis of our corpus and show that while the task is not challenging to humans, existing machine comprehension models fail to perform well on the data, even if they make use of a commonsense knowledge base. The dataset is available at http://www.sfb1102. uni-saarland.de/?page_id=2582

RACE: Large-scale ReAding Comprehension Dataset From Examinations

Abstract: We present RACE, a new dataset for benchmark evaluation of methods in the reading comprehension task. Collected from the English exams for middle and high school Chinese students in the age range between 12 to 18, RACE consists of near 28,000 passages and near 100,000 questions generated by human experts (English instructors), and covers a variety of topics which are carefully designed for evaluating the students’ ability in understanding and reasoning. In particular, the proportion of questions that requires reasoning is much larger in RACE than that in other benchmark datasets for reading comprehension, and there is a significant gap between the performance of the state-of-the-art models (43%) and the ceiling human performance (95%). We hope this new dataset can serve as a valuable resource for research and evaluation in machine comprehension. The dataset is freely available at http://www.cs.cmu.edu/~glai1/data/race/ and the code is available at https://github.com/qizhex/RACE_AR_baselines.

Crowdsourcing Multiple Choice Science Questions

Abstract: We present a novel method for obtaining high-quality, domain-targeted multiple choice questions from crowd workers. Generating these questions can be difficult without trading away originality, relevance or diversity in the answer options. Our method addresses these problems by leveraging a large corpus of domain-specific text and a small set of existing questions. It produces model suggestions for document selection and answer distractor choice which aid the human question generation process. With this method we have assembled SciQ, a dataset of 13.7K multiple choice science exam questions. We demonstrate that the method produces in-domain questions by providing an analysis of this new dataset and by showing that humans cannot distinguish the crowdsourced questions from original questions. When using SciQ as additional training data to existing questions, we observe accuracy improvements on real science exams.

Datasets used for the multiple choice setting, including split used and sample size. Validation splits were used for CosmosQA since the test split is not publicly available, and for QASC since context passages or gold answers are not available.

Tables {{table:94f2e8c4-1e68-4cff-9550-c2c13b99674a}}  (multiple choice) and {{table:22e664b7-dd67-4780-923e-ca5570dbaeee}}  (extractive QA) outline the datasets we used. Additional details such as train size and preprocessing steps are available in the references provided. When space doesn't allow CosmosQA is aliased to Cosmos, MCScript to MCS, MCScript-2.0 to MCS2, and MCTest to MCT. The only preprocessing step we performed was to filter out questions where no context passage is provided. Validation splits (as opposed to test splits) are used in the CosmosQA and QASC cases, since context passages or gold standard answers are not available for these datasets.