Very Deep Convolutional Networks for Large-Scale Image Recognition

Abstract: In this work we investigate the effect of the convolutional network depth on its accuracy in the large-scale image recognition setting. Our main contribution is a thorough evaluation of networks of increasing depth using an architecture with very small (3x3) convolution filters, which shows that a significant improvement on the prior-art configurations can be achieved by pushing the depth to 16-19 weight layers. These findings were the basis of our ImageNet Challenge 2014 submission, where our team secured the first and the second places in the localisation and classification tracks respectively. We also show that our representations generalise well to other datasets, where they achieve state-of-the-art results. We have made our two best-performing ConvNet models publicly available to facilitate further research on the use of deep visual representations in computer vision.

Densely Connected Convolutional Networks

Abstract: Recent work has shown that convolutional networks can be substantially deeper, more accurate, and efficient to train if they contain shorter connections between layers close to the input and those close to the output. In this paper, we embrace this observation and introduce the Dense Convolutional Network (DenseNet), which connects each layer to every other layer in a feed-forward fashion. Whereas traditional convolutional networks with L layers have L connections—one between each layer and its subsequent layer—our network has L(L+1)/2 direct connections. For each layer, the feature-maps of all preceding layers are used as inputs, and its own feature-maps are used as inputs into all subsequent layers. DenseNets have several compelling advantages: they alleviate the vanishing-gradient problem, strengthen feature propagation, encourage feature reuse, and substantially reduce the number of parameters. We evaluate our proposed architecture on four highly competitive object recognition benchmark tasks (CIFAR-10, CIFAR-100, SVHN, and ImageNet). DenseNets obtain significant improvements over the state-of-the-art on most of them, whilst requiring less memory and computation to achieve high performance. Code and pre-trained models are available at https://github.com/liuzhuang13/DenseNet.

Aggregated Residual Transformations for Deep Neural Networks

Abstract: We present a simple, highly modularized network architecture for image classification. Our network is constructed by repeating a building block that aggregates a set of transformations with the same topology. Our simple design results in a homogeneous, multi-branch architecture that has only a few hyper-parameters to set. This strategy exposes a new dimension, which we call cardinality (the size of the set of transformations), as an essential factor in addition to the dimensions of depth and width. On the ImageNet-1K dataset, we empirically show that even under the restricted condition of maintaining complexity, increasing cardinality is able to improve classification accuracy. Moreover, increasing cardinality is more effective than going deeper or wider when we increase the capacity. Our models, named ResNeXt, are the foundations of our entry to the ILSVRC 2016 classification task in which we secured 2nd place. We further investigate ResNeXt on an ImageNet-5K set and the COCO detection set, also showing better results than its ResNet counterpart. The code and models are publicly available online.

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

Abstract: While the Transformer architecture has become the de-facto standard for natural language processing tasks, its applications to computer vision remain limited. In vision, attention is either applied in conjunction with convolutional networks, or used to replace certain components of convolutional networks while keeping their overall structure in place. We show that this reliance on CNNs is not necessary and a pure transformer applied directly to sequences of image patches can perform very well on image classification tasks. When pre-trained on large amounts of data and transferred to multiple mid-sized or small image recognition benchmarks (ImageNet, CIFAR-100, VTAB, etc.), Vision Transformer (ViT) attains excellent results compared to state-of-the-art convolutional networks while requiring substantially fewer computational resources to train.

A ConvNet for the 2020s

Abstract: The “Roaring 20s” of visual recognition began with the introduction of Vision Transformers (ViTs), which quickly superseded ConvNets as the state-of-the-art image classification model. A vanilla ViT, on the other hand, faces difficulties when applied to general computer vision tasks such as object detection and semantic segmentation. It is the hierarchical Transformers (e.g., Swin Transformers) that reintroduced several ConvNet priors, making Transformers practically viable as a generic vision backbone and demonstrating remarkable performance on a wide variety of vision tasks. However, the effectiveness of such hybrid approaches is still largely credited to the intrinsic superiority of Transformers, rather than the inherent inductive biases of convolutions. In this work, we reexamine the design spaces and test the limits of what a pure ConvNet can achieve. We gradually “modernize” a standard ResNet toward the design of a vision Transformer, and discover several key components that contribute to the performance difference along the way. The outcome of this exploration is a family of pure ConvNet models dubbed ConvNeXt. Constructed entirely from standard ConvNet modules, ConvNeXts compete favorably with Transformers in terms of accuracy and scalability, achieving 87.8% ImageNet top-1 accuracy and outperforming Swin Transformers on COCO detection and ADE20K segmentation, while maintaining the simplicity and efficiency of standard ConvNets.

Swin Transformer: Hierarchical Vision Transformer using Shifted Windows

Abstract: This paper presents a new vision Transformer, called Swin Transformer, that capably serves as a general-purpose backbone for computer vision. Challenges in adapting Transformer from language to vision arise from differences between the two domains, such as large variations in the scale of visual entities and the high resolution of pixels in images compared to words in text. To address these differences, we propose a hierarchical Transformer whose representation is computed with Shifted windows. The shifted windowing scheme brings greater efficiency by limiting self-attention computation to non-overlapping local windows while also allowing for cross-window connection. This hierarchical architecture has the flexibility to model at various scales and has linear computational complexity with respect to image size. These qualities of Swin Transformer make it compatible with a broad range of vision tasks, including image classification (87.3 top-1 accuracy on ImageNet-1K) and dense prediction tasks such as object detection (58.7 box AP and 51.1 mask AP on COCO test-dev) and semantic segmentation (53.5 mIoU on ADE20K val). Its performance surpasses the previous state-of-the-art by a large margin of +2.7 box AP and +2.6 mask AP on COCO, and +3.2 mIoU on ADE20K, demonstrating the potential of Transformer-based models as vision backbones. The hierarchical design and the shifted window approach also prove beneficial for all-MLP architectures. The code and models are publicly available at https://github.com/microsoft/Swin-Transformer.

Performance of different vision models for image classification on the animal subset. We report the top-1 accuracy on the original data and the edited images in different domains.

We also evaluated the models' performance on the animal subclass of the ImageNet-1K dataset. The results are provided in Table {{table:9c83dfb6-e661-4d5a-a53f-e1af6eb00056}} .
We chose 10 classes from the subclass and 10 images per class.


Use a reinforce-based method to perform parameter optimization

Resource Allocation in Uplink NOMA-IoT Networks: A Reinforcement-Learning Approach

Abstract: Non-orthogonal multiple access (NOMA) exploits the potential of the power domain to enhance the connectivity for the Internet of Things (IoT). Due to time-varying communication channels, dynamic user clustering is a promising method to increase the throughput of NOMA-IoT networks. This article develops an intelligent resource allocation scheme for uplink NOMA-IoT communications. To maximise the average performance of sum rates, this work designs an efficient optimization approach based on two reinforcement learning algorithms, namely deep reinforcement learning (DRL) and SARSA-learning. For light traffic, SARSA-learning is used to explore the safest resource allocation policy with low cost. For heavy traffic, DRL is used to handle traffic-introduced huge variables. With the aid of the considered approach, this work addresses two main problems of fair resource allocation in NOMA techniques: 1) allocating users dynamically and 2) balancing resource blocks and network traffic. We analytically demonstrate that the rate of convergence is inversely proportional to network sizes. Numerical results show that: 1) Compared with the optimal benchmark scheme, the proposed DRL and SARSA-learning algorithms have lower complexity with acceptable accuracy and 2) NOMA-enabled IoT networks outperform the conventional orthogonal multiple access based IoT networks in terms of system throughput.

Joint User Pairing and Association for Multicell NOMA: A Pointer Network-based Approach

Abstract: In this paper, we investigate the joint user pairing and association problem for multicell non-orthogonal multiple access (NOMA) systems. We consider a scenario where the user equipments (UEs) are located in a multicell network equipped with multiple base stations. Each base station has multiple orthogonal physical resource blocks (PRBs). Each PRB can be allocated to a pair of UEs using NOMA. Each UE has the additional freedom to be served by any one of the base stations, which further increases the complexity of the joint user pairing and association algorithm design. Leveraging the recent success on using machine learning to solve numerical optimization problems, we formulate the joint user pairing and association problem as a combinatorial optimization problem. The solution is found using an emerging deep learning architecture called Pointer Network (PtrNet), which has a lower computational complexity compared to solutions based on iterative algorithms and has been proven to achieve near-optimal performance. The training phase of the PtrNet is based on deep reinforcement learning (DRL), and does not require the use of the optimal solution of the formulated problem as training labels. Simulation results show that the proposed joint user pairing and association scheme achieves near-optimal performance in terms of the aggregate data rate, and outperforms the random user pairing and association heuristic by up to 30%.

Summary of Existing Work on Machine Learning for NOMA

Machine learning has also been leveraged to improve transmission security in NOMA networks. In {{cite:ea4e220}}, the interaction between the source station and the unmanned aerial vehicle (UAV) was viewed as a dynamic game and a DRL algorithm was applied for dynamic power allocation at the source station to suppress the attack motivation of the UAV. The simulation results showed that this strategy improves the system data rate and suppresses the attack probability. The NOMA transmission game against jamming was investigated in {{cite:3f44be7}}, where a fast Q-based NOMA power allocation scheme that combines the hotbooting technique and Dyna architecture is proposed for a dynamic game to accelerate the learning and thus improve the communication efficiency against smart jamming. The summary of the existing work on machine learning for NOMA is given in Table {{table:f9af0f72-2520-43bc-a358-b8835991437f}} .


Reevaluating Old Stellar Populations

Abstract: Determining the properties of old stellar populations (those with age >1 Gyr) has long involved the comparison of their integrated light, either in the form of photometry or spectroscopic indexes, with empirical or synthetic templates. Here we reevaluate the properties of old stellar populations using a new set of stellar population synthesis models, designed to incorporate the effects of binary stellar evolution pathways as a function of stellar mass and age. We find that single-aged stellar population models incorporating binary stars, as well as new stellar evolution and atmosphere models, can reproduce the colours and spectral indices observed in both globular clusters and quiescent galaxies. The best fitting model populations are often younger than those derived from older spectral synthesis models, and may also lie at slightly higher metallicities.

COSMIC Variance in Binary Population Synthesis

Abstract: The formation and evolution of binary stars is a critical component of several fields in astronomy. The most numerous sources for gravitational wave observatories are inspiraling and/or merging compact binaries, while binary stars are present in nearly every electromagnetic survey regardless of the target population. Simulations of large binary populations serve to both predict and inform observations of electromagnetic and gravitational wave sources. Binary population synthesis is a tool that balances physical modeling with simulation speed to produce large binary populations on timescales of days. We present a community-developed binary population synthesis suite: COSMIC which is designed to simulate compact-object binary populations and their progenitors. As a proof of concept, we simulate the Galactic population of compact binaries and their gravitational wave signal observable by the Laser Interferometer Space Antenna (LISA). We find that $\sim10^8$ compact binaries reside in the Milky Way today, while $\sim10^4$ of them may be resolvable by LISA.

The mass spectrum of compact remnants from the parsec stellar evolution tracks

Abstract: The mass spectrum of stellar-mass black holes (BHs) is highly uncertain. Dynamical mass measurements are available only for few ( 10) BHs in X-ray binaries, while theoretical models strongly depend on the hydrodynamics of supernova (SN) explosions and on the evolution of massive stars. In this paper, we present and discuss the mass spectrum of compact remnants that we obtained with SEVN, a new public populationsynthesis code, which couples the PARSEC stellar evolution tracks with up-to-date recipes for SN explosion (depending on the Carbon-Oxygen mass of the progenitor, on the compactness of the stellar core at pre-SN stage, and on a recent two-parameter criterion based on the dimensionless entropy per nucleon at pre-SN stage). SEVN can be used both as a stand-alone code and in combination with direct-summation N-body codes (Starlab, HiGPUs). The PARSEC stellar evolution tracks currently implemented in SEVN predict signicantly larger values of the Carbon-Oxygen core mass with respect to previous models. For most of the SN recipes we adopt, this implies substantially larger BH masses at low metallicity (6 2 10 3 ), than other populationsynthesis codes. The maximum BH mass found with SEVN is 25, 60 and 130 M at metallicity Z = 2 10 2 , 2 10 3 and 2 10 4 , respectively. Mass loss by stellar winds plays a major role in determining the mass of BHs for very massive stars (> 90 M ), while the remnant mass spectrum depends mostly on the adopted SN recipe for lower progenitor masses. We discuss the implications of our results for the transition between NS and BH mass, and for the expected number of massive BHs (with mass > 25 M ) as a function of metallicity.

Population synthesis codes used by the community at the time of writing. Binary_c, COSMIC, MOBSE are based on the BSE code {{cite:d6246a5}}.

Modelling isolated binary evolution and populations

Population synthesis codes are crucial for LISA science, both in order to make forecasts for source rates, but also to develop data analysis pipelines. Indeed, ongoing work on building fast and reliable waveforms relies on the knowledge of the expected properties of the sources (masses, spins, eccentricities) and the accuracies required to detect them and measure these properties. The codes currently in use by the LISA community are listed in Table {{table:af4b2082-9d65-4c9c-84a3-be54824cb5c2}} .
Further development of these codes, in particular the inclusion of additional physical processes as well as cross-checks between the codes will help to prepare for LISA observations.


Predicting Dynamic Embedding Trajectory in Temporal Interaction Networks

Abstract: Modeling sequential interactions between users and items/products is crucial in domains such as e-commerce, social networking, and education. Representation learning presents an attractive opportunity to model the dynamic evolution of users and items, where each user/item can be embedded in a Euclidean space and its evolution can be modeled by an embedding trajectory in this space. However, existing dynamic embedding methods generate embeddings only when users take actions and do not explicitly model the future trajectory of the user/item in the embedding space. Here we propose JODIE, a coupled recurrent neural network model that learns the embedding trajectories of users and items. JODIE employs two recurrent neural networks to update the embedding of a user and an item at every interaction. Crucially, JODIE also models the future embedding trajectory of a user/item. To this end, it introduces a novel projection operator that learns to estimate the embedding of the user at any time in the future. These estimated embeddings are then used to predict future user-item interactions. To make the method scalable, we develop a t-Batch algorithm that creates time-consistent batches and leads to 9x faster training. We conduct six experiments to validate JODIE on two prediction tasks---future interaction prediction and state change prediction---using four real-world datasets. We show that JODIE outperforms six state-of-the-art algorithms in these tasks by at least 20% in predicting future interactions and 12% in state change prediction.

Learning Sequence Encoders for Temporal Knowledge Graph Completion

Abstract: Research on link prediction in knowledge graphs has mainly focused on static multi-relational data. In this work we consider temporal knowledge graphs where relations between entities may only hold for a time interval or a specific point in time. In line with previous work on static knowledge graphs, we propose to address this problem by learning latent entity and relation type representations. To incorporate temporal information, we utilize recurrent neural networks to learn time-aware representations of relation types which can be used in conjunction with existing latent factorization methods. The proposed approach is shown to be robust to common challenges in real-world KGs: the sparsity and heterogeneity of temporal expressions. Experiments show the benefits of our approach on four temporal KGs. The data sets are available under a permissive BSD-3 license.

The node and link statics for each dataset.

Moochttps://snap.stanford.edu/data/act-mooc.tar.gz dataset contains user actions on a popular MOOC platform, where nodes represent users and course activities, and links represent user actions.
Dataset scales are listed in Table {{table:a03ba50e-a8f7-4690-a4a8-14ef25de0313}} .


UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild

Abstract: We introduce UCF101 which is currently the largest dataset of human actions. It consists of 101 action classes, over 13k clips and 27 hours of video data. The database consists of realistic user uploaded videos containing camera motion and cluttered background. Additionally, we provide baseline action recognition results on this new dataset using standard bag of words approach with overall performance of 44.5%. To the best of our knowledge, UCF101 is currently the most challenging dataset of actions due to its large number of classes, large number of clips and also unconstrained nature of such clips.

Hollywood in Homes: Crowdsourcing Data Collection for Activity Understanding

Abstract: Computer vision has a great potential to help our daily lives by searching for lost keys, watering flowers or reminding us to take a pill. To succeed with such tasks, computer vision methods need to be trained from real and diverse examples of our daily dynamic scenes. While most of such scenes are not particularly exciting, they typically do not appear on YouTube, in movies or TV broadcasts. So how do we collect sufficiently many diverse but boring samples representing our lives? We propose a novel Hollywood in Homes approach to collect such data. Instead of shooting videos in the lab, we ensure diversity by distributing and crowdsourcing the whole process of video creation from script writing to video recording and annotation. Following this procedure we collect a new dataset, Charades, with hundreds of people recording videos in their own homes, acting out casual everyday activities. The dataset is composed of 9,848 annotated videos with an average length of 30 seconds, showing activities of 267 people from three continents. Each video is annotated by multiple free-text descriptions, action labels, action intervals and classes of interacted objects. In total, Charades provides 27,847 video descriptions, 66,500 temporally localized intervals for 157 action classes and 41,104 labels for 46 object classes. Using this rich data, we evaluate and provide baseline results for several tasks including action recognition and automatic description generation. We believe that the realism, diversity, and casual nature of this dataset will present unique challenges and new opportunities for computer vision community.

STAIR Actions: A Video Dataset of Everyday Home Actions

Abstract: A new large-scale video dataset for human action recognition, called STAIR Actions is introduced. STAIR Actions contains 100 categories of action labels representing fine-grained everyday home actions so that it can be applied to research in various home tasks such as nursing, caring, and security. In STAIR Actions, each video has a single action label. Moreover, for each action category, there are around 1,000 videos that were obtained from YouTube or produced by crowdsource workers. The duration of each video is mostly five to six seconds. The total number of videos is 102,462. We explain how we constructed STAIR Actions and show the characteristics of STAIR Actions compared to existing datasets for human action recognition. Experiments with three major models for action recognition show that STAIR Actions can train large models and achieve good performance. STAIR Actions can be downloaded from http://actions.stair.center

Generic action recognition datasets, number of action classes and samples, and actions used in our custom dataset

As mentioned in the Introduction, our indoor monitoring system uses RGB video streams, and an essential aspect in this research field is the availability of training data and annotated datasets for benchmarking. In our work, we have selected some of the available datasets in the state of the art: HMDB {{cite:9d8da63}}, UCF-101 {{cite:ef44e0b}}, Charades {{cite:984659e}}, and Kinetics {{cite:47972c9}} (see Table {{table:a9046832-441b-40aa-8bae-1364b54ea3ba}} ). Our purpose is to use a pre-trained network architecture in a general action dataset such as Kinetics. After that, we will apply Transfer Learning to specialize our network, partially re-training the network on actions selected from all the datasets. This considerably reduces training time while still guarantees generalization {{cite:cc26b72}}. Our selection criteria is indoor actions that provide information about lifestyle.


XGBoost: A Scalable Tree Boosting System

Abstract: Tree boosting is a highly effective and widely used machine learning method. In this paper, we describe a scalable end-to-end tree boosting system called XGBoost, which is used widely by data scientists to achieve state-of-the-art results on many machine learning challenges. We propose a novel sparsity-aware algorithm for sparse data and weighted quantile sketch for approximate tree learning. More importantly, we provide insights on cache access patterns, data compression and sharding to build a scalable tree boosting system. By combining these insights, XGBoost scales beyond billions of examples using far fewer resources than existing systems.

DeepFM: A Factorization-Machine based Neural Network for CTR Prediction

Abstract: Learning sophisticated feature interactions behind user behaviors is critical in maximizing CTR for recommender systems. Despite great progress, existing methods seem to have a strong bias towards low- or high-order interactions, or require expertise feature engineering. In this paper, we show that it is possible to derive an end-to-end learning model that emphasizes both low- and high-order feature interactions. The proposed model, DeepFM, combines the power of factorization machines for recommendation and deep learning for feature learning in a new neural network architecture. Compared to the latest Wide & Deep model from Google, DeepFM has a shared input to its "wide" and "deep" parts, with no need of feature engineering besides raw features. Comprehensive experiments are conducted to demonstrate the effectiveness and efficiency of DeepFM over the existing models for CTR prediction, on both benchmark data and commercial data.

TabNet: Attentive Interpretable Tabular Learning

Abstract: We propose a novel high-performance and interpretable canonical deep tabular data learning architecture, TabNet. TabNet uses sequential attention to choose which features to reason from at each decision step, enabling interpretability and more efficient learning as the learning capacity is used for the most salient features. We demonstrate that TabNet outperforms other variants on a wide range of non-performance-saturated tabular datasets and yields interpretable feature attributions plus insights into its global behavior. Finally, we demonstrate self-supervised learning for tabular data, significantly improving performance when unlabeled data is abundant.

FinBERT: A Pretrained Language Model for Financial Communications

Abstract: Contextual pretrained language models, such as BERT (Devlin et al., 2019), have made significant breakthrough in various NLP tasks by training on large scale of unlabeled text re-sources.Financial sector also accumulates large amount of financial communication text.However, there is no pretrained finance specific language models available. In this work,we address the need by pretraining a financial domain specific BERT models, FinBERT, using a large scale of financial communication corpora. Experiments on three financial sentiment classification tasks confirm the advantage of FinBERT over generic domain BERT model. The code and pretrained models are available at this https URL. We hope this will be useful for practitioners and researchers working on financial NLP tasks.

LLaMA: Open and Efficient Foundation Language Models

Abstract: We introduce LLaMA, a collection of foundation language models ranging from 7B to 65B parameters. We train our models on trillions of tokens, and show that it is possible to train state-of-the-art models using publicly available datasets exclusively, without resorting to proprietary and inaccessible datasets. In particular, LLaMA-13B outperforms GPT-3 (175B) on most benchmarks, and LLaMA-65B is competitive with the best models, Chinchilla-70B and PaLM-540B. We release all our models to the research community.

Financial risk prediction results on FinBench. We report F1-scores of all ten datasets in FinBench and the overall average score “Avg”. In the “Training” column, “Grid Search” means we employ grid search to find the best hyper-parameter for the Tree-based models, “From Scratch” means we train the neural models (“NN for Table”) with random weight initialization, “Tune All” means we tune all the parameters in foundation models, and “Tune Last” means we tune only the last layer/block in foundation models. The best result on each dataset is highlighted in violetviolet color. The best result of each model class on each dataset is highlighted with underline.

We report the financial risk prediction F1-scores on FinBench in Table {{table:17ad9334-f56b-459d-94ce-87792cbcd548}} .
On average, FinPT outperforms tree-based methods and previous neural models by a large margin, especially when we apply FinPT to fully fine-tune large foundation models such as GPT-2, T5, and Flan-T5.
We provide a detailed comparison to analyze the performance of different models on FinBench as follows.

Since we transform the tabular data in different datasets into unified text profiles, it is possible to conduct Profile Tuning on all datasets and then evaluate the prediction performance on each test set. As shown in Figure {{figure:7d788c0c-360d-4fc9-b186-667750320342}} , we conduct this experiment with the best model in Table {{table:17ad9334-f56b-459d-94ce-87792cbcd548}} , i.e., FinPT on Flan-T5-Base. We can observe that training on all datasets in FinBench instead of training on each separate dataset brings consistent enhancement. The performance on Dataset cf2 benefits the most due to its original low score. F1-score on Dataset ld3 improves the least mainly because ld3 occupies the majority of the benchmark.


MediaSum: A Large-scale Media Interview Dataset for Dialogue Summarization

Abstract: This paper introduces MediaSum, a large-scale media interview dataset consisting of 463.6K transcripts with abstractive summaries. To create this dataset, we collect interview transcripts from NPR and CNN and employ the overview and topic descriptions as summaries. Compared with existing public corpora for dialogue summarization, our dataset is an order of magnitude larger and contains complex multi-party conversations from multiple domains. We conduct statistical analysis to demonstrate the unique positional bias exhibited in the transcripts of televised and radioed interviews. We also show that MediaSum can be used in transfer learning to improve a model’s performance on other dialogue summarization tasks.

SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarization

Abstract: This paper introduces the SAMSum Corpus, a new dataset with abstractive dialogue summaries. We investigate the challenges it poses for automated summarization by testing several models and comparing their results with those obtained on a corpus of news articles. We show that model-generated summaries of dialogues achieve higher ROUGE scores than the model-generated summaries of news – in contrast with human evaluators’ judgement. This suggests that a challenging task of abstractive dialogue summarization requires dedicated models and non-standard quality measures. To our knowledge, our study is the first attempt to introduce a high-quality chat-dialogues corpus, manually annotated with abstractive summarizations, which can be used by the research community for further studies.

Dialogue and meeting summarization datasets statistics. The number of words for dialogue, summary, turns, and speakers are averaged across the entire dataset. The meeting dataset statistics have been calculated and dialogue dataset statistics have been derived from zhu2021mediasum.

First, there is a scarcity of large-scale meeting datasets: There are a few meeting corpora, such as AMI {{cite:88cbcf8}} and ICSI {{cite:19eaba8}}, which are rather small, on the order of a few dozens of hours each as represented in Table {{table:7cbee99e-2192-4d12-8aa0-1edf924b4da1}} . Due to this fact, meeting summarization models are usually trained on news {{cite:c525ea0}}, stories {{cite:225eba6}}, Wikipedia {{cite:3a644be}}, {{cite:d5a65b0}}, and other textual corpora, relating poorly to meetings.


Sequential Recommendation with Graph Neural Networks

Abstract: Sequential recommendation aims to leverage users' historical behaviors to predict their next interaction. Existing works have not yet addressed two main challenges in sequential recommendation. First, user behaviors in their rich historical sequences are often implicit and noisy preference signals, they cannot sufficiently reflect users' actual preferences. In addition, users' dynamic preferences often change rapidly over time, and hence it is difficult to capture user patterns in their historical sequences. In this work, we propose a graph neural network model called SURGE (short forSeqUential Recommendation with Graph neural nEtworks) to address these two issues. Specifically, SURGE integrates different types of preferences in long-term user behaviors into clusters in the graph by re-constructing loose item sequences into tight item-item interest graphs based on metric learning. This helps explicitly distinguish users' core interests, by forming dense clusters in the interest graph. Then, we perform cluster-aware and query-aware graph convolutional propagation and graph pooling on the constructed graph. It dynamically fuses and extracts users' current activated core interests from noisy user behavior sequences. We conduct extensive experiments on both public and proprietary industrial datasets. Experimental results demonstrate significant performance gains of our proposed method compared to state-of-the-art methods. Further studies on sequence length confirm that our method can model long behavioral sequences effectively and efficiently.

Knowledge-Enhanced Graph Neural Networks for Sequential Recommendation

Abstract: With the rapid increase in the popularity of big data and internet technology, sequential recommendation has become an important method to help people find items they are potentially interested in. Traditional recommendation methods use only recurrent neural networks (RNNs) to process sequential data. Although effective, the results may be unable to capture both the semantic-based preference and the complex transitions between items adequately. In this paper, we model separated session sequences into session graphs and capture complex transitions using graph neural networks (GNNs). We further link items in interaction sequences with existing external knowledge base (KB) entities and integrate the GNN-based recommender with key-value memory networks (KV-MNs) to incorporate KB knowledge. Specifically, we set a key matrix to many relation embeddings that learned from KB, corresponding to many entity attributes, and set up a set of value matrices storing the semantic-based preferences of different users for the corresponding attribute. By using a hybrid of a GNN and KV-MN, each session is represented as the combination of the current interest (i.e., sequential preference) and the global preference (i.e., semantic-based preference) of that session. Extensive experiments on three public real-world datasets show that our method performs better than baseline algorithms consistently.

Inter-sequence Enhanced Framework for Personalized Sequential Recommendation

Abstract: Modeling the sequential correlation of users' historical interactions is essential in sequential recommendation. However, the majority of the approaches mainly focus on modeling the \emph{intra-sequence} item correlation within each individual sequence but neglect the \emph{inter-sequence} item correlation across different user interaction sequences. Though several studies have been aware of this issue, their method is either simple or implicit. To make better use of such information, we propose an inter-sequence enhanced framework for the Sequential Recommendation (ISSR). In ISSR, both inter-sequence and intra-sequence item correlation are considered. Firstly, we equip graph neural networks in the inter-sequence correlation encoder to capture the high-order item correlation from the user-item bipartite graph and the item-item graph. Then, based on the inter-sequence correlation encoder, we build GRU network and attention network in the intra-sequence correlation encoder to model the item sequential correlation within each individual sequence and temporal dynamics for predicting users' preferences over candidate items. Additionally, we conduct extensive experiments on three real-world datasets. The experimental results demonstrate the superiority of ISSR over many state-of-the-art methods and the effectiveness of the inter-sequence correlation encoder.

Memory Augmented Graph Neural Networks for Sequential Recommendation

Abstract: The chronological order of user-item interactions can reveal time-evolving and sequential user behaviors in many recommender systems. The items that users will interact with may depend on the items accessed in the past. However, the substantial increase of users and items makes sequential recommender systems still face non-trivial challenges: (1) the hardness of modeling the short-term user interests; (2) the difficulty of capturing the long-term user interests; (3) the effective modeling of item co-occurrence patterns. To tackle these challenges, we propose a memory augmented graph neural network (MA-GNN) to capture both the long- and short-term user interests. Specifically, we apply a graph neural network to model the item contextual information within a short-term period and utilize a shared memory network to capture the long-range dependencies between items. In addition to the modeling of user interests, we employ a bilinear function to capture the co-occurrence patterns of related items. We extensively evaluate our model on five real-world datasets, comparing with several state-of-the-art methods and using a variety of performance metrics. The experimental results demonstrate the effectiveness of our model for the task of Top-K sequential recommendation.

Dynamic Graph Neural Networks for Sequential Recommendation

Abstract: Modeling user preference from his historical sequences is one of the core problems of sequential recommendation. Existing methods in this field are widely distributed from conventional methods to deep learning methods. However, most of them only model users’ interests within their own sequences and ignore the dynamic collaborative signals among different user sequences, making it insufficient to explore users’ preferences. We take inspiration from dynamic graph neural networks to cope with this challenge, modeling the user sequence and dynamic collaborative signals into one framework. We propose a new method named Dynamic Graph Neural Network for Sequential Recommendation (DGSR), which connects different user sequences through a dynamic graph structure, exploring the interactive behavior of users and items with time and order information. Furthermore, we design a Dynamic Graph Recommendation Network to extract user's preferences from the dynamic graph. Consequently, the next-item prediction task in sequential recommendation is converted into a link prediction between the user node and the item node in a dynamic graph. Extensive experiments on four public benchmarks show that DGSR outperforms several state-of-the-art methods. Further studies demonstrate the rationality and effectiveness of modeling user sequences through a dynamic graph.

RetaGNN: Relational Temporal Attentive Graph Neural Networks for Holistic Sequential Recommendation

Abstract: Sequential recommendation (SR) is to accurately recommend a list of items for a user based on her current accessed ones. While new-coming users continuously arrive in the real world, one crucial task is to have inductive SR that can produce embeddings of users and items without re-training. Given user-item interactions can be extremely sparse, another critical task is to have transferable SR that can transfer the knowledge derived from one domain with rich data to another domain. In this work, we aim to present the holistic SR that simultaneously accommodates conventional, inductive, and transferable settings. We propose a novel deep learning-based model, Relational Temporal Attentive Graph Neural Networks (RetaGNN), for holistic SR. The main idea of RetaGNN is three-fold. First, to have inductive and transferable capabilities, we train a relational attentive GNN on the local subgraph extracted from a user-item pair, in which the learnable weight matrices are on various relations among users, items, and attributes, rather than nodes or edges. Second, long-term and short-term temporal patterns of user preferences are encoded by a proposed sequential self-attention mechanism. Third, a relation-aware regularization term is devised for better training of RetaGNN. Experiments conducted on MovieLens, Instagram, and Book-Crossing datasets exhibit that RetaGNN can outperform state-of-the-art methods under conventional, inductive, and transferable settings. The derived attention weights also bring model explainability.

Continuous-Time Sequential Recommendation with Temporal Graph Collaborative Transformer

Abstract: In order to model the evolution of user preference, we should learn user/item embeddings based on time-ordered item purchasing sequences, which is defined as Sequential Recommendation (SR) problem. Existing methods leverage sequential patterns to model item transitions. However, most of them ignore crucial temporal collaborative signals, which are latent in evolving user-item interactions and coexist with sequential patterns. Therefore, we propose to unify sequential patterns and temporal collaborative signals to improve the quality of recommendation, which is rather challenging. Firstly, it is hard to simultaneously encode sequential patterns and collaborative signals. Secondly, it is non-trivial to express the temporal effects of collaborative signals. Hence, we design a new framework Temporal Graph Sequential Recommender (TGSRec) upon our defined continuous-time bi-partite graph. We propose a novel Temporal Collaborative Trans-former (TCT) layer in TGSRec, which advances the self-attention mechanism by adopting a novel collaborative attention. TCT layer can simultaneously capture collaborative signals from both users and items, as well as considering temporal dynamics inside sequential patterns. We propagate the information learned fromTCTlayerover the temporal graph to unify sequential patterns and temporal collaborative signals. Empirical results on five datasets show that TGSRec significantly outperforms other baselines, in average up to 22.5% and 22.1%absolute improvements in Recall@10and MRR, respectively.

Discovering Collaborative Signals for Next POI Recommendation with Iterative Seq2Graph Augmentation

Abstract: Being an indispensable component in location-based social networks, next point-of-interest (POI) recommendation recommends users unexplored POIs based on their recent visiting histories. However, existing work mainly models check-in data as isolated POI sequences, neglecting the crucial collaborative signals from cross-sequence check-in information. Furthermore, the sparse POI-POI transitions restrict the ability of a model to learn effective sequential patterns for recommendation. In this paper, we propose Sequence-to-Graph (Seq2Graph) augmentation for each POI sequence, allowing collaborative signals to be propagated from correlated POIs belonging to other sequences. We then devise a novel Sequence-to-Graph POI Recommender (SGRec), which jointly learns POI embeddings and infers a user's temporal preferences from the graph-augmented POI sequence. To overcome the sparsity of POI-level interactions, we further infuse category-awareness into SGRec with a multi-task learning scheme that captures the denser category-wise transitions. As such, SGRec makes full use of the collaborative signals for learning expressive POI representations, and also comprehensively uncovers multi-level sequential patterns for user preference modelling. Extensive experiments on two real-world datasets demonstrate the superiority of SGRec against state-of-the-art methods in next POI recommendation.

STP-UDGAT: Spatial-Temporal-Preference User Dimensional Graph Attention Network for Next POI Recommendation

Abstract: Next Point-of-Interest (POI) recommendation is a longstanding problem across the domains of Location-Based Social Networks (LBSN) and transportation. Recent Recurrent Neural Network (RNN) based approaches learn POI-POI relationships in a local view based on independent user visit sequences. This limits the model's ability to directly connect and learn across users in a global view to recommend semantically trained POIs. In this work, we propose a Spatial-Temporal-Preference User Dimensional Graph Attention Network (STP-UDGAT), a novel explore-exploit model that concurrently exploits personalized user preferences and explores new POIs in global spatial-temporal-preference (STP) neighbourhoods, while allowing users to selectively learn from other users. In addition, we propose random walks as a masked self-attention option to leverage the STP graphs' structures and find new higher-order POI neighbours during exploration. Experimental results on six real-world datasets show that our model significantly outperforms baseline and state-of-the-art methods.

Details of GNN models for sequential recommendation.

Besides the challenges of graph construction and network design, the methods of GNN-based sequential recommendation should carefully design how to extract the sequential patterns in users’ behaviors, as shown in Table {{table:ec815cce-6624-49c5-b3a2-0565ce2a97ec}} .
Since GNN has the ability of high-order relationship modeling by aggregating information from neighbor nodes, after fusing multiple sequences into one graph, it can learn representations of both users and items in different sequences, which can't be accomplished by the Markov model or recurrent neural network. Wang et al. {{cite:369ca2b}} propose a simple method that directly converts the sequence information into directed edges on the graph and then uses GNN to learn representations. Liu et al.  {{cite:8cb7426}} construct a user-item bipartite graph and an item-item graph at the same time, where the edges of the item-item graph indicate co-occurrence in a sequence, with edge weights assigned according to the number of occurrences. The representations learned by GNN are used in the final recommendation through the recurrent neural network. Different from directly converting the temporal sequence into directed edges in the graph, DGSR {{cite:daeaeed}} and TGSRec {{cite:f98dfe4}} consider the timestamps in the sequence in the process of graph construction. In the graph, each edge represents the interaction between the user and the item, along with the corresponding time attribute. Then perform convolution operations on the temporal graph to learn the representations of users and items. GES-SASRec {{cite:3769e5d}} and SGRec {{cite:7d31cd5}} focus on the learning of item representations. For an item in a sequence, GES-SASRec {{cite:3769e5d}} considers the next item of this item in other sequences, and SGRec {{cite:7d31cd5}} not only considers the next item but also considers the previous one. By aggregating the items before and after the target item in different sequences, the representation of the item is enhanced. GPR {{cite:a759780}} and GME {{cite:489902c}} construct edges between items by considering the frequency of consecutive occurrences or occurrences in the same sequence to enhance the representation. Some works are more complicated. For example, RetaGNN {{cite:6c49d9d}} considers the attributes of the items when constructing the graph, while STP-UDGAT {{cite:f21e77a}} considers the geographic location, timestamp, and frequency in the POI recommendation. Table {{table:ec815cce-6624-49c5-b3a2-0565ce2a97ec}}  summarizes the above works.
Current networks of graph neural network-based recommendation are still highly relying on an additional sequence model such as RNN, Transformer, etc., for sequential modeling. The ability of graphs in sequential modeling, such as directed edges, dynamic nodes, requires more research efforts.

{{cite:daeaeed}}

{{cite:daeaeed}}


Key-Value Memory Networks for Directly Reading Documents

Abstract: Directly reading documents and being able to answer questions from them is an unsolved challenge. To avoid its inherent difficulty, question answering (QA) has been directed towards using Knowledge Bases (KBs) instead, which has proven effective. Unfortunately KBs often suffer from being too restrictive, as the schema cannot support certain types of answers, and too sparse, e.g. Wikipedia contains much more information than Freebase. In this work we introduce a new method, Key-Value Memory Networks, that makes reading documents more viable by utilizing different encodings in the addressing and output stages of the memory read operation. To compare using KBs, information extraction or Wikipedia documents directly in a single framework we construct an analysis tool, WikiMovies, a QA dataset that contains raw text alongside a preprocessed KB, in the domain of movies. Our method reduces the gap between all three settings. It also achieves state-of-the-art results on the existing WikiQA benchmark.

SQuAD: 100,000+ Questions for Machine Comprehension of Text

Abstract: We present the Stanford Question Answering Dataset (SQuAD), a new reading comprehension dataset consisting of 100,000+ questions posed by crowdworkers on a set of Wikipedia articles, where the answer to each question is a segment of text from the corresponding reading passage. We analyze the dataset to understand the types of reasoning required to answer the questions, leaning heavily on dependency and constituency trees. We build a strong logistic regression model, which achieves an F1 score of 51.0%, a significant improvement over a simple baseline (20%). However, human performance (86.8%) is much higher, indicating that the dataset presents a good challenge problem for future research. The dataset is freely available at https://stanford-qa.com

Quasar: Datasets for Question Answering by Search and Reading

Abstract: We present two new large-scale datasets aimed at evaluating systems designed to comprehend a natural language query and extract its answer from a large corpus of text. The Quasar-S dataset consists of 37000 cloze-style (fill-in-the-gap) queries constructed from definitions of software entity tags on the popular website Stack Overflow. The posts and comments on the website serve as the background corpus for answering the cloze questions. The Quasar-T dataset consists of 43000 open-domain trivia questions and their answers obtained from various internet sources. ClueWeb09 serves as the background corpus for extracting these answers. We pose these datasets as a challenge for two related subtasks of factoid Question Answering: (1) searching for relevant pieces of text that include the correct answer to a query, and (2) reading the retrieved text to answer the query. We also describe a retrieval system for extracting relevant sentences and documents from the corpus given a query, and include these in the release for researchers wishing to only focus on (2). We evaluate several baselines on both datasets, ranging from simple heuristics to powerful neural models, and show that these lag behind human performance by 16.4% and 32.1% for Quasar-S and -T respectively. The datasets are available at this https URL .

TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension

Abstract: We present TriviaQA, a challenging reading comprehension dataset containing over 650K question-answer-evidence triples. TriviaQA includes 95K question-answer pairs authored by trivia enthusiasts and independently gathered evidence documents, six per question on average, that provide high quality distant supervision for answering the questions. We show that, in comparison to other recently introduced large-scale datasets, TriviaQA (1) has relatively complex, compositional questions, (2) has considerable syntactic and lexical variability between questions and corresponding answer-evidence sentences, and (3) requires more cross sentence reasoning to find answers. We also present two baseline algorithms: a feature-based classifier and a state-of-the-art neural network, that performs well on SQuAD reading comprehension. Neither approach comes close to human performance (23% and 40% vs. 80%), suggesting that TriviaQA is a challenging testbed that is worth significant future study.

HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering

Abstract: Existing question answering (QA) datasets fail to train QA systems to perform complex reasoning and provide explanations for answers. We introduce HotpotQA, a new dataset with 113k Wikipedia-based question-answer pairs with four key features: (1) the questions require finding and reasoning over multiple supporting documents to answer; (2) the questions are diverse and not constrained to any pre-existing knowledge bases or knowledge schemas; (3) we provide sentence-level supporting facts required for reasoning, allowing QA systems to reason with strong supervision and explain the predictions; (4) we offer a new type of factoid comparison questions to test QA systems' ability to extract relevant facts and perform necessary comparison. We show that HotpotQA is challenging for the latest QA systems, and the supporting facts enable models to improve performance and make explainable predictions.

ELI5: Long Form Question Answering

Abstract: We introduce the first large-scale corpus for long form question answering, a task requiring elaborate and in-depth answers to open-ended questions. The dataset comprises 270K threads from the Reddit forum “Explain Like I’m Five” (ELI5) where an online community provides answers to questions which are comprehensible by five year olds. Compared to existing datasets, ELI5 comprises diverse questions requiring multi-sentence answers. We provide a large set of web documents to help answer the question. Automatic and human evaluations show that an abstractive model trained with a multi-task objective outperforms conventional Seq2Seq, language modeling, as well as a strong extractive baseline.However, our best model is still far from human performance since raters prefer gold responses in over 86% of cases, leaving ample opportunity for future improvement.

Open-domain question answering benchmarks.

Theoretically, because of the end-to-end nature of ODQA, any existing QA dataset that contains objective question-answer pairs can be curated to serve in this scenario by pairing it with a large external knowledge source that can be used to answer its questions, e.g., Wikipedia.
Hence, besides those datasets that were originally designed for ODQA (e.g., CuratedTREC{{cite:ec8f102}}, Quasar-T{{cite:cdf42ad}}), many famous large-scale MRC datasets(e.g., SQuAD{{cite:a7189d0}}, TRIVIA QA{{cite:079f06f}}, Natural Questions{{cite:365aaba}}, etc.) were curated to serve as benchmarks for ODQA systems(e.g., DrQA{{cite:b2d6cfe}}, ORQA{{cite:d1e0cec}}, etc.).
Next, we will introduce the most widely-used benchmarks in modern ODQA systems. The information on these datasets is summarized in Table {{table:c8b48a14-6748-41eb-97fa-2dd1c5d05450}} .


Btrim: A fast, lightweight adapter and quality trimming program for next-generation sequencing technologies

Abstract: Btrim is a fast and lightweight software to trim adapters and low quality regions in reads from ultra high-throughput next-generation sequencing machines. It also can reliably identify barcodes and assign the reads to the original samples. Based on a modified Myers's bit-vector dynamic programming algorithm, Btrim can handle indels in adapters and barcodes. It removes low quality regions and trims off adapters at both or either end of the reads. A typical trimming of 30M reads with two sets of adapter pairs can be done in about a minute with a small memory footprint. Btrim is a versatile stand-alone tool that can be used as the first step in virtually all next-generation sequence analysis pipelines. The program is available at \url{http://graphics.med.yale.edu/trim/}.

Trimmomatic: a flexible trimmer for Illumina sequence data

Abstract: Motivation: Although many next-generation sequencing (NGS) read preprocessing tools already existed, we could not find any tool or combination of tools that met our requirements in terms of flexibility, correct handling of paired-end data and high performance. We have developed Trimmomatic as a more flexible and efficient preprocessing tool, which could correctly handle paired-end data. Results: The value of NGS read preprocessing is demonstrated for both reference-based and reference-free tasks. Trimmomatic is shown to produce output that is at least competitive with, and in many cases superior to, that produced by other tools, in all scenarios tested. Availability and implementation: Trimmomatic is licensed under GPL V3. It is cross-platform (Java 1.5+ required) and available at http://www.usadellab.org/cms/index.php?page=trimmomatic Contact: usadel@bio1.rwth-aachen.de Supplementary information: Supplementary data are available at Bioinformatics online.

Atropos: specific, sensitive, and speedy trimming of sequencing reads

Abstract: A key step in the transformation of raw sequencing reads into biological insights is the trimming of adapter sequences and low-quality bases. Read trimming has been shown to increase the quality and reliability while decreasing the computational requirements of downstream analyses. Many read trimming software tools are available; however, no tool simultaneously provides the accuracy, computational efficiency, and feature set required to handle the types and volumes of data generated in modern sequencing-based experiments. Here we introduce Atropos and show that it trims reads with high sensitivity and specificity while maintaining leading-edge speed. Compared to other state-of-the-art read trimming tools, Atropos achieves significant increases in trimming accuracy while remaining competitive in execution times. Furthermore, Atropos maintains high accuracy even when trimming data with elevated rates of sequencing errors. The accuracy, high performance, and broad feature set offered by Atropos makes it an appropriate choice for the pre-processing of Illumina, ABI SOLiD, and other current-generation short-read sequencing datasets. Atropos is open source and free software written in Python (3.3+) and available at https://github.com/jdidion/atropos.

Main methodologies for the removal of adapters and low quality sequences. The methodologies are ordered chronologically considering the year of publication.

Therefore, for this task, the main methodologies and implementations (software) available in the literature are presented in Table {{table:751bbc2e-fa31-4dc1-bd12-947ae0c283e7}} .


Playing Atari with Deep Reinforcement Learning

Abstract: We present the first deep learning model to successfully learn control policies directly from high-dimensional sensory input using reinforcement learning. The model is a convolutional neural network, trained with a variant of Q-learning, whose input is raw pixels and whose output is a value function estimating future rewards. We apply our method to seven Atari 2600 games from the Arcade Learning Environment, with no adjustment of the architecture or learning algorithm. We find that it outperforms all previous approaches on six of the games and surpasses a human expert on three of them.

Deep Recurrent Q-Learning for Partially Observable MDPs

Abstract: Deep Reinforcement Learning has yielded proficient controllers for complex tasks. However, these controllers have limited memory and rely on being able to perceive the complete game screen at each decision point. To address these shortcomings, this article investigates the effects of adding recurrency to a Deep Q-Network (DQN) by replacing the first post-convolutional fully-connected layer with a recurrent LSTM. The resulting \textit{Deep Recurrent Q-Network} (DRQN), although capable of seeing only a single frame at each timestep, successfully integrates information through time and replicates DQN's performance on standard Atari games and partially observed equivalents featuring flickering game screens. Additionally, when trained with partial observations and evaluated with incrementally more complete observations, DRQN's performance scales as a function of observability. Conversely, when trained with full observations and evaluated with partial observations, DRQN's performance degrades less than DQN's. Thus, given the same length of history, recurrency is a viable alternative to stacking a history of frames in the DQN's input layer and while recurrency confers no systematic advantage when learning to play the game, the recurrent net can better adapt at evaluation time if the quality of observations changes.

Continuous Deep Q-Learning with Model-based Acceleration

Abstract: Model-free reinforcement learning has been successfully applied to a range of challenging problems, and has recently been extended to handle large neural network policies and value functions. However, the sample complexity of model-free algorithms, particularly when using high-dimensional function approximators, tends to limit their applicability to physical systems. In this paper, we explore algorithms and representations to reduce the sample complexity of deep reinforcement learning for continuous control tasks. We propose two complementary techniques for improving the efficiency of such algorithms. First, we derive a continuous variant of the Q-learning algorithm, which we call normalized adantage functions (NAF), as an alternative to the more commonly used policy gradient and actor-critic methods. NAF representation allows us to apply Q-learning with experience replay to continuous tasks, and substantially improves performance on a set of simulated robotic control tasks. To further improve the efficiency of our approach, we explore the use of learned models for accelerating model-free reinforcement learning. We show that iteratively refitted local linear models are especially effective for this, and demonstrate substantially faster learning on domains where such models are applicable.

Deep Q-Learning architectures in testbed

Models 1-4 (Figure {{table:1bc1acf4-e018-4a3f-a7bd-1fb3915b0418}} ) are the most commonly used DQN architectures found in literature. Model 5 shows great potential in continuous environments, comparable to environments from Chapter . Models 6 and 7 are two novel approaches using Capsule Layers in conjunction with Convolution layers {{cite:d0b4d04}}, {{cite:5fb7872}}.

Models 1-7 are implemented in the Keras/Tensorflow framework according to the definitions found in the illustrated papers. Table {{table:96dcd811-b3cb-4415-8dfe-f674ddc46001}}  shows the architecture of the DQN models found in Table {{table:1bc1acf4-e018-4a3f-a7bd-1fb3915b0418}} . Filter and stride count is intentionally left out because these are considered as hyper-parameters. Hyperparameters are manually tuned by trial and error. Table {{table:e62a2d1f-01a1-4ea4-88fe-4addc63715f5}}  outlines the parameters that are tuned individually for each of the architectures.

RL algorithms are known to be computationally intensive and are thus difficult to train for environments with large state-spaces {{cite:0386e41}}. Models are trained using hardware specified in Appendix . Chapter  proposed 7 DQN architectures that could potentially control an agent well within the environments. Model 1 and 6 from Table {{table:1bc1acf4-e018-4a3f-a7bd-1fb3915b0418}}  was selected as the primary research area to limit the scope of this thesis Training time for 7 models in 5 environments: {{formula:a18ecee9-2300-4c97-8fe4-90955bb9bee6}}  days (Approx 7 days per experiment). To increase training stability for all environments, hyper-parameters from Table {{table:e62a2d1f-01a1-4ea4-88fe-4addc63715f5}}  is tuned further per environment. The datasets are populated with 20% artificial training data, generated from CCDN. Table {{table:86f7b7a4-a594-4774-a6c0-34ad275a4834}}  illustrates updated hyper-parameters that performed best when experimenting with CapsNet and ConvNet based models. The DQN models use SDG to optimize its parameters. Initial training data is sampled using random-play strategies, gradually moving into exploitation using {{formula:ce7abf1a-965e-48f2-896c-7531424f0525}} -greedy.


Self-attention and multi-scale gradients

A Self-attention Guided Multi-scale Gradient GAN for Diversified X-ray Image Synthesis

Abstract: Imbalanced image datasets are commonly available in the domain of biomedical image analysis. Biomedical images contain diversified features that are significant in predicting targeted diseases. Generative Adversarial Networks (GANs) are utilized to address the data limitation problem via the generation of synthetic images. Training challenges such as mode collapse, non-convergence, and instability degrade a GAN's performance in synthesizing diversified and high-quality images. In this work, MSG-SAGAN, an attention-guided multi-scale gradient GAN architecture is proposed to model the relationship between long-range dependencies of biomedical image features and improves the training performance using a flow of multi-scale gradients at multiple resolutions in the layers of generator and discriminator models. The intent is to reduce the impact of mode collapse and stabilize the training of GAN using an attention mechanism with multi-scale gradient learning for diversified X-ray image synthesis. Multi-scale Structural Similarity Index Measure (MS-SSIM) and Frechet Inception Distance (FID) are used to identify the occurrence of mode collapse and evaluate the diversity of synthetic images generated. The proposed architecture is compared with the multi-scale gradient GAN (MSG-GAN) to assess the diversity of generated synthetic images. Results indicate that the MSG-SAGAN outperforms MSG-GAN in synthesizing diversified images as evidenced by the MS-SSIM and FID scores.

Data Augmentation for Skin Lesion using Self-Attention based Progressive Generative Adversarial Network

Abstract: Deep Neural Networks (DNNs) show a significant impact on medical imaging. One significant problem with adopting DNNs for skin cancer classification is that the class frequencies in the existing datasets are imbalanced. This problem hinders the training of robust and well-generalizing models. Data Augmentation addresses this by using existing data more effectively. However, standard data augmentation implementations are manually designed and produce only limited reasonably alternative data. Instead, Generative Adversarial Networks (GANs) is utilized to generate a much broader set of augmentations. This paper proposes a novel enhancement for the progressive generative adversarial networks (PGAN) using self-attention mechanism. Self-attention mechanism is used to directly model the long-range dependencies in the feature maps. Accordingly, self-attention complements PGAN to generate fine-grained samples that comprise clinically-meaningful information. Moreover, the stabilization technique was applied to the enhanced generative model. To train the generative models, ISIC 2018 skin lesion challenge dataset was used to synthesize highly realistic skin lesion samples for boosting further the classification result. We achieve an accuracy of 70.1% which is 2.8% better than the non-augmented one of 67.3%.

Data Augmentation For Medical MR Image Using Generative Adversarial Networks

Abstract: Computer-assisted diagnosis (CAD) based on deep learning has become a crucial diagnostic technology in the medical industry, effectively improving diagnosis accuracy. However, the scarcity of brain tumor Magnetic Resonance (MR) image datasets causes the low performance of deep learning algorithms. The distribution of transformed images generated by traditional data augmentation (DA) intrinsically resembles the original ones, resulting in a limited performance in terms of generalization ability. This work improves Progressive Growing of GANs with a structural similarity loss function (PGGAN-SSIM) to solve image blurriness problems and model collapse. We also explore other GAN-based data augmentation to demonstrate the effectiveness of the proposed model. Our results show that PGGAN-SSIM successfully generates 256x256 realistic brain tumor MR images which fill the real image distribution uncovered by the original dataset. Furthermore, PGGAN-SSIM exceeds other GAN-based methods, achieving promising performance improvement in Frechet Inception Distance (FID) and Multi-scale Structural Similarity (MS-SSIM).

Modified architectures of GAN variants for biomedical image generation

Unconditional GANs suffer from intra-class mode collapse while conditional GANs suffer from intra-class as well as inter-class problems. Table {{table:e017f23d-a90f-4443-a5ec-946869cb0a49}}  summarizes the GAN variants with their contributions to specific biomedical imaging modalities. These contributions are application-specific and partially alleviate the mode collapse problem for the given variant of GANs.


Complicated Table Structure Recognition

Abstract: The task of table structure recognition aims to recognize the internal structure of a table, which is a key step to make machines understand tables. Currently, there are lots of studies on this task for different file formats such as ASCII text and HTML. It also attracts lots of attention to recognize the table structures in PDF files. However, it is hard for the existing methods to accurately recognize the structure of complicated tables in PDF files. The complicated tables contain spanning cells which occupy at least two columns or rows. To address the issue, we propose a novel graph neural network for recognizing the table structure in PDF files, named GraphTSR. Specifically, it takes table cells as input, and then recognizes the table structures by predicting relations among cells. Moreover, to evaluate the task better, we construct a large-scale table structure recognition dataset from scientific papers, named SciTSR, which contains 15,000 tables from PDF files and their corresponding structure labels. Extensive experiments demonstrate that our proposed model is highly effective for complicated tables and outperforms state-of-the-art baselines over a benchmark dataset and our new constructed dataset.

CascadeTabNet: An approach for end to end table detection and structure recognition from image-based documents

Abstract: An automatic table recognition method for interpretation of tabular data in document images majorly involves solving two problems of table detection and table structure recognition. The prior work involved solving both problems independently using two separate approaches. More recent works signify the use of deep learning-based solutions while also attempting to design an end to end solution. In this paper, we present an improved deep learning-based end to end approach for solving both problems of table detection and structure recognition using a single Convolution Neural Network (CNN) model. We propose CascadeTabNet: a Cascade mask Region-based CNN High-Resolution Network (Cascade mask R-CNN HRNet) based model that detects the regions of tables and recognizes the structural body cells from the detected tables at the same time. We evaluate our results on ICDAR 2013, ICDAR 2019 and Table-Bank public datasets. We achieved 3rd rank in ICDAR 2019 post-competition results for table detection while attaining the best accuracy results for the ICDAR 2013 and Table-Bank dataset. We also attain the highest accuracy results on the ICDAR 2019 table structure recognition dataset. Additionally, we demonstrate effective transfer learning and image augmentation techniques that enable CNNs to achieve very accurate table detection results. Code and dataset has been made available at: https://github.com/DevashishPrasad/CascadeTabNet

The Benefits of Close-Domain Fine-Tuning for Table Detection in Document Images

Abstract: A correct localisation of tables in a document is instrumental for determining their structure and extracting their contents; therefore, table detection is a key step in table understanding. Nowadays, the most successful methods for table detection in document images employ deep learning algorithms; and, particularly, a technique known as fine-tuning. In this context, such a technique exports the knowledge acquired to detect objects in natural images to detect tables in document images. However, there is only a vague relation between natural and document images, and fine-tuning works better when there is a close relation between the source and target task. In this paper, we show that it is more beneficial to employ fine-tuning from a closer domain. To this aim, we train different object detection algorithms (namely, Mask R-CNN, RetinaNet, SSD and YOLO) using the TableBank dataset (a dataset of images of academic documents designed for table detection and recognition), and fine-tune them for several heterogeneous table detection datasets. Using this approach, we considerably improve the accuracy of the detection models fine-tuned from natural images (in mean a 17%, and, in the best case, up to a 60%).

CDeC-Net: Composite Deformable Cascade Network for Table Detection in Document Images

Abstract: Localizing page elements/objects such as tables, figures, equations, etc. is the primary step in extracting information from document images. We propose a novel end-to-end trainable deep network, (cnec-xet) for detecting tables present in the documents. The proposed network consists of a multistage extension of Mask R-CNN with a dual backbone having deformable convolution for detecting tables varying in scale with high detection accuracy at higher IoU threshold. We empirically evaluate CDeC-Net on the publicly available benchmark datasets with extensive experiments. Our solution has three important properties: (i) a single trained model CDeC-Net‡that performs well across all the popular benchmark datasets; (ii) we report excellent performances across multiple, including higher, thresholds of IoU; (iii) by following the same protocol of the recent papers for each of the benchmarks, we consistently demonstrate the superior quantitative performance. Our code and models are publicly available at https://github.com/mdv3101/CDeCNet for enabling reproducibility of the results.

Open source code for most of the studies articles in Table Detection

Several open source frameworks for creating generic deep learning models, most of which are written in Python, are available online, including TensorFlow, Keras, PyTorch, and MXNet.The open-source projects for table detection and structure recognition are summarized in Table {{table:2aa3cbdd-44af-40c1-8d21-fcefb3d70501}} . Many of the authors have also made open-source implementations of their proposed models available. TensorFlow and PyTorch are the most often utilized frameworks in these open source projects.


Localization based Sequential Grouping for Continuous Speech Separation

Abstract: This study investigates robust speaker localization for continuous speech separation and speaker diarization, where we use speaker directions to group non-contiguous segments of the same speaker. Assuming that speakers do not move and are located in different directions, the direction of arrival (DOA) information provides an informative cue for accurate sequential grouping and speaker diarization. Our system is block-online in the following sense. Given a block of frames with at most two speakers, we apply a two-speaker separation model to separate (and enhance) the speakers, estimate the DOA of each separated speaker, and group the separation results across blocks based on the DOA estimates. Speaker diarization and speaker-attributed speech recognition results on the LibriCSS corpus demonstrate the effectiveness of the proposed algorithm.

Integration of Speech Separation, Diarization, and Recognition for Multi-Speaker Meetings: System Description, Comparison, and Analysis

Abstract: Multi-speaker speech recognition of unsegmented recordings has diverse applications such as meeting transcription and automatic subtitle generation. With technical advances in systems dealing with speech separation, speaker diarization, and automatic speech recognition (ASR) in the last decade, it has become possible to build pipelines that achieve reasonable error rates on this task. In this paper, we propose an end-to-end modular system for the LibriCSS meeting data, which combines independently trained separation, diarization, and recognition components, in that order. We study the effect of different state-of-the-art methods at each stage of the pipeline, and report results using task-specific metrics like SDR and DER, as well as downstream WER. Experiments indicate that the problem of overlapping speech for diarization and ASR can be effectively mitigated with the presence of a well-trained separation module. Our best system achieves a speaker-attributed WER of 12.7%, which is close to that of a non-overlapping ASR.

Transcribe-to-Diarize: Neural Speaker Diarization for Unlimited Number of Speakers Using End-to-End Speaker-Attributed ASR

Abstract: This paper presents Transcribe-to-Diarize, a new approach for neural speaker diarization that uses an end-to-end (E2E) speaker-attributed automatic speech recognition (SA-ASR). The E2E SA-ASR is a joint model that was recently proposed for speaker counting, multi-talker speech recognition, and speaker identification from monaural audio that contains overlapping speech. Although the E2E SA-ASR model originally does not estimate any time-related information, we show that the start and end times of each word can be estimated with sufficient accuracy from the internal state of the E2E SA-ASR by adding a small number of learnable parameters. Similar to the target-speaker voice activity detection (TS-VAD)-based diarization method, the E2E SA-ASR model is applied to estimate speech activity of each speaker while it has the advantages of (i) handling unlimited number of speakers, (ii) leveraging linguistic information for speaker diarization, and (iii) simultaneously generating speaker-attributed transcriptions. Experimental results on the LibriCSS and AMI corpora show that the proposed method achieves significantly better diarization error rate than various existing speaker diarization methods when the number of speakers is unknown, and achieves a comparable performance to TS-VAD when the number of speakers is given in advance. The proposed method simultaneously generates speaker-attributed transcription with state-of-the-art accuracy.

Speaker Activity Driven Neural Speech Extraction

Abstract: Target speech extraction, which extracts the speech of a target speaker in a mixture given auxiliary speaker clues, has recently received increased interest. Various clues have been investigated such as pre-recorded enrollment utterances, direction information, or video of the target speaker. In this paper, we explore the use of speaker activity information as an auxiliary clue for single-channel neural network-based speech extraction. We propose a speaker activity driven speech extraction neural network (ADEnet) and show that it can achieve performance levels competitive with enrollment-based approaches, without the need for pre-recordings. We further demonstrate the potential of the proposed approach for processing meeting-like recordings, where the speaker activity is obtained from a diarization system. We show that this simple yet practical approach can successfully extract speakers after diarization, which results in improved ASR performance, especially in high overlapping conditions, with a relative word error rate reduction of up to 25%.

Performance Comparisons of Speaker-attributed ASR Systems on LibriCSS Dataset.

We compare the speaker-attributed performance of the proposed SSND model with other representative algorithms in Table {{table:dcbd672f-1bec-4a96-802d-1f5132d41d01}} . The CSS-based MISO-BF-MISO system coupled with DOA-based diarization and a hybrid ASR model {{cite:943cf27}} reports a 12.98% cpWER {{cite:0b1816e}}. Another system outlined in {{cite:943cf27}} employs mask-based MVDR, x-vector+SC diarization, and E2E ASR, and achieves a 12.7% cpWER {{cite:943cf27}}. A single-channel E2E speaker-attributed ASR system {{cite:d01db6e}} derives diarization estimates from the internal state of the recognizer, and achieves a 11.6% cpWER.
The system in {{cite:cebd027}} uses TS-VAD for diarization and reports the cpWER scores of 18.8% and 11.2% for single- and multi-channel setups utilizing a speakerbeam and GSS, respectively.
Finally, the TS-SEP system {{cite:fa54921}} reaches the cpWER values of 6.42% and 5.36% with E2E and E2E-SSL ASR, respectively. The cpWER results in Table {{table:dcbd672f-1bec-4a96-802d-1f5132d41d01}}  demonstrate that our proposed SSND model surpasses all previous results, achieving the remarkable cpWER scores of 5.13% and 3.22% for E2E and E2E-SSL, respectively. Our results establish a new state-of-the-art benchmark for speaker-attributed ASR on the LibriCSS dataset.


Some behavior-specific subsequences of items

Modeling behavior from behavior transition relations, containing homogeneous behavior transitions intra each kind of behavior-specific subsequences.

Focus on the task of voucher redemption rate prediction and model the relationship between multiple behaviors and vouchers effectively.

Beyond Clicks: Modeling Multi-Relational Item Graph for Session-Based Target Behavior Prediction

Abstract: Session-based target behavior prediction aims to predict the next item to be interacted with specific behavior types (e.g., clicking). Although existing methods for session-based behavior prediction leverage powerful representation learning approaches to encode items’ sequential relevance in a low-dimensional space, they suffer from several limitations. Firstly, they focus on only utilizing the same type of user behavior for prediction, but ignore the potential of taking other behavior data as auxiliary information. This is particularly crucial when the target behavior is sparse but important (e.g., buying or sharing an item). Secondly, item-to-item relations are modeled separately and locally in one behavior sequence, and they lack a principled way to globally encode these relations more effectively. To overcome these limitations, we propose a novel Multi-relational Graph Neural Network model for Session-based target behavior Prediction, namely MGNN-SPred for short. Specifically, we build a Multi-Relational Item Graph (MRIG) based on all behavior sequences from all sessions, involving target and auxiliary behavior types. Based on MRIG, MGNN-SPred learns global item-to-item relations and further obtains user preferences w.r.t. current target and auxiliary behavior sequences, respectively. In the end, MGNN-SPred leverages a gating mechanism to adaptively fuse user representations for predicting next item interacted with target behavior. The extensive experiments on two real-world datasets demonstrate the superiority of MGNN-SPred by comparing with state-of-the-art session-based prediction methods, validating the benefits of leveraging auxiliary behavior and learning item-to-item relations over MRIG.

DMBGN: Deep Multi-Behavior Graph Networks for Voucher Redemption Rate Prediction

Abstract: In E-commerce, vouchers are important marketing tools to enhance users' engagement and boost sales and revenue. The likelihood that a user redeems a voucher is a key factor in voucher distribution decision. User-item Click-Through-Rate (CTR) models are often applied to predict the user-voucher redemption rate. However, the voucher scenario involves more complicated relations among users, items and vouchers. The users' historical behavior in a voucher collection activity reflects users' voucher usage patterns, which is nevertheless overlooked by the CTR-based solutions. In this paper, we propose a Deep Multi-behavior Graph Networks (DMBGN) to shed light on this field for the voucher redemption rate prediction. The complex structural user-voucher-item relationships are captured by a User-Behavior Voucher Graph (UVG). User behavior happening both before and after voucher collection is taken into consideration, and a high-level representation is extracted by Higher-order Graph Neural Networks. On top of a sequence of UVGs, an attention network is built which can help to learn users' long-term voucher redemption preference. Extensive experiments on three large-scale production datasets demonstrate the proposed DMBGN model is effective, with 10% to 16% relative AUC improvement over Deep Neural Networks (DNN), and 2% to 4% AUC improvement over Deep Interest Network (DIN). Source code and a sample dataset are made publicly available to facilitate future research.

Data & modeling perspectives and features used in works based on GNN learning architecture.

In MBSR, there are lots of works achieving great recommendation performance based on GNN, such as MGNN-SPred {{cite:ef27bb1}}, DMBGN {{cite:51fcf28}}, GPG4HSR {{cite:eafaa65}}, BGNN {{cite:2fe2021}} and BA-GNN {{cite:c5008f0}}. We describe some of them in detail below, and summarize the data perspective, the modeling perspective and the characteristics of these works in Table {{table:e943be85-8993-49eb-8c78-55681da7ee8c}} .


Transfers discrminitive information from one or more teacher networks to a student network for recovering the missing data

Calculates similarity metrics across available modalities and optimizes the mutual information

Employs GAN and its modifications in the missing modality model framework

Important domain knowledge from the full-modal network is usually not gained by the student model {{cite:f41ca5a}}.Exhibits significant training expenses when working with complex and large teacher networks. Capacity mismatch between teacher and student could result in no improvement in student network's outcomes {{cite:2a83768}}.

When insufficient number of modalities are available, applying this strategy will not necessarily assure that the lost data is recovered since there won't be enough features to reconstruct the missing data {{cite:24db7bc}}. Some earlier models tended to confine the network's structure {{cite:5ec7be4}}.

Could generate undesirable imputation noise, when imputing or synthesizing the missing modality {{cite:1c38cbd}}. GANs might show a non-converging nature. Extensive training costs. Generator might become unstable.

HAD-Net: A Hierarchical Adversarial Knowledge Distillation Network for Improved Enhanced Tumour Segmentation Without Post-Contrast Images

Abstract: Segmentation of enhancing tumours or lesions from MRI is important for detecting new disease activity in many clinical contexts. However, accurate segmentation requires the inclusion of medical images (e.g., T1 post contrast MRI) acquired after injecting patients with a contrast agent (e.g., Gadolinium), a process no longer thought to be safe. Although a number of modality-agnostic segmentation networks have been developed over the past few years, they have been met with limited success in the context of enhancing pathology segmentation. In this work, we present HAD-Net, a novel offline adversarial knowledge distillation (KD) technique, whereby a pre-trained teacher segmentation network, with access to all MRI sequences, teaches a student network, via hierarchical adversarial training, to better overcome the large domain shift presented when crucial images are absent during inference. In particular, we apply HAD-Net to the challenging task of enhancing tumour segmentation when access to post-contrast imaging is not available. The proposed network is trained and tested on the BraTS 2019 brain tumour segmentation challenge dataset, where it achieves performance improvements in the ranges of 16% - 26% over (a) recent modality-agnostic segmentation methods (U-HeMIS, U-HVED), (b) KD-Net adapted to this problem, (c) the pre-trained student network and (d) a non-hierarchical version of the network (AD-Net), in terms of Dice scores for enhancing tumour (ET). The network also shows improvements in tumour core (TC) Dice scores. Finally, the network outperforms both the baseline student network and AD-Net in terms of uncertainty quantification for enhancing tumour segmentation based on the BraTs 2019 uncertainty challenge metrics. Our code is publicly available at: https://github.com/SaverioVad/HAD_Net

Cross-Modal Information Maximization for Medical Imaging: CMIM

Abstract:
In hospitals, data are siloed to specific information systems that make the same information available under different modalities such as the different medical imaging exams the patient undergoes (CT scans, MRI, PET, Ultrasound, etc.) and their associated radiology reports. This offers unique opportunities to obtain and use at train-time those multiple views of the same information that might not always be available at test-time.
In this paper, we propose an innovative framework that makes the most of available data by learning good representations of a multi-modal input that are resilient to modality dropping at test-time, using recent advances in mutual information maximization. By maximizing cross-modal information at train time, we are able to outperform several state-of-the-art baselines in two different settings, medical image classification, and segmentation. In particular, our method is shown to have a strong impact on the inference-time performance of weaker modalities.

Missing MRI Pulse Sequence Synthesis using Multi-Modal Generative Adversarial Network

Abstract: Magnetic resonance imaging (MRI) is being increasingly utilized to assess, diagnose, and plan treatment for a variety of diseases. The ability to visualize tissue in varied contrasts in the form of MR pulse sequences in a single scan provides valuable insights to physicians, as well as enabling automated systems performing downstream analysis. However many issues like prohibitive scan time, image corruption, different acquisition protocols, or allergies to certain contrast materials may hinder the process of acquiring multiple sequences for a patient. This poses challenges to both physicians and automated systems since complementary information provided by the missing sequences is lost. In this paper, we propose a variant of generative adversarial network (GAN) capable of leveraging redundant information contained within multiple available sequences in order to generate one or more missing sequences for a patient scan. The proposed network is designed as a multi-input, multi-output network which combines information from all the available pulse sequences, implicitly infers which sequences are missing, and synthesizes the missing ones in a single forward pass. We demonstrate and validate our method on two brain MRI datasets each with four sequences, and show the applicability of the proposed method in simultaneously synthesizing all missing sequences in any possible scenario where either one, two, or three of the four sequences may be missing. We compare our approach with competing unimodal and multi-modal methods, and show that we outperform both quantitatively and qualitatively.

A comparative overview between networks that compensate for missing modalities.

This subsection compares and contrasts missing modality compensation approaches provided in Sections 3.1 - 3.5. Table {{table:588888ec-91a5-4cb5-afb7-dc61ce08dbaa}}  highlights their key techniques and describes the benefits and drawbacks of each model to provide the reader a clearer picture. As detailed in Table {{table:588888ec-91a5-4cb5-afb7-dc61ce08dbaa}} , the synthesis approach performs the compensation based on the image reconstruction from an atlas sample. In practice, these approaches usually fail to reconstruct the missing information and result in no performance gain. On the other hand, the common latent space representation methods perform the information retrieval by modelling joint information from all modalities. However, these methods do not perform well when more than one modality is missing.


Automated Specification-Based Testing of REST APIs

Abstract: Nowadays, REpresentational State Transfer Application Programming Interfaces (REST APIs) are widely used in web applications, hence a plethora of test cases are developed to validate the APIs calls. We propose a solution that automates the generation of test cases for REST APIs based on their specifications. In our approach, apart from the automatic generation of test cases, we provide an option for the user to influence the test case generation process. By adding user interaction, we aim to augment the automatic generation of APIs test cases with human testing expertise and specific context. We use the latest version of OpenAPI 3.x and a wide range of coverage metrics to analyze the functionality and performance of the generated test cases, and non-functional metrics to analyze the performance of the APIs. The experiments proved the effectiveness and practicability of our method.

Open Problems in Fuzzing RESTful APIs: A Comparison of Tools

Abstract: RESTful APIs are a type of web services that are widely used in industry. In the last few years, a lot of effort in the research community has been spent in designing novel techniques to automatically fuzz those APIs to find faults in them. Many real faults were automatically found in a large variety of RESTful APIs. However, usually the analyzed fuzzers treat the APIs as black-box, and no analysis of what is actually covered in these systems is done. Therefore, although these fuzzers are clearly useful for practitioners, we do not know what are their current limitations and actual effectiveness. Solving this is a necessary step to be able to design better, more efficient and effective techniques. To address this issue, in this paper we compare seven state-of-the-art fuzzers on 18 open-source and one industrial RESTful APIs. We then analyzed the source code of which parts of these APIs the fuzzers fail to generate tests for. This analysis points to clear limitations of these current fuzzers, listing concrete challenges for the research community to follow up on.

Common Open Challenges

Most of the papers we studied have mentioned some left objectives, to be addressed in future work.
Among the selected papers, there were 73 papers which have explicitly mentioned at least one open challenge for future work.
This is usually stated in the Conclusion section of these articles, or in specific Future Work section.
We collected all of these objectives, and identified the most common challenges among them, as shown in Table {{table:ff7f6ae9-660f-4fa8-8fdb-d8a0f6c340ac}} .
Note that, in most cases, a typical challenge is to design better techniques to obtain better code coverage and fault finding results.
Here, we rather focus on more specific challenges related to testing RESTful APIs.

Based on Table {{table:ff7f6ae9-660f-4fa8-8fdb-d8a0f6c340ac}} ,
the most common objective left as future work is Tool support that is mentioned in 25 papers.
Research in this domain is based on tool prototypes.
Due to the complexity of handling RESTful APIs, these tools require major engineering effort.
An early prototype can provide the base to experiments with some research ideas, which can already be of use and be publishable.
Work is then left to improve these tools to be able to be applicable on more case studies, and to be more user-friendly for practitioners.
For example, in {{cite:8d91509}} the authors proposed an approach that supports OpenAPI v2, and plan to enabling its support for OpenAPI v3 as well.
Another example, supporting authentication is needed to enable testing endpoints which needs the client to be authenticated, which was specified as needed future work in {{cite:b7e2c4a}}.
In these cases, not all techniques presented in the literature support authentication, or the full specs of the OpenAPI standard.
Still, even with partial support, it is possible to provide and evaluate novel techniques.