Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Dataset

Abstract: Virtual assistants such as Google Assistant, Alexa and Siri provide a conversational interface to a large number of services and APIs spanning multiple domains. Such systems need to support an ever-increasing number of services with possibly overlapping functionality. Furthermore, some of these services have little to no training data available. Existing public datasets for task-oriented dialogue do not sufficiently capture these challenges since they cover few domains and assume a single static ontology per domain. In this work, we introduce the the Schema-Guided Dialogue (SGD) dataset, containing over 16k multi-domain conversations spanning 16 domains. Our dataset exceeds the existing task-oriented dialogue corpora in scale, while also highlighting the challenges associated with building large-scale virtual assistants. It provides a challenging testbed for a number of tasks including language understanding, slot filling, dialogue state tracking and response generation. Along the same lines, we present a schema-guided paradigm for task-oriented dialogue, in which predictions are made over a dynamic set of intents and slots, provided as input, using their natural language descriptions. This allows a single dialogue system to easily support a large number of services and facilitates simple integration of new services without requiring additional training data. Building upon the proposed paradigm, we release a model for dialogue state tracking capable of zero-shot generalization to new APIs, while remaining competitive in the regular setting.

Taskmaster-1: Toward a Realistic and Diverse Dialog Dataset

Abstract: A significant barrier to progress in data-driven approaches to building dialog systems is the lack of high quality, goal-oriented conversational data. To help satisfy this elementary requirement, we introduce the initial release of the Taskmaster-1 dataset which includes 13,215 task-based dialogs comprising six domains. Two procedures were used to create this collection, each with unique advantages. The first involves a two-person, spoken “Wizard of Oz” (WOz) approach in which trained agents and crowdsourced workers interact to complete the task while the second is “self-dialog” in which crowdsourced workers write the entire dialog themselves. We do not restrict the workers to detailed scripts or to a small knowledge base and hence we observe that our dataset contains more realistic and diverse conversations in comparison to existing datasets. We offer several baseline models including state of the art neural seq2seq architectures with benchmark performance as well as qualitative human evaluations. Dialogs are labeled with API calls and arguments, a simple and cost effective approach which avoids the requirement of complex annotation schema. The layer of abstraction between the dialog model and the service provider API allows for a given model to interact with multiple services that provide similar functionally. Finally, the dataset will evoke interest in written vs. spoken language, discourse patterns, error handling and other linguistic phenomena related to dialog system research, development and design.

Key-Value Retrieval Networks for Task-Oriented Dialogue

Abstract: Neural task-oriented dialogue systems often struggle to smoothly interface with a knowledge base. In this work, we seek to address this problem by proposing a new neural dialogue agent that is able to effectively sustain grounded, multi-domain discourse through a novel key-value retrieval mechanism. The model is end-to-end differentiable and does not need to explicitly model dialogue state or belief trackers. We also release a new dataset of 3,031 dialogues that are grounded through underlying knowledge bases and span three distinct tasks in the in-car personal assistant space: calendar scheduling, weather information retrieval, and point-of-interest navigation. Our architecture is simultaneously trained on data from all domains and significantly outperforms a competitive rule-based system and other existing neural dialogue architectures on the provided domains according to both automatic and human evaluation metrics.

Frames: a corpus for adding memory to goal-oriented dialogue systems

Abstract: This paper proposes a new dataset, Frames, composed of 1369 human-human dialogues with an average of 15 turns per dialogue. This corpus contains goal-oriented dialogues between users who are given some constraints to book a trip and assistants who search a database to find appropriate trips. The users exhibit complex decision-making behaviour which involve comparing trips, exploring different options, and selecting among the trips that were discussed during the dialogue. To drive research on dialogue systems towards handling such behaviour, we have annotated and released the dataset and we propose in this paper a task called frame tracking. This task consists of keeping track of different semantic frames throughout each dialogue. We propose a rule-based baseline and analyse the frame tracking task through this baseline.

Neural Belief Tracker: Data-Driven Dialogue State Tracking

Abstract: One of the core components of modern spoken dialogue systems is the belief tracker, which estimates the user’s goal at every step of the dialogue. However, most current approaches have difficulty scaling to larger, more complex dialogue domains. This is due to their dependency on either: a) Spoken Language Understanding models that require large amounts of annotated training data; or b) hand-crafted lexicons for capturing some of the linguistic variation in users’ language. We propose a novel Neural Belief Tracking (NBT) framework which overcomes these problems by building on recent advances in representation learning. NBT models reason over pre-trained word vectors, learning to compose them into distributed representations of user utterances and dialogue context. Our evaluation on two datasets shows that this approach surpasses past limitations, matching the performance of state-of-the-art models which rely on hand-crafted semantic lexicons and outperforming them when such lexicons are not provided.

A Network-based End-to-End Trainable Task-oriented Dialogue System

Abstract: Teaching machines to accomplish tasks by conversing naturally with humans is challenging. Currently, developing task-oriented dialogue systems requires creating multiple components and typically this involves either a large amount of handcrafting, or acquiring costly labelled datasets to solve a statistical learning problem for each component. In this work we introduce a neural network-based text-in, text-out end-to-end trainable goal-oriented dialogue system along with a new way of collecting dialogue data based on a novel pipe-lined Wizard-of-Oz framework. This approach allows us to develop dialogue systems easily and without making too many assumptions about the task at hand. The results show that the model can converse with human subjects naturally whilst helping them to accomplish tasks in a restaurant search domain.

Data statistics for task-oriented dialogue pre-training.

We collect nine different task-oriented datasets which are English-based, human-human, multi-turn and publicly available. In total, there are 100,707 dialogues, which contain 1,388,152 utterances over 60 domains. Dataset statistics is shown in Table {{table:aaf81db8-ddad-4fc7-b0a9-18ff2baa19be}} .


learning rate, network, prior, index dimension

Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning

Abstract: Deep learning tools have gained tremendous attention in applied machine learning. However such tools for regression and classification do not capture model uncertainty. In comparison, Bayesian models offer a mathematically grounded framework to reason about model uncertainty, but usually come with a prohibitive computational cost. In this paper we develop a new theoretical framework casting dropout training in deep neural networks (NNs) as approximate Bayesian inference in deep Gaussian processes. A direct result of this theory gives us tools to model uncertainty with dropout NNs -- extracting information from existing models that has been thrown away so far. This mitigates the problem of representing uncertainty in deep learning without sacrificing either computational complexity or test accuracy. We perform an extensive study of the properties of dropout's uncertainty. Various network architectures and non-linearities are assessed on tasks of regression and classification, using MNIST as an example. We show a considerable improvement in predictive log-likelihood and RMSE compared to existing state-of-the-art methods, and finish by using dropout's uncertainty in deep reinforcement learning.

Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles

Abstract: Deep neural networks (NNs) are powerful black box predictors that have recently achieved impressive performance on a wide spectrum of tasks. Quantifying predictive uncertainty in NNs is a challenging and yet unsolved problem. Bayesian NNs, which learn a distribution over weights, are currently the state-of-the-art for estimating predictive uncertainty; however these require significant modifications to the training procedure and are computationally expensive compared to standard (non-Bayesian) NNs. We propose an alternative to Bayesian NNs that is simple to implement, readily parallelizable, requires very little hyperparameter tuning, and yields high quality predictive uncertainty estimates. Through a series of experiments on classification and regression benchmarks, we demonstrate that our method produces well-calibrated uncertainty estimates which are as good or better than approximate Bayesian NNs. To assess robustness to dataset shift, we evaluate the predictive uncertainty on test examples from known and unknown distributions, and show that our method is able to express higher uncertainty on out-of-distribution examples. We demonstrate the scalability of our method by evaluating predictive uncertainty estimates on ImageNet.

Epistemic Neural Networks

Abstract: Intelligence relies on an agent's knowledge of what it does not know. This capability can be assessed based on the quality of joint predictions of labels across multiple inputs. In principle, ensemble-based approaches produce effective joint predictions, but the computational costs of training large ensembles can become prohibitive. We introduce the epinet: an architecture that can supplement any conventional neural network, including large pretrained models, and can be trained with modest incremental computation to estimate uncertainty. With an epinet, conventional neural networks outperform very large ensembles, consisting of hundreds or more particles, with orders of magnitude less computation. The epinet does not fit the traditional framework of Bayesian neural networks. To accommodate development of approaches beyond BNNs, such as the epinet, we introduce the epistemic neural network (ENN) as an interface for models that produce joint predictions.

Summary of agent architectures, full details in Appendix .

We consider ENNs based around fine-tuning a pretrained BERT architecture.
At a high level, these ENNs work by branching from the final hidden layer in the BERT network.
Table {{table:915f13ea-b8d0-473b-8fc0-af2af7e3deb1}}  outlines these architectures, together with the hyperparameters we tune.


Pitfalls of Graph Neural Network Evaluation

Abstract: Semi-supervised node classification in graphs is a fundamental problem in graph mining, and the recently proposed graph neural networks (GNNs) have achieved unparalleled results on this task. Due to their massive success, GNNs have attracted a lot of attention, and many novel architectures have been put forward. In this paper we show that existing evaluation strategies for GNN models have serious shortcomings. We show that using the same train/validation/test splits of the same datasets, as well as making significant changes to the training procedure (e.g. early stopping criteria) precludes a fair comparison of different architectures. We perform a thorough empirical evaluation of four prominent GNN models and show that considering different splits of the data leads to dramatically different rankings of models. Even more importantly, our findings suggest that simpler GNN architectures are able to outperform the more sophisticated ones if the hyperparameters and the training procedure are tuned fairly for all models.

Heterogeneous Graph Attention Network

Abstract: Graph neural network, as a powerful graph representation technique based on deep learning, has shown superior performance and attracted considerable research interest. However, it has not been fully considered in graph neural network for heterogeneous graph which contains different types of nodes and links. The heterogeneity and rich semantic information bring great challenges for designing a graph neural network for heterogeneous graph. Recently, one of the most exciting advancements in deep learning is the attention mechanism, whose great potential has been well demonstrated in various areas. In this paper, we first propose a novel heterogeneous graph neural network based on the hierarchical attention, including node-level and semantic-level attentions. Specifically, the node-level attention aims to learn the importance between a node and its meta-path based neighbors, while the semantic-level attention is able to learn the importance of different meta-paths. With the learned importance from both node-level and semantic-level attention, the importance of node and meta-path can be fully considered. Then the proposed model can generate node embedding by aggregating features from meta-path based neighbors in a hierarchical manner. Extensive experimental results on three real-world heterogeneous graphs not only show the superior performance of our proposed model over the state-of-the-arts, but also demonstrate its potentially good interpretability for graph analysis.

Image-based Recommendations on Styles and Substitutes

Abstract: Humans inevitably develop a sense of the relationships between objects, some of which are based on their appearance. Some pairs of objects might be seen as being alternatives to each other (such as two pairs of jeans), while others may be seen as being complementary (such as a pair of jeans and a matching shirt). This information guides many of the choices that people make, from buying clothes to their interactions with each other. We seek here to model this human sense of the relationships between objects based on their appearance. Our approach is not based on fine-grained modeling of user annotations but rather on capturing the largest dataset possible and developing a scalable method for uncovering human notions of the visual relationships within. We cast this as a network inference problem defined on graphs of related images, and provide a large-scale dataset for the training and evaluation of the same. The system we develop is capable of recommending which clothes and accessories will go well together (and which will not), amongst a host of other applications.

Summary of real-world datasets

{{formula:6bbb14bf-f71d-4102-ada5-175685928ad7}}  Datasets.
Table {{table:cb4f7e08-e8bd-4ca9-b933-df185c66a8b8}}  summarizes the 12 real-world datasets from 5 domains used for evaluating the performance of our method.
CORA {{cite:b319974}}, CiteSeer, PubMed {{cite:939b14b}}, and DBLP {{cite:7d02ceb}} are citation networks.
Coauthor-CS, Coauthor-Phy {{cite:282f7db}}, and ACM {{cite:53042d8}} represent co-authorship relations between authors and papers.
BlogCatalog and Flickr {{cite:17bd759}} are social networks.
Amazon-CS and Amazon-Photo {{cite:49dce0e}} are graphs of goods that are usually bought together.
UAI {{cite:211d833}} is a graph of Internet document network including the reference relations.
To simulate the situation of coupled graphs, we partitioned the datasets into subgraphs as the preprocess.
Each graph was partitioned into {{formula:95e9b577-1ab9-4861-82c8-c6fb6085e8cd}}  parts, and each part was distributed to one party.


GNNExplainer: Generating Explanations for Graph Neural Networks

Abstract: Graph Neural Networks (GNNs) are a powerful tool for machine learning on graphs. GNNs combine node feature information with the graph structure by recursively passing neural messages along edges of the input graph. However, incorporating both graph structure and feature information leads to complex models and explaining predictions made by GNNs remains unsolved. Here we propose GnnExplainer, the first general, model-agnostic approach for providing interpretable explanations for predictions of any GNN-based model on any graph-based machine learning task. Given an instance, GnnExplainer identifies a compact subgraph structure and a small subset of node features that have a crucial role in GNN's prediction. Further, GnnExplainer can generate consistent and concise explanations for an entire class of instances. We formulate GnnExplainer as an optimization task that maximizes the mutual information between a GNN's prediction and distribution of possible subgraph structures. Experiments on synthetic and real-world graphs show that our approach can identify important graph structures as well as node features, and outperforms alternative baseline approaches by up to 43.0% in explanation accuracy. GnnExplainer provides a variety of benefits, from the ability to visualize semantically relevant structures to interpretability, to giving insights into errors of faulty GNNs.

Towards Self-Explainable Graph Neural Network

Abstract: Graph Neural Networks (GNNs), which generalize the deep neural networks to graph-structured data, have achieved great success in modeling graphs. However, as an extension of deep learning for graphs, GNNs lack explainability, which largely limits their adoption in scenarios that demand the transparency of models. Though many efforts are taken to improve the explainability of deep learning, they mainly focus on i.i.d data, which cannot be directly applied to explain the predictions of GNNs because GNNs utilize both node features and graph topology to make predictions. There are only very few work on the explainability of GNNs and they focus on post-hoc explanations. Since post-hoc explanations are not directly obtained from the GNNs, they can be biased and misrepresent the true explanations. Therefore, in this paper, we study a novel problem of self-explainable GNNs which can simultaneously give predictions and explanations. We propose a new framework which can find K-nearest labeled nodes for each unlabeled node to give explainable node classification, where nearest labeled nodes are found by interpretable similarity module in terms of both node similarity and local structure similarity. Extensive experiments on real-world and synthetic datasets demonstrate the effectiveness of the proposed framework for explainable node classification.

Parameterized Explainer for Graph Neural Network

Abstract: Despite recent progress in Graph Neural Networks (GNNs), explaining predictions made by GNNs remains a challenging open problem. The leading method independently addresses the local explanations (i.e., important subgraph structure and node features) to interpret why a GNN model makes the prediction for a single instance, e.g. a node or a graph. As a result, the explanation generated is painstakingly customized for each instance. The unique explanation interpreting each instance independently is not sufficient to provide a global understanding of the learned GNN model, leading to a lack of generalizability and hindering it from being used in the inductive setting. Besides, as it is designed for explaining a single instance, it is challenging to explain a set of instances naturally (e.g., graphs of a given class). In this study, we address these key challenges and propose PGExplainer, a parameterized explainer for GNNs. PGExplainer adopts a deep neural network to parameterize the generation process of explanations, which enables PGExplainer a natural approach to explaining multiple instances collectively. Compared to the existing work, PGExplainer has better generalization ability and can be utilized in an inductive setting easily. Experiments on both synthetic and real-life datasets show highly competitive performance with up to 24.7\% relative improvement in AUC on explaining graph classification over the leading baseline.

Explainability in Graph Neural Networks: A Taxonomic Survey

Abstract: Deep learning methods are achieving ever-increasing performance on many artificial intelligence tasks. A major limitation of deep models is that they are not amenable to interpretability. This limitation can be circumvented by developing post hoc techniques to explain predictions, giving rise to the area of explainability. Recently, explainability of deep models on images and texts has achieved significant progress. In the area of graph data, graph neural networks (GNNs) and their explainability are experiencing rapid developments. However, there is neither a unified treatment of GNN explainability methods, nor a standard benchmark and testbed for evaluations. In this survey, we provide a unified and taxonomic view of current GNN explainability methods. Our unified and taxonomic treatments of this subject shed lights on the commonalities and differences of existing methods and set the stage for further methodological developments. To facilitate evaluations, we provide a testbed for GNN explainability, including datasets, common algorithms and evaluation metrics. Furthermore, we conduct comprehensive experiments to compare and analyze the performance of many techniques. Altogether, this work provides a unified methodological treatment of GNN explainability and a standardized testbed for evaluations.

Datasets for Explainability of GNNs

A summary of representative benchmark datasets is provided in Table {{table:15cb9496-d480-4d44-9d0a-de9f6c7a8123}}  along with their key statistics, tasks and papers used the datasets.


One weird trick for parallelizing convolutional neural networks

Abstract: I present a new way to parallelize the training of convolutional neural networks across multiple GPUs. The method scales significantly better than all alternatives when applied to modern convolutional neural networks.

Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour

Abstract: Deep learning thrives with large neural networks and large datasets. However, larger networks and larger datasets result in longer training times that impede research and development progress. Distributed synchronous SGD offers a potential solution to this problem by dividing SGD minibatches over a pool of parallel workers. Yet to make this scheme efficient, the per-worker workload must be large, which implies nontrivial growth in the SGD minibatch size. In this paper, we empirically show that on the ImageNet dataset large minibatches cause optimization difficulties, but when these are addressed the trained networks exhibit good generalization. Specifically, we show no loss of accuracy when training with large minibatch sizes up to 8192 images. To achieve this result, we adopt a hyper-parameter-free linear scaling rule for adjusting learning rates as a function of minibatch size and develop a new warmup scheme that overcomes optimization challenges early in training. With these simple techniques, our Caffe2-based system trains ResNet-50 with a minibatch size of 8192 on 256 GPUs in one hour, while matching small minibatch accuracy. Using commodity hardware, our implementation achieves ~90% scaling efficiency when moving from 8 to 256 GPUs. Our findings enable training visual recognition models on internet-scale data with high efficiency.

State-of-the-art large-batch training and test accuracy.

The intuition of linear scaling is related to the number of iterations. Let us use {{formula:f8326f26-564a-429e-bc45-8f32e2b18427}} , {{formula:0c6b4e38-4a44-4c8d-8f09-2603d1a389e0}} , and {{formula:e5e06df5-06e2-4a01-a5a9-058be1d1d703}}  to denote the batch size, the learning rate, and the number of iterations. If we increase the the batch size from {{formula:808d8d56-bf6e-4558-8619-90a57d8cce40}}  to {{formula:bca28f61-1e32-473b-9056-95f5754c7408}} , then the number of iterations is reduced from {{formula:551a706f-2ee4-4b23-a0a8-b886af45067f}}  to {{formula:18a79427-6234-4ebc-935b-3884b6976cd8}} . This means that the frequency of weight updating reduced by {{formula:37830a02-b26b-4d14-a72b-75b5d3e36d2e}}  times. Thus, we make the updating of each iteration {{formula:b66f7a11-b191-46c3-a4df-ae166d847632}}  more efficient by enlarging the learning rate by {{formula:e728cef8-a8d4-4a36-8ad7-7115a1c6fa41}}  times.
The purpose of a warmup scheme is to avoid the situation in which the algorithm diverges at the beginning because we have to use a very large learning rate based on linear scaling.
With these techniques, researchers can use the relatively large batch in a certain range (Table {{table:ee24cc6b-6e05-4dbb-bb9c-992af928b1c1}} ). However, we observe that state-of-the-art approaches can only scale batch size to 1024 for AlexNet and 8192 for ResNet-50.
If we increase the batch size to 4096 for AlexNet, we only achieve 53.1% in 100 epochs (Table {{table:0233a303-5c0c-4b75-84db-07013c493dba}} ). Our target is to achieve 58% accuracy even when using large batch sizes.


ProjectQ: An Open Source Software Framework for Quantum Computing

Abstract: We introduce ProjectQ, an open source software effort for quantum computing. The first release features a compiler framework capable of targeting various types of hardware, a high-performance simulator with emulation capabilities, and compiler plug-ins for circuit drawing and resource estimation. We introduce our Python-embedded domain-specific language, present the features, and provide example implementations for quantum algorithms. The framework allows testing of quantum algorithms through simulation and enables running them on actual quantum hardware using a back-end connecting to the IBM Quantum Experience cloud service. Through extension mechanisms, users can provide back-ends to further quantum hardware, and scientists working on quantum compilation can provide plug-ins for additional compilation, optimization, gate synthesis, and layout strategies.

Classical Concepts in Quantum Programming

Abstract: The rapid progress of computer technology has been accompanied by a corresponding evolution of software development, from hardwired components and binary machine code to high level programming languages, which allowed to master the increasing hardware complexity and fully exploit its potential. This paper investigates, how classical concepts like hardware abstraction, hierarchical programs, data types, memory management, flow of control and structured programming can be used in quantum computing. The experimental language QCL will be introduced as an example, how elements like irreversible functions, local variables and conditional branching, which have no direct quantum counterparts, can be implemented, and how non-classical features like the reversibility of unitary transformation or the non-observability of quantum states can be accounted for within the framework of a procedural programming language.

Qibo: a framework for quantum simulation with hardware acceleration

Abstract: We present Qibo, a new open-source software for fast evaluation of quantum circuits and adiabatic evolution which takes full advantage of hardware accelerators. The growing interest in quantum computing and the recent developments of quantum hardware devices motivates the development of new advanced computational tools focused on performance and usage simplicity. In this work we introduce a new quantum simulation framework that enables developers to delegate all complicated aspects of hardware or platform implementation to the library so they can focus on the problem and quantum algorithms at hand. This software is designed from scratch with simulation performance, code simplicity and user friendly interface as target goals. It takes advantage of hardware acceleration such as multi-threading CPU, single GPU and multi-GPU devices.

Quipper: a scalable quantum programming language

Abstract: The field of quantum algorithms is vibrant. Still, there is currently a lack of programming languages for describing quantum computation on a practical scale, i.e., not just at the level of toy problems. We address this issue by introducing Quipper, a scalable, expressive, functional, higher-order quantum programming language. Quipper has been used to program a diverse set of non-trivial quantum algorithms, and can generate quantum gate representations using trillions of gates. It is geared towards a model of computation that uses a classical computer to control a quantum device, but is not dependent on any particular model of quantum hardware. Quipper has proven effective and easy to use, and opens the door towards using formal methods to analyze quantum algorithms.

LIQUi|>: A Software Design Architecture and Domain-Specific Language for Quantum Computing

Abstract: Languages, compilers, and computer-aided design tools will be essential for scalable quantum computing, which promises an exponential leap in our ability to execute complex tasks. LIQUi|> is a modular software architecture designed to control quantum hardware. It enables easy programming, compilation, and simulation of quantum algorithms and circuits, and is independent of a specific quantum architecture. LIQUi|> contains an embedded, domain-specific language designed for programming quantum algorithms, with F# as the host language. It also allows the extraction of a circuit data structure that can be used for optimization, rendering, or translation. The circuit can also be exported to external hardware and software environments. Two different simulation environments are available to the user which allow a trade-off between number of qubits and class of operations. LIQUi|> has been implemented on a wide range of runtimes as back-ends with a single user front-end. We describe the significant components of the design architecture and how to express any given quantum algorithm.

t|ket⟩: a retargetable compiler for NISQ devices

Abstract: We present t|ket⟩, a quantum software development platform produced by Cambridge Quantum Computing Ltd. The heart of t|ket⟩ is a language-agnostic optimising compiler designed to generate code for a variety of NISQ devices, which has several features designed to minimise the influence of device error. The compiler has been extensively benchmarked and outperforms most competitors in terms of circuit optimisation and qubit routing.

Strawberry Fields: A Software Platform for Photonic Quantum Computing

Abstract: We introduce Strawberry Fields, an open-source quantum programming architecture for light-based quantum computers, and detail its key features. Built in Python, Strawberry Fields is a full-stack library for design, simulation, optimization, and quantum machine learning of continuous-variable circuits. The platform consists of three main components: (i) an API for quantum programming based on an easy-to-use language named Blackbird; (ii) a suite of three virtual quantum computer backends, built in NumPy and TensorFlow, each targeting specialized uses; and (iii) an engine which can compile Blackbird programs on various backends, including the three built-in simulators, and -- in the near future -- photonic quantum information processors. The library also contains examples of several paradigmatic algorithms, including teleportation, (Gaussian) boson sampling, instantaneous quantum polynomial, Hamiltonian simulation, and variational quantum circuit optimization.

Comparison of relevant quantum computing frameworks. Legend: S: Simulation on CPU, E: Execution on quantum devices, QML: has quantum machine learning facilities. Domain-specific languages are embedded in the language denoted between parenthesis.

A comparison of frameworks using the quantum circuit model is shown in Table {{table:9e15e550-c592-4981-801d-135c2b303a88}} .
Most frameworks allow to import and export of circuits in the OpenQASM format {{cite:fb93ac5}}, an open-source specification for quantum circuits. This facilitates the porting of quantum software among the different platforms.


Scanning the Internet for ROS: A View of Security in Robotics Research

Abstract: Because robots can directly perceive and affect the physical world, security issues take on particular importance. In this paper, we describe the results of our work on scanning the entire IPv4 address space of the Internet for instances of the Robot Operating System (ROS), a widely used robotics platform for research. Our results identified that a number of hosts supporting ROS are exposed to the public Internet, thereby allowing anyone to access robotic sensors and actuators. As a proof of concept, and with consent, we were able to read image sensor information and move the robot of a research group in a US university. This paper gives an overview of our findings, including the geographic distribution of publicly-accessible platforms, the sorts of sensor and actuator data that is available, as well as the different kinds of robots and sensors that our scan uncovered. Additionally, we offer recommendations on best practices to mitigate these security issues in the future.

Security and Performance Considerations in ROS 2: A Balancing Act

Abstract: Robot Operating System (ROS) 2 is a ground-up re-design of ROS 1 to support performance critical cyber-physical systems (CPSs) using the Data Distribution Service (DDS) middleware. Accordingly, the security of ROS 2 is highly reliant on the security of its DDS communication protocol. However, finding a balance between the performance and security is non-trivial task. Inappropriate security implementations may cause not only significant loss on performance of the system, but also security failures in the system. In this paper, we provide an analysis of the DDS security protocol as well as an overview on how to find the balance between performance and security. To accomplish this, we evaluate the latency and throughput of the communication protocols of ROS 2 in both wired and wireless networks, and measure the efficiency loss caused by the enabling of security protocols such as Virtual Private Network (VPN) and DDS security protocol in ROS 2 in both network setups. The result can be directly used by robotics developers to find the optimal and balanced settings of ROS 2 applications. Additionally, we analyzed the security specification of DDS using existing security standards and tested the implementation of the DDS protocol by performing static analysis. The results of this work can be used to enhance the security of ROS 2.

Classification of the Related Work

We further classified the identified publications in the literature based on the (1) utilized approach static or dynamic, and (2) the goal of the analysis. The goals are divided into security, quality, and physical units. We excluded the papers that do not provide or utilize a tool for assessing ROS like  {{cite:c291768}}, {{cite:f58ca71}}. Although some papers such as {{cite:c804a2f}} do not evaluate the ROS framework, we decided to include them in the classifications because they utilize ROS in the conducted evaluation. Table {{table:8ae3fbd1-8972-4f28-9a1d-2ee05e98b20c}}  presents our classification of the related work. We can conclude from this table that dynamic analysis is a popular approach to perform a security assessment, but static analysis is a common approach to investigate a certain software property. Another observation based on the results in Table {{table:8ae3fbd1-8972-4f28-9a1d-2ee05e98b20c}} , the majority of prior work targets ROS1. We also notice that the majority of the dynamic analysis approaches are concentrated around the network layer than the application layer, except ROSPenTo and Roschaos {{cite:6959965}}, which exploit the usage of certain ROS APIs.


Polar Bear Optimization Algorithm: Meta-Heuristic with Fast Population Movement and Dynamic Birth and Death Mechanism

Abstract: In the proposed article, we present a nature-inspired optimization algorithm, which we called Polar Bear Optimization Algorithm (PBO). The inspiration to develop the algorithm comes from the way polar bears hunt to survive in harsh arctic conditions. These carnivorous mammals are active all year round. Frosty climate, unfavorable to other animals, has made polar bears adapt to the specific mode of exploration and hunting in large areas, not only over ice but also water. The proposed novel mathematical model of the way polar bears move in the search for food and hunt can be a valuable method of optimization for various theoretical and practical problems. Optimization is very similar to nature, similarly to search for optimal solutions for mathematical models animals search for optimal conditions to develop in their natural environments. In this method. we have used a model of polar bear behaviors as a search engine for optimal solutions. Proposed simulated adaptation to harsh winter conditions is an advantage for local and global search, while birth and death mechanism controls the population. Proposed PBO was evaluated and compared to other meta-heuristic algorithms using sample test functions and some classical engineering problems. Experimental research results were compared to other algorithms and analyzed using various parameters. The analysis allowed us to identify the leading advantages which are rapid recognition of the area by the relevant population and efficient birth and death mechanism to improve global and local search within the solution space.

Wisdom of Artificial Crowds—A Metaheuristic Algorithm for Optimization

Abstract: Finding optimal solutions to NP-Hard problems requires exponential time with respect to the size of the problem. Consequently, heuristic methods are usually utilized to obtain approximate solutions to problems of such difficulty. In this paper, a novel swarm-based nature-inspired metaheuristic algorithm for optimization is proposed. Inspired by human collective intelligence, Wisdom of Artificial Crowds (WoAC) algorithm relies on a group of simulated intelligent agents to arrive at independent solutions aggregated to produce a solution which in many cases is superior to individual solutions of all participating agents. We illustrate superior performance of WoAC by comparing it against another bio-inspired approach, the Genetic Algorithm, on one of the classical NP-Hard problems, the Travelling Salesperson Problem. On average a 3% - 10% improvement in quality of solutions is observed with little computational overhead.

Algorithms based on mammals behavior (part 2).

A more complex behavior can be observed in several primates like spider monkey ( {{cite:034dfb9}}), blue monkeys ( {{cite:c433ab4}}), ageist monkeys which age could bring a difference for better performance to solve problems ( {{cite:43562f6}}) and sexual behavior with individual intelligence of chimps ( {{cite:b1f7f6d}}). Humans are considered primates and have the same behavior but in a more sophisticated way to form tribes ( {{cite:aeba027}}), groups ( {{cite:a4e9558}}, {{cite:ded72fe}}) or aggregates as crowds ( {{cite:83e764a}}) (More complex behaviors unique for humans are not considered here because it is very difficult to be expressed by other species) All algorithms based on mammals behavior are shown in Tables {{table:885dfe6c-3ac5-40ba-94b3-1ea467147baf}}  and {{table:513c1e65-8c75-4450-9dad-2bd77e2896a9}} .


Describing Textures in the Wild

Abstract: Patterns and textures are key characteristics of many natural objects: a shirt can be striped, the wings of a butterfly can be veined, and the skin of an animal can be scaly. Aiming at supporting this dimension in image understanding, we address the problem of describing textures with semantic attributes. We identify a vocabulary of forty-seven texture terms and use them to describe a large dataset of patterns collected "in the wild". The resulting Describable Textures Dataset (DTD) is a basis to seek the best representation for recognizing describable texture attributes in images. We port from object recognition to texture recognition the Improved Fisher Vector (IFV) and Deep Convolutional-network Activation Features (DeCAF), and show that surprisingly, they both outperform specialized texture descriptors not only on our problem, but also in established material recognition datasets. We also show that our describable attributes are excellent texture descriptors, transferring between datasets and tasks, in particular, combined with IFV and DeCAF, they significantly outperform the state-of-the-art by more than 10% on both FMD and KTH-TIPS-2b benchmarks. We also demonstrate that they produce intuitive descriptions of materials and Internet images.

EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classification

Abstract: In this paper, we present a patch-based land use and land cover classification approach using Sentinel-2 satellite images. The Sentinel-2 satellite images are openly and freely accessible, and are provided in the earth observation program Copernicus. We present a novel dataset, based on these images that covers 13 spectral bands and is comprised of ten classes with a total of 27 000 labeled and geo-referenced images. Benchmarks are provided for this novel dataset with its spectral bands using state-of-the-art deep convolutional neural networks. An overall classification accuracy of 98.57% was achieved with the proposed novel dataset. The resulting classification system opens a gate toward a number of earth observation applications. We demonstrate how this classification system can be used for detecting land use and land cover changes, and how it can assist in improving geographical maps. The geo-referenced dataset EuroSAT is made publicly available at https://github.com/phelber/eurosat.

Remote Sensing Image Scene Classification: Benchmark and State of the Art

Abstract: Remote sensing image scene classification plays an important role in a wide range of applications and hence has been receiving remarkable attention. During the past years, significant efforts have been made to develop various data sets or present a variety of approaches for scene classification from remote sensing images. However, a systematic review of the literature concerning data sets and methods for scene classification is still lacking. In addition, almost all existing data sets have a number of limitations, including the small scale of scene classes and the image numbers, the lack of image variations and diversity, and the saturation of accuracy. These limitations severely limit the development of new approaches especially deep learning-based methods. This paper first provides a comprehensive review of the recent progress. Then, we propose a large-scale data set, termed “NWPU-RESISC45,” which is a publicly available benchmark for REmote Sensing Image Scene Classification (RESISC), created by Northwestern Polytechnical University (NWPU). This data set contains 31 500 images, covering 45 scene classes with 700 images in each class. The proposed NWPU-RESISC45 1) is large-scale on the scene classes and the total image number; 2) holds big variations in translation, spatial resolution, viewpoint, object pose, illumination, background, and occlusion; and 3) has high within-class diversity and between-class similarity. The creation of this data set will enable the community to develop and evaluate various data-driven algorithms. Finally, several representative methods are evaluated using the proposed data set, and the results are reported as a useful baseline for future research.

CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning

Abstract: When building artificial intelligence systems that can reason and answer questions about visual data, we need diagnostic tests to analyze our progress and discover short-comings. Existing benchmarks for visual question answering can help, but have strong biases that models can exploit to correctly answer questions without reasoning. They also conflate multiple sources of error, making it hard to pinpoint model weaknesses. We present a diagnostic dataset that tests a range of visual reasoning abilities. It contains minimal biases and has detailed annotations describing the kind of reasoning each question requires. We use this dataset to analyze a variety of modern visual reasoning systems, providing novel insights into their abilities and limitations.

Fine-Grained Visual Classification of Aircraft

Abstract: This paper introduces FGVC-Aircraft, a new dataset containing 10,000 images of aircraft spanning 100 aircraft models, organised in a three-level hierarchy. At the finer level, differences between models are often subtle but always visually measurable, making visual recognition challenging but possible. A benchmark is obtained by defining corresponding classification tasks and evaluation protocols, and baseline results are presented. The construction of this dataset was made possible by the work of aircraft enthusiasts, a strategy that can extend to the study of number of other object classes. Compared to the domains usually considered in fine-grained visual classification (FGVC), for example animals, aircraft are rigid and hence less deformable. They, however, present other interesting modes of variation, including purpose, size, designation, structure, historical style, and branding.

Statistics of used datasets.

See Table {{table:d2d30b2b-c417-47df-b223-7dc8c88fff89}} .


Toy Car Running Dataset {{cite:f1470ec}}

Audio-Based Event Detection at Different SNR Settings Using Two-Dimensional Spectrogram Magnitude Representations

Abstract: Audio-based event detection poses a number of different challenges that are not encountered in other fields, such as image detection. Challenges such as ambient noise, low Signal-to-Noise Ratio (SNR) and microphone distance are not yet fully understood. If the multimodal approaches are to become better in a range of fields of interest, audio analysis will have to play an integral part. Event recognition in autonomous vehicles (AVs) is such a field at a nascent stage that can especially leverage solely on audio or can be part of the multimodal approach. In this manuscript, an extensive analysis focused on the comparison of different magnitude representations of the raw audio is presented. The data on which the analysis is carried out is part of the publicly available MIVIA Audio Events dataset. Single channel Short-Time Fourier Transform (STFT), mel-scale and Mel-Frequency Cepstral Coefficients (MFCCs) spectrogram representations are used. Furthermore, aggregation methods of the aforementioned spectrogram representations are examined; the feature concatenation compared to the stacking of features as separate channels. The effect of the SNR on recognition accuracy and the generalization of the proposed methods on datasets that were both seen and not seen during training are studied and reported.

Batch Uniformization for Minimizing Maximum Anomaly Score of DNN-based Anomaly Detection in Sounds

Abstract: Use of an autoencoder (AE) as a normal model is a state-of-the-art technique for unsupervised-anomaly detection in sounds (ADS). The AE is trained to minimize the sample mean of the anomaly score of normal sounds in a mini-batch. One problem with this approach is that the anomaly score of rare-normal sounds becomes higher than that of frequent-normal sounds, because the sample mean is strongly affected by frequent-normal samples, resulting in preferentially decreasing the anomaly score of frequent-normal samples. To decrease anomaly scores for both frequent- and rare-normal sounds, we propose batch uniformization, a training method for unsupervised-ADS for minimizing a weighted average of the anomaly score on each sample in a mini-batch. We used the reciprocal of the probabilistic density of each sample as the weight, more intuitively, a large weight is given for rare-normal sounds. Such a weight works to give a constant anomaly score for both frequent- and rare-normal sounds. Since the probabilistic density is unknown, we estimate it by using the kernel density estimation on each training mini-batch. Verification- and objective-experiments show that the proposed batch uniformization improves the performance of unsupervised-ADS.

Anomaly Detection in audio presented in the included studies of the SR (Automatic Search).

Tables {{table:489eafe4-1fb3-4bce-8881-5a47967bde97}} , {{table:a46c76e1-fda6-4171-ba10-a2e4a43bf9ce}} , and {{table:0a6e481c-6fae-4134-9ae4-e11ce47599de}}  shows a synthesis of the 31 primary studies analyzed including: dataset, audio features, ML model and evaluation method for each one. A global analysis is presented in this session taking into account the most relevant topics.


ALBERT: A Lite BERT for Self-supervised Learning of Language Representations

Abstract: Increasing model size when pretraining natural language representations often results in improved performance on downstream tasks. However, at some point further model increases become harder due to GPU/TPU memory limitations and longer training times. To address these problems, we present two parameter-reduction techniques to lower memory consumption and increase the training speed of BERT. Comprehensive empirical evidence shows that our proposed methods lead to models that scale much better compared to the original BERT. We also use a self-supervised loss that focuses on modeling inter-sentence coherence, and show it consistently helps downstream tasks with multi-sentence inputs. As a result, our best model establishes new state-of-the-art results on the GLUE, RACE, and \squad benchmarks while having fewer parameters compared to BERT-large. The code and the pretrained models are available at this https URL.

RoBERTa: A Robustly Optimized BERT Pretraining Approach

Abstract: Language model pretraining has led to significant performance gains but careful comparison between different approaches is challenging. Training is computationally expensive, often done on private datasets of different sizes, and, as we will show, hyperparameter choices have significant impact on the final results. We present a replication study of BERT pretraining (Devlin et al., 2019) that carefully measures the impact of many key hyperparameters and training data size. We find that BERT was significantly undertrained, and can match or exceed the performance of every model published after it. Our best model achieves state-of-the-art results on GLUE, RACE and SQuAD. These results highlight the importance of previously overlooked design choices, and raise questions about the source of recently reported improvements. We release our models and code.

XLNet: Generalized Autoregressive Pretraining for Language Understanding

Abstract: With the capability of modeling bidirectional contexts, denoising autoencoding based pretraining like BERT achieves better performance than pretraining approaches based on autoregressive language modeling. However, relying on corrupting the input with masks, BERT neglects dependency between the masked positions and suffers from a pretrain-finetune discrepancy. In light of these pros and cons, we propose XLNet, a generalized autoregressive pretraining method that (1) enables learning bidirectional contexts by maximizing the expected likelihood over all permutations of the factorization order and (2) overcomes the limitations of BERT thanks to its autoregressive formulation. Furthermore, XLNet integrates ideas from Transformer-XL, the state-of-the-art autoregressive model, into pretraining. Empirically, under comparable experiment settings, XLNet outperforms BERT on 20 tasks, often by a large margin, including question answering, natural language inference, sentiment analysis, and document ranking.

DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing

Abstract: This paper presents a new pre-trained language model, DeBERTaV3, which improves the original DeBERTa model by replacing mask language modeling (MLM) with replaced token detection (RTD), a more sample-efficient pre-training task. Our analysis shows that vanilla embedding sharing in ELECTRA hurts training efficiency and model performance. This is because the training losses of the discriminator and the generator pull token embeddings in different directions, creating the"tug-of-war"dynamics. We thus propose a new gradient-disentangled embedding sharing method that avoids the tug-of-war dynamics, improving both training efficiency and the quality of the pre-trained model. We have pre-trained DeBERTaV3 using the same settings as DeBERTa to demonstrate its exceptional performance on a wide range of downstream natural language understanding (NLU) tasks. Taking the GLUE benchmark with eight tasks as an example, the DeBERTaV3 Large model achieves a 91.37% average score, which is 1.37% over DeBERTa and 1.91% over ELECTRA, setting a new state-of-the-art (SOTA) among the models with a similar structure. Furthermore, we have pre-trained a multi-lingual model mDeBERTa and observed a larger improvement over strong baselines compared to English models. For example, the mDeBERTa Base achieves a 79.8% zero-shot cross-lingual accuracy on XNLI and a 3.6% improvement over XLM-R Base, creating a new SOTA on this benchmark. We have made our pre-trained models and inference code publicly available at https://github.com/microsoft/DeBERTa.

ConvBERT: Improving BERT with Span-based Dynamic Convolution

Abstract: Pre-trained language models like BERT and its variants have recently achieved impressive performance in various natural language understanding tasks. However, BERT heavily relies on the global self-attention block and thus suffers large memory footprint and computation cost. Although all its attention heads query on the whole input sequence for generating the attention map from a global perspective, we observe some heads only need to learn local dependencies, which means the existence of computation redundancy. We therefore propose a novel span-based dynamic convolution to replace these self-attention heads to directly model local dependencies. The novel convolution heads, together with the rest self-attention heads, form a new mixed attention block that is more efficient at both global and local context learning. We equip BERT with this mixed attention design and build a ConvBERT model. Experiments have shown that ConvBERT significantly outperforms BERT and its variants in various downstream tasks, with lower training cost and fewer model parameters. Remarkably, ConvBERTbase model achieves 86.4 GLUE score, 0.7 higher than ELECTRAbase, while using less than 1/4 training cost. Code and pre-trained models will be released.

MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices

Abstract: Natural Language Processing (NLP) has recently achieved great success by using huge pre-trained models with hundreds of millions of parameters. However, these models suffer from heavy model sizes and high latency such that they cannot be deployed to resource-limited mobile devices. In this paper, we propose MobileBERT for compressing and accelerating the popular BERT model. Like the original BERT, MobileBERT is task-agnostic, that is, it can be generically applied to various downstream NLP tasks via simple fine-tuning. Basically, MobileBERT is a thin version of BERT_LARGE, while equipped with bottleneck structures and a carefully designed balance between self-attentions and feed-forward networks. To train MobileBERT, we first train a specially designed teacher model, an inverted-bottleneck incorporated BERT_LARGE model. Then, we conduct knowledge transfer from this teacher to MobileBERT. Empirical studies show that MobileBERT is 4.3x smaller and 5.5x faster than BERT_BASE while achieving competitive results on well-known benchmarks. On the natural language inference tasks of GLUE, MobileBERT achieves a GLUE score of 77.7 (0.6 lower than BERT_BASE), and 62 ms latency on a Pixel 4 phone. On the SQuAD v1.1/v2.0 question answering task, MobileBERT achieves a dev F1 score of 90.0/79.2 (1.5/2.1 higher than BERT_BASE).

DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter

Abstract: As Transfer Learning from large-scale pre-trained models becomes more prevalent in Natural Language Processing (NLP), operating these large models in on-the-edge and/or under constrained computational training or inference budgets remains challenging. In this work, we propose a method to pre-train a smaller general-purpose language representation model, called DistilBERT, which can then be fine-tuned with good performances on a wide range of tasks like its larger counterparts. While most prior work investigated the use of distillation for building task-specific models, we leverage knowledge distillation during the pre-training phase and show that it is possible to reduce the size of a BERT model by 40%, while retaining 97% of its language understanding capabilities and being 60% faster. To leverage the inductive biases learned by larger models during pre-training, we introduce a triple loss combining language modeling, distillation and cosine-distance losses. Our smaller, faster and lighter model is cheaper to pre-train and we demonstrate its capabilities for on-device computations in a proof-of-concept experiment and a comparative on-device study.

A list of the various models used in this study in addition to the number of parameters and GLUE score. We have designated three different sizes as base

To keep the code accessible, we chose a range of models that were both popular, accessible through a single APIhuggingface.co and achieved high GLUE scores {{cite:e26a895}}. The selection of models, their references, and an approximation of their GLUE scores, and their respective sizes in millions of parameters is given in Table {{table:671d09fe-af2c-4c79-898c-983e69b07f27}} .


Methods, model architectures, performance evaluation.

Model architectures and categorization, datasets, evaluation metrics, challenges, and insights.

Methods, the taxonomy of CNN models, datasets, and applications.

Datasets, model architecture, loss functions, evaluation metrics, insights

Models, datasets, algorithms, and evaluation of crowd motion analysis, behavior recognition, and anomaly detection.

Convolutional-Neural Network-Based Image Crowd Counting: Review, Categorization, Analysis, and Performance Evaluation

Abstract: Traditional handcrafted crowd-counting techniques in an image are currently transformed via machine-learning and artificial-intelligence techniques into intelligent crowd-counting techniques. This paradigm shift offers many advanced features in terms of adaptive monitoring and the control of dynamic crowd gatherings. Adaptive monitoring, identification/recognition, and the management of diverse crowd gatherings can improve many crowd-management-related tasks in terms of efficiency, capacity, reliability, and safety. Despite many challenges, such as occlusion, clutter, and irregular object distribution and nonuniform object scale, convolutional neural networks are a promising technology for intelligent image crowd counting and analysis. In this article, we review, categorize, analyze (limitations and distinctive features), and provide a detailed performance evaluation of the latest convolutional-neural-network-based crowd-counting techniques. We also highlight the potential applications of convolutional-neural-network-based crowd-counting techniques. Finally, we conclude this article by presenting our key observations, providing strong foundation for future research directions while designing convolutional-neural-network-based crowd-counting techniques. Further, the article discusses new advancements toward understanding crowd counting in smart cities using the Internet of Things (IoT).

CNN-based Density Estimation and Crowd Counting: A Survey

Abstract: Accurately estimating the number of objects in a single image is a challenging yet meaningful task and has been applied in many applications such as urban planning and public safety. In the various object counting tasks, crowd counting is particularly prominent due to its specific significance to social security and development. Fortunately, the development of the techniques for crowd counting can be generalized to other related fields such as vehicle counting and environment survey, if without taking their characteristics into account. Therefore, many researchers are devoting to crowd counting, and many excellent works of literature and works have spurted out. In these works, they are must be helpful for the development of crowd counting. However, the question we should consider is why they are effective for this task. Limited by the cost of time and energy, we cannot analyze all the algorithms. In this paper, we have surveyed over 220 works to comprehensively and systematically study the crowd counting models, mainly CNN-based density map estimation methods. Finally, according to the evaluation metrics, we select the top three performers on their crowd counting datasets and analyze their merits and drawbacks. Through our analysis, we expect to make reasonable inference and prediction for the future development of crowd counting, and meanwhile, it can also provide feasible solutions for the problem of object counting in other fields. We provide the density maps and prediction results of some mainstream algorithm in the validation set of NWPU dataset for comparison and testing. Meanwhile, density map generation and evaluation tools are also provided. All the codes and evaluation results are made publicly available at this https URL.

Advances in Convolution Neural Networks Based Crowd Counting and Density Estimation

Abstract: Automatically estimating the number of people in unconstrained scenes is a crucial yet challenging task in different real-world applications, including video surveillance, public safety, urban planning, and traffic monitoring. In addition, methods developed to estimate the number of people can be adapted and applied to related tasks in various fields, such as plant counting, vehicle counting, and cell microscopy. Many challenges and problems face crowd counting, including cluttered scenes, extreme occlusions, scale variation, and changes in camera perspective. Therefore, in the past few years, tremendous research efforts have been devoted to crowd counting, and numerous excellent techniques have been proposed. The significant progress in crowd counting methods in recent years is mostly attributed to advances in deep convolution neural networks (CNNs) as well as to public crowd counting datasets. In this work, we review the papers that have been published in the last decade and provide a comprehensive survey of the recent CNNs based crowd counting techniques. We briefly review detection-based, regression-based, and traditional density estimation based approaches. Then, we delve into detail regarding the deep learning based density estimation approaches and recently published datasets. In addition, we discuss the potential applications of crowd counting and in particular its applications using unmanned aerial vehicle (UAV) images.

Revisiting Crowd Counting: State-of-the-art, Trends, and Future Perspectives

Abstract: Crowd counting is an effective tool for situational awareness in public places. Automated crowd counting using images and videos is an interesting yet challenging problem that has gained significant attention in computer vision. Over the past few years, various deep learning methods have been developed to achieve state-of-the-art performance. The methods evolved over time vary in many aspects such as model architecture, input pipeline, learning paradigm, computational complexity, and accuracy gains etc. In this paper, we present a systematic and comprehensive review of the most significant contributions in the area of crowd counting. Although few surveys exist on the topic, our survey is most up-to date and different in several aspects. First, it provides a more meaningful categorization of the most significant contributions by model architectures, learning methods (i.e., loss functions), and evaluation methods (i.e., evaluation metrics). We chose prominent and distinct works and excluded similar works. We also sort the well-known crowd counting models by their performance over benchmark datasets. We believe that this survey can be a good resource for novice researchers to understand the progressive developments and contributions over time and the current state-of-the-art.

Crowded Scene Analysis: A Survey

Abstract: Automated scene analysis has been a topic of great interest in computer vision and cognitive science. Recently, with the growth of crowd phenomena in the real world, crowded scene analysis has attracted much attention. However, the visual occlusions and ambiguities in crowded scenes, as well as the complex behaviors and scene semantics, make the analysis a challenging task. In the past few years, an increasing number of works on the crowded scene analysis have been reported, which covered different aspects including crowd motion pattern learning, crowd behavior and activity analyses, and anomaly detection in crowds. This paper surveys the state-of-the-art techniques on this topic. We first provide the background knowledge and the available features related to crowded scenes. Then, existing models, popular algorithms, evaluation protocols, and system performance are provided corresponding to different aspects of the crowded scene analysis. We also outline the available datasets for performance evaluation. Finally, some research problems and promising future directions are presented with discussions.

Related studies on crowd visual analysis.

Several studies have been conducted in the past which typically focus on individual areas of crowd analysis. For example, {{cite:0bef58e}}, {{cite:9e4178b}}, {{cite:8d892b1}}, {{cite:e1fd0eb}}, {{cite:a090741}}, {{cite:f4f04fb}}, {{cite:da3d164}}, {{cite:a1f538f}} covers crowd counting research and mainly discusses the advancements in model architectures, benchmarking, and datasets. {{cite:aca8dfe}}, {{cite:78149c2}}, {{cite:0d23cc8}} focus on crowd motion analysis, discussing crowd motion predictions, flow classification, and behavior analysis using motion patterns. {{cite:6ea5211}}, {{cite:44dffa5}}, {{cite:a6b0b68}} studies anomaly detection with focus on methods, datasets and comparisons of results. Unlike the aforementioned studies, few studies also cover the multiple aspects of crowd analysis. For instance, {{cite:5671636}} covers motion analysis, behavior recognition, and anomaly detection but do not cover counting, density estimation, object detection, crowd prediction etc. {{cite:4e9535d}} covers density estimation, motion detection, and behavior recognition, but do not cover object detection, and anomaly detection. Similarly, {{cite:f87cbbc}} covers only on physics-inspired methods for crowd analysis and focuses on motion analysis in videos. Table {{table:2f5839af-fa86-4e28-8387-8d21729ba24a}}  presents a list of recent survey articles on the topic.


Character-level Convolutional Networks for Text Classification

Abstract: This article offers an empirical exploration on the use of character-level convolutional networks (ConvNets) for text classification. We constructed several large-scale datasets to show that character-level convolutional networks could achieve state-of-the-art or competitive results. Comparisons are offered against traditional models such as bag of words, n-grams and their TFIDF variants, and deep learning models such as word-based ConvNets and recurrent neural networks.

A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference

Abstract: This paper introduces the Multi-Genre Natural Language Inference (MultiNLI) corpus, a dataset designed for use in the development and evaluation of machine learning models for sentence understanding. At 433k examples, this resource is one of the largest corpora available for natural language inference (a.k.a. recognizing textual entailment), improving upon available resources in both its coverage and difficulty. MultiNLI accomplishes this by offering data from ten distinct genres of written and spoken English, making it possible to evaluate systems on nearly the full complexity of the language, while supplying an explicit setting for evaluating cross-genre domain adaptation. In addition, an evaluation using existing machine learning models designed for the Stanford NLI corpus shows that it represents a substantially more difficult task than does that corpus, despite the two showing similar levels of inter-annotator agreement.

Evaluation datasets. Each dataset is distributed to 1000 clients. Label quantity of each class follows the non-iid label distribution in {{cite:214e396}} where {{formula:2249daf3-0cf8-41b4-88fc-dac78f47f7a9}} .

Dataset and models
We perform our evaluation on four English datasets and manually designed promptsWe try 6, 2, 6, 4 different prompts for each datasets separately and report the chosen one that performs best.
The verbalizers are the same as previous literature {{cite:7192058}}.,
detailed information is shown in Table {{table:914d9585-26b9-4f10-91cc-631c6ce98973}} .
(1) AGNEWS {{cite:8585817}} is a news classification dataset. Given headline and text body, news needs to be classified as one of the four categories: World, Sports, Business or Science/Tech.
(2) MNLI {{cite:e3a82a8}} is a sentence understanding dataset. Given text pairs x = (a, b), the task is to find out whether a implies b, a and b contradict each other or neither.
(3) YELP Review Full (YELP-F) {{cite:8585817}} is a restaurant rating dataset. Given a customer's review, text should be estimated on a 1-5 star scale.
(4) YAHOO {{cite:8585817}} is a text classification dataset. Given a question and an answer, one of ten possible categories needs to be assigned.
All experiments are conducted using the same pre-trained model, RoBERTa-large (355M parameters) {{cite:07d105c}}, which we load from the transformers {{cite:e0910cc}} library.


and the causal estimation error
channels

Mutual Information, Relative Entropy, and Estimation in the Poisson Channel

Abstract: Let $X$ be a non-negative random variable and let the conditional distribution of a random variable $Y$, given $X$, be ${Poisson}(\gamma \cdot X)$, for a parameter $\gamma \geq 0$. We identify a natural loss function such that: 1) The derivative of the mutual information between $X$ and $Y$ with respect to $\gamma$ is equal to the \emph{minimum} mean loss in estimating $X$ based on $Y$, regardless of the distribution of $X$. 2) When $X \sim P$ is estimated based on $Y$ by a mismatched estimator that would have minimized the expected loss had $X \sim Q$, the integral over all values of $\gamma$ of the excess mean loss is equal to the relative entropy between $P$ and $Q$. For a continuous time setting where $X^T = \{X_t, 0 \leq t \leq T \}$ is a non-negative stochastic process and the conditional law of $Y^T=\{Y_t, 0\le t\le T\}$, given $X^T$, is that of a non-homogeneous Poisson process with intensity function $\gamma \cdot X^T$, under the same loss function: 1) The minimum mean loss in \emph{causal} filtering when $\gamma = \gamma_0$ is equal to the expected value of the minimum mean loss in \emph{non-causal} filtering (smoothing) achieved with a channel whose parameter $\gamma$ is uniformly distributed between 0 and $\gamma_0$. Bridging the two quantities is the mutual information between $X^T$ and $Y^T$. 2) This relationship between the mean losses in causal and non-causal filtering holds also in the case where the filters employed are mismatched, i.e., optimized assuming a law on $X^T$ which is not the true one. Bridging the two quantities in this case is the sum of the mutual information and the relative entropy between the true and the mismatched distribution of $Y^T$. Thus, relative entropy quantifies the excess estimation loss due to mismatch in this setting. These results parallel those recently found for the Gaussian channel.

Directed Information, Causal Estimation, and Communication in Continuous Time

Abstract: A notion of directed information between two continuous-time processes is proposed. A key component in the definition is taking an infimum over all possible partitions of the time interval, which plays a role no less significant than the supremum over "space" partitions inherent in the definition of mutual information. Properties and operational interpretations in estimation and communication are then established for the proposed notion of directed information. For the continuous-time additive white Gaussian noise channel, it is shown that Duncan's classical relationship between causal estimation and information continues to hold in the presence of feedback upon replacing mutual information by directed information. A parallel result is established for the Poisson channel. The utility of this relationship is then demonstrated in computing the directed information rate between the input and output processes of a continuous-time Poisson channel with feedback, where the channel input process is constrained to be constant between events at the channel output. Finally, the capacity of a wide class of continuous-time channels with feedback is established via directed information, characterizing the fundamental limit on reliable communication.

Optimum estimation via gradients of partition functions and information measures: a statistical-mechanical perspective

Abstract: In continuation to a recent work on the statistical--mechanical analysis of minimum mean square error (MMSE) estimation in Gaussian noise via its relation to the mutual information (the I-MMSE relation), here we propose a simple and more direct relationship between optimum estimation and certain information measures (e.g., the information density and the Fisher information), which can be viewed as partition functions and hence are amenable to analysis using statistical--mechanical techniques. The proposed approach has several advantages, most notably, its applicability to general sources and channels, as opposed to the I-MMSE relation and its variants which hold only for certain classes of channels (e.g., additive white Gaussian noise channels). We then demonstrate the derivation of the conditional mean estimator and the MMSE in a few examples. Two of these examples turn out to be generalizable to a fairly wide class of sources and channels. For this class, the proposed approach is shown to yield an approximate conditional mean estimator and an MMSE formula that has the flavor of a single-letter expression. We also show how our approach can easily be generalized to situations of mismatched estimation.

Recent research results of the I-MMSE relationships for non-Gaussian channels

Some work consider more general channel models.
A scalar channel with arbitrary additive
noise is studied in {{cite:813392f}}.
It is found that the increase in the mutual information
due to the improvement in the channel quality equals
to the correlation of two conditional mean estimates
associated with the input and the noise respectively.
D. P. Palomar et al. represent the derivative of mutual
information in terms of the conditional input estimation given
the output for general channels (not necessarily additive noise) {{cite:eb64ece}}.
The general results in {{cite:eb64ece}} embrace most of popular channel models including
the binary symmetric channel, the binary erasure channel,
the discrete memoryless channel, the scalar/vector Gaussian channel,
an arbitrary additive-noise channel, and the Poisson channel.
For a scalar channel with arbitrary additive noise,
D. Guo obtain the derivative of relative entropy with respect
to the energy of perturbation, which can be expressed as a mean
squared difference of the score functions of the two distributions {{cite:cc20a8e}}.
Furthermore, Y. Wu et al. analyze the properties of the MMSE as a function of
the input-output joint distribution {{cite:3924f89}}. It is proved in {{cite:3924f89}}
that the MMSE is a concave function of the input-output joint distribution.
N. Merhav analyze the MMSE for both cases of matched and mismatched estimations
by using the statistical-mechanical techniques {{cite:bdafbe9}}. The main advantage
of the results in {{cite:bdafbe9}} is that they apply for very general vector channels
with arbitrary joint input-output probability distributions. By exploiting the statistical-mechanical techniques,
N. Merhav also establish relationships between rate-distortion function and the MMSE for a general
scalar channel {{cite:0b1b6b5}}. Then, simple lower and upper bounds on the rate-distortion function can
be obtained based on the bounds of the MMSE. A brief summation
of recent research results
of the I-MMSE relationships for non-Gaussian channels
is given in Table {{table:3e14d6df-c76f-481d-986e-006e52748bdd}} .


Semi-adversarial Networks: Convolutional Autoencoders for Imparting Privacy to Face Images

Abstract: In this paper, we design and evaluate a convolutional autoencoder that perturbs an input face image to impart privacy to a subject. Specifically, the proposed autoencoder transforms an input face image such that the transformed image can be successfully used for face recognition but not for gender classification. In order to train this autoencoder, we propose a novel training scheme, referred to as semi-adversarial training in this work. The training is facilitated by attaching a semi-adversarial module consisting of an auxiliary gender classifier and an auxiliary face matcher to the autoencoder. The objective function utilized for training this network has three terms: one to ensure that the perturbed image is a realistic face image; another to ensure that the gender attributes of the face are confounded; and a third to ensure that biometric recognition performance due to the perturbed image is not impacted. Extensive experiments confirm the efficacy of the proposed architecture in extending gender privacy to face images.

Gender Privacy: An Ensemble of Semi Adversarial Networks for Confounding Arbitrary Gender Classifiers

Abstract: Recent research has proposed the use of Semi Adversarial Networks (SAN) for imparting privacy to face images. SANs are convolutional autoencoders that perturb face images such that the perturbed images cannot be reliably used by an attribute classifier (e.g., a gender classifier) but can still be used by a face matcher for matching purposes. However, the generalizability of SANs across multiple arbitrary gender classifiers has not been demonstrated in the literature. In this work, we tackle the generalization issue by designing an ensemble SAN model that generates a diverse set of perturbed outputs for a given input face image. This is accomplished by enforcing diversity among the individual models in the ensemble through the use of different data augmentation techniques. The goal is to ensure that at least one of the perturbed output faces will confound an arbitrary, previously unseen gender classifier. Extensive experiments using different unseen gender classifiers and face matchers are performed to demonstrate the efficacy of the proposed paradigm in imparting gender privacy to face images.

SensitiveNets: Learning Agnostic Representations with Application to Face Recognition

Abstract: This work proposes a new neural network feature representation that help to leave out sensitive information in the decision-making process of pattern recognition and machine learning algorithms. The aim of this work is to develop a learning method capable to remove certain information from the feature space without drop of performance in a recognition task based on that feature space. Our work is in part motivated by the new international regulation for personal data protection, which forces data controllers to avoid discriminative hazards while managing sensitive data of users. Our method is based on a triplet loss learning generalization that introduces a sensitive information removal process. The method is evaluated on face recognition technologies using state-of-the-art algorithms and publicly available benchmarks. In addition, we present a new annotation dataset with balanced distribution between genders and ethnic origins. The dataset includes more than 120K images from 24K identities with variety of poses, image quality, facial expressions, and illumination. The experiments demonstrate that it is possible to reduce sensitive information such as gender or ethnicity in the feature representation while retaining competitive performance in a face recognition task.

Overview of existing methods for imparting soft-biometric privacy and their comparison based on three criteria: transferability, generalizability, and retention of matching performance; transferability refers to the ability to generate perturbations that can successfully confound a different gender classifier, whereas generalizability is a stronger criterion for the ability to confound any arbitrary unseen gender classifier.

While these methods directly apply perturbations to face images, recently, new techniques have emerged where perturbations were applied to face representation vectors computed by face matchers {{cite:0cd924c}}, {{cite:079f02e}}. In particular, Morales et al. {{cite:0cd924c}} proposed a neural-network-based model, called SensitiveNet, that is able to remove soft-biometric information from face representation vectors. Therefore, any attribute classifier trained on face representation vectors may not be able to extract such sensitive information. However, these methods are based on the assumption that only face representation vectors are stored in a biometric database. This scheme is not desirable in many applications since only storing face representations results in 1) losing human interpretability, and 2) losing backward matching compatibility when the face matcher is updated. An overview of existing techniques and their properties (transferability, generalization to arbitrary attribute classifiers, and retaining matching utility) is shown in Table {{table:3560e8ad-2a9d-4160-bdf8-007ab0e734e3}} .


EEG, BVP (HR), EDA, EMG, temp., hand and head gestures

Collection and Validation of Psychophysiological Data from Professional and Amateur Players: a Multimodal eSports Dataset

Abstract: Proper training and analytics in eSports require accurately collected and annotated data. Most eSports research focuses exclusively on in-game data analysis, and there is a lack of prior work involving eSports athletes' psychophysiological data. In this paper, we present a dataset collected from professional and amateur teams in 22 matches in League of Legends video game with more than 40 hours of recordings. Recorded data include the players' physiological activity, e.g. movements, pulse, saccades, obtained from various sensors, self-reported aftermatch survey, and in-game data. An important feature of the dataset is simultaneous data collection from five players, which facilitates the analysis of sensor data on a team level. Upon the collection of dataset we carried out its validation. In particular, we demonstrate that stress and concentration levels for professional players are less correlated, meaning more independent playstyle. Also, we show that the absence of team communication does not affect the professional players as much as amateur ones. To investigate other possible use cases of the dataset, we have trained classical machine learning algorithms for skill prediction and player re-identification using 3-minute sessions of sensor data. Best models achieved 0.856 and 0.521 (0.10 for a chance level) accuracy scores on a validation set for skill prediction and player re-id problems, respectively. The dataset is available at https://github.com/smerdov/eSports Sensors Dataset.

The Arousal video Game AnnotatIoN (AGAIN) Dataset

Abstract: How can we model affect in a general fashion, across dissimilar tasks, and to which degree are such general representations of affect even possible? To address such questions and enable research towards general affective computing, this paper introduces The Arousal video Game AnnotatIoN (AGAIN) dataset. AGAIN is a large-scale affective corpus that features over 1,100 in-game videos (with corresponding gameplay data) from nine different games, which are annotated for arousal from 124 participants in a first-person continuous fashion. Even though AGAIN is created for the purpose of investigating the generality of affective computing across dissimilar tasks, affect modelling can be studied within each of its 9 specific interactive games. To the best of our knowledge AGAIN is the largest -- over 37 hours of annotated video and game logs -- and most diverse publicly available affective dataset based on games as interactive affect elicitors.

Affective Game Computing Datasets. “N/A” indicates that a category is “not-applicable”.

Table {{table:81d9aeda-3332-4b2a-884b-33ba624bcb99}}  presents our survey of 11 datasets which can be used for affective game computing research. Most of the datasets presented here are quite recent, (i.e. 8 of the 11 are released in 2019 or later) showing the potential and the emerging nature of the field. Moreover, most of the examined datasets focus on one specific context; i.e. the genre of the game. While the AGAIN dataset {{cite:134aaa2}} was designed explicitly to offer a wide array of different games, the Atari-HEAD {{cite:b5750ce}} and MUMBAI {{cite:881558b}} datasets also provide more varied inputs in their own niche (arcade- and board-games, respectively). Many game datasets contain video data of the gameplay footage, which is ideal for deep learning applications and for mapping pixels to affect directly {{cite:11abde6}}, {{cite:0185d84}}. While pixel-based affect detection might be a harder task on datasets collected on board or social games (i.e. MUMBAI {{cite:881558b}} and GAME-ON {{cite:10c432b}}), datasets that provide a large amount of gameplay footage are ideal for computer vision methods (i.e. Atari-HEAD {{cite:b5750ce}} and AGAIN {{cite:134aaa2}}). Five out of the eleven affective game computing corpora feature physiological signals that are more commonly used in traditional affective computing research, while only 3 of them feature eye-tracking data. In addition to traditional features, many game datasets (5 of the 11 surveyed) contain behavioural and contextual data in the form of game telemetry and player input. This type of data has proven to be robust as a predictor of game-related emotional states {{cite:d5f606a}}, {{cite:8bfd0c8}}, {{cite:92c9e06}}.

Reflecting over Table {{table:81d9aeda-3332-4b2a-884b-33ba624bcb99}}  one can observe a substantial increase in interest and availability of game-based datasets over the last few years; there are still, however, many aspects of player experience that have not made it to an affect corpus as of yet. Moreover the design of the various data collection protocols comes with limitations. Most notably the wide spectrum of labels used across the corpora makes it hard to compare datasets and transfer any knowledge gained across corpora. While it is understandable that game-related outcomes are more important than basic affective dimensions for game user researchers, some level of standardisation would go a long way towards making these datasets more accessible and reusable. Nevertheless, affective corpora—as the ones surveyed in Table {{table:81d9aeda-3332-4b2a-884b-33ba624bcb99}} —that are based on interactive elicitation bring a new perspective to affective computing at large, as such elicitation methods were largely unsupported by traditional datasets up until recently.


Reproduction study using public data of: Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs

Abstract: We have attempted to reproduce the results in Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs, published in JAMA 2016; 316(22), using publicly available data sets. We re-implemented the main method in the original study since the source code is not available. The original study used non-public fundus images from EyePACS and three hospitals in India for training. We used a different EyePACS data set from Kaggle. The original study used the benchmark data set Messidor-2 to evaluate the algorithm’s performance. We used another distribution of the Messidor-2 data set, since the original data set is no longer available. In the original study, ophthalmologists re-graded all images for diabetic retinopathy, macular edema, and image gradability. We have one diabetic retinopathy grade per image for our data sets, and we assessed image gradability ourselves. We were not able to reproduce the original study’s results with publicly available data. Our algorithm’s area under the receiver operating characteristic curve (AUC) of 0.951 (95% CI, 0.947-0.956) on the Kaggle EyePACS test set and 0.853 (95% CI, 0.835-0.871) on Messidor-2 did not come close to the reported AUC of 0.99 on both test sets in the original study. This may be caused by the use of a single grade per image, or different data. This study shows the challenges of reproducing deep learning method results, and the need for more replication and reproduction studies to validate deep learning methods, especially for medical image analysis. Our source code and instructions are available at: https://github.com/mikevoets/jama16-retina-replication.

Blended Multi-Modal Deep ConvNet Features for Diabetic Retinopathy Severity Prediction

Abstract: Diabetic Retinopathy (DR) is one of the major causes of visual impairment and blindness across the world. It is usually found in patients who suffer from diabetes for a long period. The major focus of this work is to derive optimal representation of retinal images that further helps to improve the performance of DR recognition models. To extract optimal representation, features extracted from multiple pre-trained ConvNet models are blended using proposed multi-modal fusion module. These final representations are used to train a Deep Neural Network (DNN) used for DR identification and severity level prediction. As each ConvNet extracts different features, fusing them using 1D pooling and cross pooling leads to better representation than using features extracted from a single ConvNet. Experimental studies on benchmark Kaggle APTOS 2019 contest dataset reveals that the model trained on proposed blended feature representations is superior to the existing methods. In addition, we notice that cross average pooling based fusion of features from Xception and VGG16 is the most appropriate for DR recognition. With the proposed model, we achieve an accuracy of 97.41%, and a kappa statistic of 94.82 for DR identification and an accuracy of 81.7% and a kappa statistic of 71.1% for severity level prediction. Another interesting observation is that DNN with dropout at input layer converges more quickly when trained using blended features, compared to the same model trained using uni-modal deep features.

Multi-categorical deep learning neural network to classify retinal images: A pilot study employing small database

Abstract: Deep learning emerges as a powerful tool for analyzing medical images. Retinal disease detection by using computer-aided diagnosis from fundus image has emerged as a new method. We applied deep learning convolutional neural network by using MatConvNet for an automated detection of multiple retinal diseases with fundus photographs involved in STructured Analysis of the REtina (STARE) database. Dataset was built by expanding data on 10 categories, including normal retina and nine retinal diseases. The optimal outcomes were acquired by using a random forest transfer learning based on VGG-19 architecture. The classification results depended greatly on the number of categories. As the number of categories increased, the performance of deep learning models was diminished. When all 10 categories were included, we obtained results with an accuracy of 30.5%, relative classifier information (RCI) of 0.052, and Cohen’s kappa of 0.224. Considering three integrated normal, background diabetic retinopathy, and dry age-related macular degeneration, the multi-categorical classifier showed accuracy of 72.8%, 0.283 RCI, and 0.577 kappa. In addition, several ensemble classifiers enhanced the multi-categorical classification performance. The transfer learning incorporated with ensemble classifier of clustering and voting approach presented the best performance with accuracy of 36.7%, 0.053 RCI, and 0.225 kappa in the 10 retinal diseases classification problem. First, due to the small size of datasets, the deep learning techniques in this study were ineffective to be applied in clinics where numerous patients suffering from various types of retinal disorders visit for diagnosis and treatment. Second, we found that the transfer learning incorporated with ensemble classifiers can improve the classification performance in order to detect multi-categorical retinal diseases. Further studies should confirm the effectiveness of algorithms with large datasets obtained from hospitals.

Data sets used in the reviewed papers.

Table {{table:bd8280d3-626f-4439-87ec-2defd0d88906}}  demonstrates the datasets used in the reviewed articles, and in Figure {{figure:5c68a4c7-32a8-4011-a179-cb0efa50813c}}  the frequencies of the used datasets are shown.
As shown in Figure {{figure:5c68a4c7-32a8-4011-a179-cb0efa50813c}} , many publicly available DR datasets were utilized for DR classification.
Among them, the Diabetic Retinopathy Detection Dataset on Kaggle was the most preferred because of its size and accessibility.
Besides, it is noteworthy that many non-public datasets were also used in most studies {{cite:7ab63b7}}, {{cite:72a6a2f}}, {{cite:0e4f79e}}, {{cite:a0f20f3}}, {{cite:5adf2cb}}, {{cite:adefb4d}}, {{cite:e3bc872}}, {{cite:52d0e6e}}, {{cite:5acd647}}.
Deep neural networks work better on large datasets, and the size of the data set is a very important parameter in the network's performance.
For this reason, most studies have used more than one dataset to improve classification performance {{cite:1bbc44a}}, {{cite:46611d2}}, {{cite:8d73286}}, {{cite:0e4f79e}}, {{cite:f8b0089}}, {{cite:5f198f6}}, {{cite:4ad2146}}, {{cite:5c001cf}}, {{cite:3093bc6}}, {{cite:4c8f0f5}}, {{cite:9b22bd8}}, {{cite:1282fa4}}, {{cite:970c30c}}.