Multilingual Knowledge Graph Embeddings for Cross-lingual Knowledge Alignment

Abstract: Many recent works have demonstrated the benefits of knowledge graph embeddings in completing monolingual knowledge graphs. Inasmuch as related knowledge bases are built in several different languages, achieving cross-lingual knowledge alignment will help people in constructing a coherent knowledge base, and assist machines in dealing with different expressions of entity relationships across diverse human languages. Unfortunately, achieving this highly desirable cross-lingual alignment by human labor is very costly and error-prone. Thus, we propose MTransE, a translation-based model for multilingual knowledge graph embeddings, to provide a simple and automated solution. By encoding entities and relations of each language in a separated embedding space, MTransE provides transitions for each embedding vector to its cross-lingual counterparts in other spaces, while preserving the functionalities of monolingual embeddings. We deploy three different techniques to represent cross-lingual transitions, namely axis calibration, translation vectors, and linear transformations, and derive five variants for MTransE using different loss functions. Our models can be trained on partially aligned graphs, where just a small portion of triples are aligned with their cross-lingual counterparts. The experiments on cross-lingual entity matching and triple-wise alignment verification show promising results, with some variants consistently outperforming others on different tasks. We also explore how MTransE preserves the key properties of its monolingual counterpart.

Co-training Embeddings of Knowledge Graphs and Entity Descriptions for Cross-lingual Entity Alignment

Abstract: Multilingual knowledge graph (KG) embeddings provide latent semantic representations of entities and structured knowledge with cross-lingual inferences, which benefit various knowledge-driven cross-lingual NLP tasks. However, precisely learning such cross-lingual inferences is usually hindered by the low coverage of entity alignment in many KGs. Since many multilingual KGs also provide literal descriptions of entities, in this paper, we introduce an embedding-based approach which leverages a weakly aligned multilingual KG for semi-supervised cross-lingual learning using entity descriptions. Our approach performs co-training of two embedding models, i.e. a multilingual KG embedding model and a multilingual literal description embedding model. The models are trained on a large Wikipedia-based trilingual dataset where most entity alignment is unknown to training. Experimental results show that the performance of the proposed approach on the entity alignment task improves at each iteration of co-training, and eventually reaches a stage at which it significantly surpasses previous approaches. We also show that our approach has promising abilities for zero-shot entity alignment, and cross-lingual KG completion.

Multi-view Knowledge Graph Embedding for Entity Alignment

Abstract: We study the problem of embedding-based entity alignment between knowledge graphs (KGs). Previous works mainly focus on the relational structure of entities. Some further incorporate another type of features, such as attributes, for refinement. However, a vast of entity features are still unexplored or not equally treated together, which impairs the accuracy and robustness of embedding-based entity alignment. In this paper, we propose a novel framework that unifies multiple views of entities to learn embeddings for entity alignment. Specifically, we embed entities based on the views of entity names, relations and attributes, with several combination strategies. Furthermore, we design some cross-KG inference methods to enhance the alignment between two KGs. Our experiments on real-world datasets show that the proposed framework significantly outperforms the state-of-the-art embedding-based entity alignment methods. The selected views, cross-KG inference and combination strategies all contribute to the performance improvement.

Learning to Exploit Long-term Relational Dependencies in Knowledge Graphs

Abstract: We study the problem of knowledge graph (KG) embedding. A widely-established assumption to this problem is that similar entities are likely to have similar relational roles. However, existing related methods derive KG embeddings mainly based on triple-level learning, which lack the capability of capturing long-term relational dependencies of entities. Moreover, triple-level learning is insufficient for the propagation of semantic information among entities, especially for the case of cross-KG embedding. In this paper, we propose recurrent skipping networks (RSNs), which employ a skipping mechanism to bridge the gaps between entities. RSNs integrate recurrent neural networks (RNNs) with residual learning to efficiently capture the long-term relational dependencies within and between KGs. We design an end-to-end framework to support RSNs on different tasks. Our experimental results showed that RSNs outperformed state-of-the-art embedding-based methods for entity alignment and achieved competitive performance for KG completion.

Cross-lingual Knowledge Graph Alignment via Graph Matching Neural Network

Abstract: Previous cross-lingual knowledge graph (KG) alignment studies rely on entity embeddings derived only from monolingual KG structural information, which may fail at matching entities that have different facts in two KGs. In this paper, we introduce the topic entity graph, a local sub-graph of an entity, to represent entities with their contextual information in KG. From this view, the KB-alignment task can be formulated as a graph matching problem; and we further propose a graph-attention based solution, which first matches all entities in two topic entity graphs, and then jointly model the local matching information to derive a graph-level matching vector. Experiments show that our model outperforms previous state-of-the-art methods by a large margin.

Multi-Channel Graph Neural Network for Entity Alignment

Abstract: Entity alignment typically suffers from the issues of structural heterogeneity and limited seed alignments. In this paper, we propose a novel Multi-channel Graph Neural Network model (MuGNN) to learn alignment-oriented knowledge graph (KG) embeddings by robustly encoding two KGs via multiple channels. Each channel encodes KGs via different relation weighting schemes with respect to self-attention towards KG completion and cross-KG attention for pruning exclusive entities respectively, which are further combined via pooling techniques. Moreover, we also infer and transfer rule knowledge for completing two KGs consistently. MuGNN is expected to reconcile the structural differences of two KGs, and thus make better use of seed alignments. Extensive experiments on five publicly available datasets demonstrate our superior performance (5% Hits@1 up on average). Source code and data used in the experiments can be accessed at https://github.com/thunlp/MuGNN .

Overview of related work in the field of entity alignment for knowledge graphs with their used datasets and metrics.

In this section we review previous work for the entity alignment for Knowledge Graphs and revise datasets and current evaluation process. We believe this is useful for practitioners, since we discover some pitfalls, especially when implementing evaluation scores and selecting datasets for comparison. The overview of methods, datasets and metrics is provided in Table {{table:2cea3de1-7beb-40cf-948d-8146355fad68}} .


WebVision Database: Visual Learning and Understanding from Web Data

Abstract: In this paper, we present a study on learning visual recognition models from large scale noisy web data. We build a new database called WebVision, which contains more than $2.4$ million web images crawled from the Internet by using queries generated from the 1,000 semantic concepts of the benchmark ILSVRC 2012 dataset. Meta information along with those web images (e.g., title, description, tags, etc.) are also crawled. A validation set and test set containing human annotated images are also provided to facilitate algorithmic development. Based on our new database, we obtain a few interesting observations: 1) the noisy web images are sufficient for training a good deep CNN model for visual recognition; 2) the model learnt from our WebVision database exhibits comparable or even better generalization ability than the one trained from the ILSVRC 2012 dataset when being transferred to new datasets and tasks; 3) a domain adaptation issue (a.k.a., dataset bias) is observed, which means the dataset can be used as the largest benchmark dataset for visual domain adaptation. Our new WebVision database and relevant studies in this work would benefit the advance of learning state-of-the-art visual models with minimum supervision based on web data.

CleanNet: Transfer Learning for Scalable Image Classifier Training with Label Noise

Abstract: In this paper, we study the problem of learning image classification models with label noise. Existing approaches depending on human supervision are generally not scalable as manually identifying correct or incorrect labels is time-consuming, whereas approaches not relying on human supervision are scalable but less effective. To reduce the amount of human supervision for label noise cleaning, we introduce CleanNet, a joint neural embedding network, which only requires a fraction of the classes being manually verified to provide the knowledge of label noise that can be transferred to other classes. We further integrate CleanNet and conventional convolutional neural network classifier into one framework for image classification learning. We demonstrate the effectiveness of the proposed algorithm on both of the label noise detection task and the image classification on noisy data task on several large-scale datasets. Experimental results show that CleanNet can reduce label noise detection error rate on held-out classes where no human supervision available by 41.5% compared to current weakly supervised methods. It also achieves 47% of the performance gain of verifying all images with only 3.2% images verified on an image classification task. Source code and dataset will be available at kuanghuei.github.io/CleanNetProject.

Main data sets used in literature for noisy labels.

Table {{table:72e72171-c40a-41b1-96f3-9bf6829a746e}}  shows the main information about the most used data sets for evaluation of solutions in noisy label environments.


Multimodal Unsupervised Image-to-Image Translation

Abstract: Unsupervised image-to-image translation is an important and challenging problem in computer vision. Given an image in the source domain, the goal is to learn the conditional distribution of corresponding images in the target domain, without seeing any pairs of corresponding images. While this conditional distribution is inherently multimodal, existing approaches make an overly simplified assumption, modeling it as a deterministic one-to-one mapping. As a result, they fail to generate diverse outputs from a given source domain image. To address this limitation, we propose a Multimodal Unsupervised Image-to-image Translation (MUNIT) framework. We assume that the image representation can be decomposed into a content code that is domain-invariant, and a style code that captures domain-specific properties. To translate an image to another domain, we recombine its content code with a random style code sampled from the style space of the target domain. We analyze the proposed framework and establish several theoretical results. Extensive experiments with comparisons to the state-of-the-art approaches further demonstrates the advantage of the proposed framework. Moreover, our framework allows users to control the style of translation outputs by providing an example style image. Code and pretrained models are available at https://github.com/nvlabs/MUNIT

One-Shot Unsupervised Cross Domain Translation

Abstract: Given a single image x from domain A and a set of images from domain B, our task is to generate the analogous of x in B. We argue that this task could be a key AI capability that underlines the ability of cognitive agents to act in the world and present empirical evidence that the existing unsupervised domain translation methods fail on this task. Our method follows a two step process. First, a variational autoencoder for domain B is trained. Then, given the new sample x, we create a variational autoencoder for domain A by adapting the layers that are close to the image in order to directly fit x, and only indirectly adapt the other layers. Our experiments indicate that the new method does as well, when trained on one sample x, as the existing domain transfer methods, when these enjoy a multitude of training samples from domain A. Our code is made publicly available at this https URL

Image to Image Translation for Domain Adaptation

Abstract: We propose a general framework for unsupervised domain adaptation, which allows deep neural networks trained on a source domain to be tested on a different target domain without requiring any training annotations in the target domain. This is achieved by adding extra networks and losses that help regularize the features extracted by the backbone encoder network. To this end we propose the novel use of the recently proposed unpaired image-toimage translation framework to constrain the features extracted by the encoder network. Specifically, we require that the features extracted are able to reconstruct the images in both domains. In addition we require that the distribution of features extracted from images in the two domains are indistinguishable. Many recent works can be seen as specific cases of our general framework. We apply our method for domain adaptation between MNIST, USPS, and SVHN datasets, and Amazon, Webcam and DSLR Office datasets in classification tasks, and also between GTA5 and Cityscapes datasets for a segmentation task. We demonstrate state of the art performance on each of these datasets.

Comparison of losses and network architecture between the proposed method and our baselines. Feature cycle A (resp. B) is the feature cycle loss for images encoded from domain {{formula:8595f768-34f3-4723-b369-512caf5d601f}} . GAN Z is the loss used by {{cite:33dd507}} for domain confusion in the latent space. GAN A (resp. B) is the GAN loss requiring the translated images from domain {{formula:b9e6f812-26dc-46ec-ac44-0e4deac81bb2}}  to match the distribution of domain {{formula:14861258-4b5d-4386-a8d0-aeddb12128d3}} . "Disentangled" refers to the feature cycle presented in {{cite:e221d09}}, where there are separated encoders for style and content, which results in two different latent spaces correspondingly. Our feature cycle involves a unified latent space.

Tab. {{table:81f265c2-510e-464b-a90a-8145aa625d1e}}  summarizes the differences in losses and network architecture between the proposed method and the baselines, as well as a recent domain adaptation work (a different but related task) called I2I work {{cite:33dd507}}.


SqueezeNext: Hardware-Aware Neural Network Design

Abstract: One of the main barriers for deploying neural networks on embedded systems has been large memory and power consumption of existing neural networks. In this work, we introduce SqueezeNext, a new family of neural network architectures whose design was guided by considering previous architectures such as SqueezeNet, as well as by simulation results on a neural network accelerator. This new network is able to match AlexNet's accuracy on the ImageNet benchmark with 112× fewer parameters, and one of its deeper variants is able to achieve VGG-19 accuracy with only 4.4 Million parameters, (31× smaller than VGG-19). SqueezeNext also achieves better top-5 classification accuracy with 1.3× fewer parameters as compared to MobileNet, but avoids using depthwise-separable convolutions that are inefficient on some mobile processor platforms. This wide range of accuracy gives the user the ability to make speed-accuracy tradeoffs, depending on the available resources on the target hardware. Using hardware simulation results for power and inference speed on an embedded system has guided us to design variations of the baseline model that are 2.59×/8.26× faster and 2.25×/7.5× more energy efficient as compared to SqueezeNet/AlexNet without any accuracy degradation.

MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications

Abstract: We present a class of efficient models called MobileNets for mobile and embedded vision applications. MobileNets are based on a streamlined architecture that uses depth-wise separable convolutions to build light weight deep neural networks. We introduce two simple global hyper-parameters that efficiently trade off between latency and accuracy. These hyper-parameters allow the model builder to choose the right sized model for their application based on the constraints of the problem. We present extensive experiments on resource and accuracy tradeoffs and show strong performance compared to other popular models on ImageNet classification. We then demonstrate the effectiveness of MobileNets across a wide range of applications and use cases including object detection, finegrain classification, face attributes and large scale geo-localization.

MobileNetV2: Inverted Residuals and Linear Bottlenecks

Abstract: In this paper we describe a new mobile architecture, MobileNetV2, that improves the state of the art performance of mobile models on multiple tasks and benchmarks as well as across a spectrum of different model sizes. We also describe efficient ways of applying these mobile models to object detection in a novel framework we call SSDLite. Additionally, we demonstrate how to build mobile semantic segmentation models through a reduced form of DeepLabv3 which we call Mobile DeepLabv3. is based on an inverted residual structure where the shortcut connections are between the thin bottleneck layers. The intermediate expansion layer uses lightweight depthwise convolutions to filter features as a source of non-linearity. Additionally, we find that it is important to remove non-linearities in the narrow layers in order to maintain representational power. We demonstrate that this improves performance and provide an intuition that led to this design. Finally, our approach allows decoupling of the input/output domains from the expressiveness of the transformation, which provides a convenient framework for further analysis. We measure our performance on ImageNet [1] classification, COCO object detection [2], VOC image segmentation [3]. We evaluate the trade-offs between accuracy, and number of operations measured by multiply-adds (MAdd), as well as actual latency, and the number of parameters.

ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices

Abstract: We introduce an extremely computation-efficient CNN architecture named ShuffleNet, which is designed specially for mobile devices with very limited computing power (e.g., 10-150 MFLOPs). The new architecture utilizes two new operations, pointwise group convolution and channel shuffle, to greatly reduce computation cost while maintaining accuracy. Experiments on ImageNet classification and MS COCO object detection demonstrate the superior performance of ShuffleNet over other structures, e.g. lower top-1 error (absolute 7.8%) than recent MobileNet [12] on ImageNet classification task, under the computation budget of 40 MFLOPs. On an ARM-based mobile device, ShuffleNet achieves ~13Ã— actual speedup over AlexNet while maintaining comparable accuracy.

Densely Connected Convolutional Networks

Abstract: Recent work has shown that convolutional networks can be substantially deeper, more accurate, and efficient to train if they contain shorter connections between layers close to the input and those close to the output. In this paper, we embrace this observation and introduce the Dense Convolutional Network (DenseNet), which connects each layer to every other layer in a feed-forward fashion. Whereas traditional convolutional networks with L layers have L connections—one between each layer and its subsequent layer—our network has L(L+1)/2 direct connections. For each layer, the feature-maps of all preceding layers are used as inputs, and its own feature-maps are used as inputs into all subsequent layers. DenseNets have several compelling advantages: they alleviate the vanishing-gradient problem, strengthen feature propagation, encourage feature reuse, and substantially reduce the number of parameters. We evaluate our proposed architecture on four highly competitive object recognition benchmark tasks (CIFAR-10, CIFAR-100, SVHN, and ImageNet). DenseNets obtain significant improvements over the state-of-the-art on most of them, whilst requiring less memory and computation to achieve high performance. Code and pre-trained models are available at https://github.com/liuzhuang13/DenseNet.

Going deeper with convolutions

Abstract: We propose a deep convolutional neural network architecture codenamed Inception that achieves the new state of the art for classification and detection in the ImageNet Large-Scale Visual Recognition Challenge 2014 (ILSVRC14). The main hallmark of this architecture is the improved utilization of the computing resources inside the network. By a carefully crafted design, we increased the depth and width of the network while keeping the computational budget constant. To optimize quality, the architectural decisions were based on the Hebbian principle and the intuition of multi-scale processing. One particular incarnation used in our submission for ILSVRC14 is called GoogLeNet, a 22 layers deep network, the quality of which is assessed in the context of classification and detection.

Rethinking the Inception Architecture for Computer Vision

Abstract: Convolutional networks are at the core of most state of-the-art computer vision solutions for a wide variety of tasks. Since 2014 very deep convolutional networks started to become mainstream, yielding substantial gains in various benchmarks. Although increased model size and computational cost tend to translate to immediate quality gains for most tasks (as long as enough labeled data is provided for training), computational efficiency and low parameter count are still enabling factors for various use cases such as mobile vision and big-data scenarios. Here we are exploring ways to scale up networks in ways that aim at utilizing the added computation as efficiently as possible by suitably factorized convolutions and aggressive regularization. We benchmark our methods on the ILSVRC 2012 classification challenge validation set demonstrate substantial gains over the state of the art: 21:2% top-1 and 5:6% top-5 error for single frame evaluation using a network with a computational cost of 5 billion multiply-adds per inference and with using less than 25 million parameters. With an ensemble of 4 models and multi-crop evaluation, we report 3:5% top-5 error and 17:3% top-1 error on the validation set and 3:6% top-5 error on the official test set.

Squeeze-and-Excitation Networks

Abstract: Convolutional neural networks are built upon the convolution operation, which extracts informative features by fusing spatial and channel-wise information together within local receptive fields. In order to boost the representational power of a network, several recent approaches have shown the benefit of enhancing spatial encoding. In this work, we focus on the channel relationship and propose a novel architectural unit, which we term the "Squeeze-and-Excitation" (SE) block, that adaptively recalibrates channel-wise feature responses by explicitly modelling interdependencies between channels. We demonstrate that by stacking these blocks together, we can construct SENet architectures that generalise extremely well across challenging datasets. Crucially, we find that SE blocks produce significant performance improvements for existing state-of-the-art deep architectures at minimal additional computational cost. SENets formed the foundation of our ILSVRC 2017 classification submission which won first place and significantly reduced the top-5 error to 2.251%, achieving a ~25% relative improvement over the winning entry of 2016. Code and models are available at https://github.com/hujie-frank/SENet.

DNN groups with similar architectural building blocks. In each group, the DNN shown in bold is the reference DNN for our experiments.

Power measurement on GPU: We use an inbuilt nvidia-smi utility to measure the power on Nvidia's GPUs. Since nvidia-smi is a high-level utility and sampling rate depends on that of the inbuilt power sensor in GPU {{cite:ab67de1}}, accurately measuring the power consumption on GPU is a non-trivial task and requires extra care. The inbuilt sensor's sampling rate is not only quite low but also GPU specific, i.e., varies from GPU to GPU. For instance, on P100 and P4000 GPUs, the sampling rate is {{formula:ed18acfc-19e6-481b-b3fa-efd9aca7e552}}  and {{formula:8facf5cf-075b-4711-bba6-51dd334c11d0}} , respectively. To circumvent the challenge mentioned above, we run DNNs for a higher number of iterations (forward pass and backward pass simultaneously) and get the stabilized power readings. More precisely, at batch size one, we run 500 iterations for DNNs with lower inference time (e.g., SqueezeNet and GoogLeNet), and 100 iterations for the DNNs with higher inference time. Note that at larger batch sizes, we run the same number of iterations for all the DNNs (in Table {{table:0bb95bbc-2bf6-4f22-b9b5-39c4281b19bd}} ). Similar to {{cite:491ca54}}, we sample the readings of nvidia-smi tool at every 0.1 ms, and once the power reading becomes stable, we take the average power (reported by nvidia-smi) as the power consumption of a DNN.

Distinctive Properties of Building Blocks in Compact DNNs

In this section, we first discuss the architectural building blocks in compact DNNs, such as fire module, inception module, dense block, along with their fine-grained architectural components, such as branching and residual/skip connections. (Table {{table:0bb95bbc-2bf6-4f22-b9b5-39c4281b19bd}} ). This analysis of architectural ingredients in compact DNNs would help in reverse engineer the basic building blocks in compact DNNs through the side-channel attacks. Then, we show the GPU kernel (cuBLAS) profiling results for all the compact DNNs listed in Table {{table:0bb95bbc-2bf6-4f22-b9b5-39c4281b19bd}} . We then analyze the properties of these cuBLAS kernels, which would help decipher the fine-grained architectural components present in building blocks of compact DNNs.

Architectural Building Blocks in Compact DNNs

Most compact DNN designs employ a single building block in a repeated manner to reduce the number of design hyper-parameter choices and make design optimization easier. Also, it makes DNN's design more regular and modular. These basic building blocks in compact DNNs can be broadly categorized as fire module, depthwise convolution (DWConv), channel shuffling, dense block, and inception module (Table {{table:0bb95bbc-2bf6-4f22-b9b5-39c4281b19bd}} ).
For example, SqueezeNet and SqueezeNext use the fire module, whereas MobileNet and ShuffleNet DNNs use DWConv.

The building-blocks mentioned above in compact DNNs are made up of fine-grained architectural ingredients illustrated in the last four columns of Table {{table:0bb95bbc-2bf6-4f22-b9b5-39c4281b19bd}} . For example, dense blocks have skip connections, whereas the fire module in SqueezeNet has pointwise convolution (PWConv) and branching. We notice that building blocks in compact DNNs share standard fine-grained architectural components, but they are arranged in a different manner. Thus, our intuition is, these basic blocks would exhibit the distinctive characteristics which can be exploited by an adversary to reverse engineer the architecture of these basic blocks.

To understand the effect of architectural building blocks on GPU's compute and memory resource utilization, we perform kernel-level profiling using nvprof on P100 GPU. We analyze three cuBLAS kernels Gemmk1, Gemv2T and Gemv2N because these kernels are present in all the compact DNNs listed in Table {{table:0bb95bbc-2bf6-4f22-b9b5-39c4281b19bd}} , except SqueezeNet variants. The percentage contribution of the aforementioned cuBLAS kernels in one iteration of forward and backward pass (successively) is shown in Table {{table:c355160f-80da-40a2-9b6b-12f6c9cc497d}} .

Notice that SqueezeNext variants have very high {{formula:9a55ecb7-36f1-40fc-a3b3-18df29a78760}}  ratio and 1.0-G-SqNxt-23 has higher {{formula:2fa54fdb-c0c6-42f5-9f1c-75f0d0ec1535}}  (Table {{table:b1701b30-2ba2-4daa-b5b4-96d1a828c20c}} ) than AlexNet even when the former has 112{{formula:d8b196d2-a780-408e-8334-2ee2803595d9}}  fewer {{formula:7186f8e0-466d-4f03-8a75-7abd5ba0db7b}}  than the latter (Table {{table:ab807230-3747-4050-81d0-01eb1995adc6}} ) . The fire module in SqueezeNext has all the fine-grained architectural components listed in last four columns in Table {{table:0bb95bbc-2bf6-4f22-b9b5-39c4281b19bd}}  which renders {{formula:99880f00-d47b-4163-8493-9e8c244a5428}}  ratio very high.
As shown in Figure {{figure:572602c2-79d7-4636-bcd0-f10d42b16a25}} (a) and {{figure:572602c2-79d7-4636-bcd0-f10d42b16a25}} (c), the increase in {{formula:48767df5-9668-4248-b781-2442033cd438}}  with {{formula:9f003355-adf8-4212-8d86-b97257664c6f}}  for all the compact DNNs is substantially high compared to that in AlexNet. In fact, in DenseNet, {{formula:90900880-a220-43de-bbb5-3f09cd0bd6c8}}  increases exponentially with {{formula:d2733442-5e50-48b3-9393-d07726a9c51e}}  because the number of skip connections in dense blocks vary exponentially with the number of layers {{cite:a966d9c}}.

In this section, we describe our proposed two-stage attack methodology, “DeepPeep,” which exposes the crucial security vulnerabilities present in compact DNNs due to its design principles. “DeepPeep,” decipher the architecture of the compact DNN (termed “victim DNN”) running on GPU in a cloud. Based on the similarity of the building block, we group the compact DNNs into six groups, as shown in Table {{table:0bb95bbc-2bf6-4f22-b9b5-39c4281b19bd}} . In each group, we take one DNN as the reference, which is highlighted as bold in Table {{table:0bb95bbc-2bf6-4f22-b9b5-39c4281b19bd}} . The attack proceeds in two stages:

Figure {{figure:95c9a6d3-a961-489d-acd0-637dc20572e0}}  summarizes the steps for inter-group prediction stage of DeepPeep attack model. In this prediction stage, the inputs are {{formula:1ba92c33-da98-4918-a926-6ed3ae15ef0c}} , {{formula:44478667-dfa8-42a1-a511-9e378285385a}} , {{formula:67a44b15-1cba-4b98-ab92-653bb88f8ce4}} , {{formula:9d36dc59-0869-4096-b262-aed39b5aa646}} , {{formula:37af142d-45d6-4dc0-a033-104cc7b83176}} , and percentage contribution of cuBLAS kernels Gemv2T, Gemv2N and Gemmk1 (Table {{table:c355160f-80da-40a2-9b6b-12f6c9cc497d}} ). The outputs of inter-group prediction stage are the DNN groups listed in Table {{table:0bb95bbc-2bf6-4f22-b9b5-39c4281b19bd}} . We show the {{formula:fda6437d-fcc7-404b-a62c-e98a76541894}}  and {{formula:8eac0436-9326-43c7-9526-e725574d3a03}}  values in Figure {{figure:77019101-af5e-4b2b-8e88-f1843411fdf9}} , and {{formula:1f25e298-9f94-4367-b688-00fe291f9dcb}}  and {{formula:45662317-f921-4429-af14-a68ac0bf407f}}  ratio in Figure {{figure:572602c2-79d7-4636-bcd0-f10d42b16a25}} . We experimentally demonstrate the variation of aforementioned metrics with the batch size ({{formula:0e1cecb7-36b4-4c3d-aed3-a010f1f26ce8}} ) on both P100 and P4000 GPUs.

Once the group of a DNN is known, we further predict the exact DNN in the group, through side-channel attacks. We choose one DNN in each group as the reference DNN (shown in bold in Table {{table:0bb95bbc-2bf6-4f22-b9b5-39c4281b19bd}} ). In the intra-group prediction stage of DeepPeep attack model, shown in Figure {{figure:1f9137b0-3676-441f-a5a5-0ecef11fdfe5}} , we compare the characteristics of the victim DNN with those of the reference DNN. Since DNNs in most of the groups exhibit very distinct characteristics at a larger {{formula:7e3511db-553d-4594-b734-e8df29947d8b}} , they are easily distinguishable through side-channel attacks. Since the DenseNet group has only one DNN, there is no need to perform intra-group prediction.

Applicability on Other DNN Architectures

There are two broad strategies for designing compact DNNs. The first strategy seeks to significantly reduce the computational complexity and/or the number of parameters with a negligible loss in predictive performance. Examples of this strategy include DWConv in MobileNet and ShuffleNet variants, and fire module in SqueezeNet and SqueezeNext variants. The second strategy boosts the representational power of DNNs in a fixed budget of MAC operations and the number of parameters. The representational power of DNNs is improved either by improving feature reuse or through multi-scale feature learning. DNNs employ residual connections (element-wise addition) and skip connections (concatenation of fmaps from previous layers) to improve the feature reuse {{cite:053ba39}}. Residual connections have been implemented in an aggressive version of the fire module, which forms the building blocks of SqueezeNext variants {{cite:79b949b}}. Skip connections have been implemented in the dense blocks of DenseNet {{cite:a966d9c}}. Evidently, the compact DNNs used in our experiments (Table {{table:0bb95bbc-2bf6-4f22-b9b5-39c4281b19bd}} ) are employed all the commonly used design heuristics in the architecture of building blocks. Besides, we have included compact DNNs that use design techniques, such as channel shuffling, which are not commonly used in the design of compact DNNs. In summary, the compact DNNs used in our experiments (Table {{table:0bb95bbc-2bf6-4f22-b9b5-39c4281b19bd}} ) employ an extensive set of design heuristics, and hence, they are a good representative of state-of-the-art compact DNNs.

Reasons for Human-Observation Based Approach in “DeepPeep”

2. Sample size: Neural networks require a huge dataset to learn and develop the experience. Transfer learning can mitigate this issue; however, it is limited to only correlated problems. That is, neural networks are quite inefficient, as compared to a human, to find the relationship between entirely different experiences. As shown in Table {{table:0bb95bbc-2bf6-4f22-b9b5-39c4281b19bd}} , we have very few data points (DNNs) for each fine-grained architectural components and building blocks.


Detect: Define approach of detecting trolls on Twitter based on textual features

Detect: Develop machine learning models to detect trolls, apply the models on currently active accounts

Describe: Characterize image-posting activity of trolls

Describe; assess the impact: Characterize trolls' operations and their influence on the greater Web

Describe; assess the impact: Identify and compare strategies of different groups of trolls (IRA vs Iranian), analyze the impact

TexTrolls: Identifying Russian Trolls on Twitter from a Textual Perspective

Abstract: The online new emerging suspicious users, that usually are called trolls, are one of the main sources of hate, fake, and deceptive online messages. Some agendas are utilizing these harmful users to spread incitement tweets, and as a consequence, the audience get deceived. The challenge in detecting such accounts is that they conceal their identities which make them disguised in social media, adding more difficulty to identify them using just their social network information. Therefore, in this paper, we propose a text-based approach to detect the online trolls such as those that were discovered during the US 2016 presidential elections. Our approach is mainly based on textual features which utilize thematic information, and profiling features to identify the accounts from their way of writing tweets. We deduced the thematic information in a unsupervised way and we show that coupling them with the textual features enhanced the performance of the proposed model. In addition, we find that the proposed profiling features perform the best comparing to the textual features.

Still out there: Modeling and Identifying Russian Troll Accounts on Twitter

Abstract: There is evidence that Russia’s Internet Research Agency attempted to interfere with the 2016 U.S. election by running fake accounts on Twitter—often referred to as “Russian trolls”. In this work, we: 1) develop machine learning models that predict whether a Twitter account is a Russian troll within a set of 170K control accounts; and, 2) demonstrate that it is possible to use this model to find active accounts on Twitter still likely acting on behalf of the Russian state. Using both behavioral and linguistic features, we show that it is possible to distinguish between a troll and a non-troll with a precision of 78.5% and an AUC of 98.9%, under cross-validation. Applying the model to out-of-sample accounts still active today, we find that up to 2.6% of top journalists’ mentions are occupied by Russian trolls. These findings imply that the Russian trolls are very likely still active today. Additional analysis shows that they are not merely software-controlled bots, and manage their online identities in various complex ways. Finally, we argue that if it is possible to discover these accounts using externally-accessible data, then the platforms—with access to a variety of private internal signals—should succeed at similar or better rates.

Characterizing the Use of Images by State-Sponsored Troll Accounts on Twitter

Abstract:
Anecdotal evidence has emerged suggesting that state-sponsored organizations, like the Russian Internet Research Agency, have exploited mainstream social. Their primary goal is apparently to conduct information warfare operations to manipulate public opinion using accounts disguised as "normal" people. To increase engagement and credibility of their posts, these accounts regularly share images. However, the use of images by state-sponsored accounts has yet to be examined by the research community.
In this work, we address this gap by analyzing a ground truth dataset of 1.8M images posted to Twitter by called Russian trolls. More specifically, we analyze the content of the images, as well as the posting activity of the accounts. Among other things, we find that image posting activity of Russian trolls is tightly coupled with real-world events, and that their targets, as well as the content shared, changed over time. When looking at the interplay between domains that shared the same images as state-sponsored trolls, we find clear cut differences in the origin and/or spread of images across the Web. Overall, our findings provide new insight into how state-sponsored trolls operate, and specifically how they use imagery to achieve their goals.

Disinformation Warfare: Understanding State-Sponsored Trolls on Twitter and Their Influence on the Web

Abstract: Over the past couple of years, anecdotal evidence has emerged linking coordinated campaigns by state-sponsored actors with efforts to manipulate public opinion on the Web, often around major political events, through dedicated accounts, or “trolls.” Although they are often involved in spreading disinformation on social media, there is little understanding of how these trolls operate, what type of content they disseminate, and most importantly their influence on the information ecosystem. In this paper, we shed light on these questions by analyzing 27K tweets posted by 1K Twitter users identified as having ties with Russia’s Internet Research Agency and thus likely state-sponsored trolls. We compare their behavior to a random set of Twitter users, finding interesting differences in terms of the content they disseminate, the evolution of their account, as well as their general behavior and use of Twitter. Then, using Hawkes Processes, we quantify the influence that trolls had on the dissemination of news on social platforms like Twitter, Reddit, and 4chan. Overall, our findings indicate that Russian trolls managed to stay active for long periods of time and to reach a substantial number of Twitter users with their tweets. When looking at their ability of spreading news content and making it viral, however, we find that their effect on social platforms was minor, with the significant exception of news published by the Russian state-sponsored news outlet RT (Russia Today).

Who Let The Trolls Out?: Towards Understanding State-Sponsored Trolls

Abstract: Recent evidence has emerged linking coordinated campaigns by state-sponsored actors to manipulate public opinion on the Web. Campaigns revolving around major political events are enacted via mission-focused ?trolls." While trolls are involved in spreading disinformation on social media, there is little understanding of how they operate, what type of content they disseminate, how their strategies evolve over time, and how they influence the Web's in- formation ecosystem. In this paper, we begin to address this gap by analyzing 10M posts by 5.5K Twitter and Reddit users identified as Russian and Iranian state-sponsored trolls. We compare the behavior of each group of state-sponsored trolls with a focus on how their strategies change over time, the different campaigns they embark on, and differences between the trolls operated by Russia and Iran. Among other things, we find: 1) that Russian trolls were pro-Trump while Iranian trolls were anti-Trump; 2) evidence that campaigns undertaken by such actors are influenced by real-world events; and 3) that the behavior of such actors is not consistent over time, hence detection is not straightforward. Using Hawkes Processes, we quantify the influence these accounts have on pushing URLs on four platforms: Twitter, Reddit, 4chan's Politically Incorrect board (/pol/), and Gab. In general, Russian trolls were more influential and efficient in pushing URLs to all the other platforms with the exception of /pol/ where Iranians were more influential. Finally, we release our source code to ensure the reproducibility of our results and to encourage other researchers to work on understanding other emerging kinds of state-sponsored troll accounts on Twitter.

Quantitative studies of the activity of organized trolls on social media

The data on troll accounts and tweets released by Twitter has started a wave of quantitative research on Russian trolls. Table {{table:41a45a88-485a-449d-b550-9bfde834ddce}}  provides an overview of such studies.

Quantitative troll studies have typically analyzed textual data from Twitter (Table {{table:41a45a88-485a-449d-b550-9bfde834ddce}} ). However, a few studies also examined Facebook ads purchased by IRA {{cite:9788488}}, images posted on Twitter {{cite:d386dd1}}, as well as posts on other social networks, such as Reddit {{cite:8cfd0b0}} and the blogging platform LiveJournal {{cite:b722456}} (popular in Russia). Furthermore, the first multi-platform troll studies have started to appear {{cite:76dc46b}}.


Diagnosis the identified patients to classify (in to patients' categories) and tracking the progress Covid-19 patients

Build a framework which uses smartphone sensors to detect Covid-19

Classify patients in to non-COVID 19 infection, COVID-19 infection, and no infection from X-ray images using deep CNN model

Detect COVID-19 from X-ray images using transfer learning with CNN

Detect COVID-19 from X-ray images using deep CNN model

Develop a deep learning model CoroNet using the Xception CNN to detect Covid-19 from Xray images

Detect COVID-19 from X-ray images using Generative Adversarial Network (GAN) and Deep learning transfer

Efficiently diagnosis COVID-19 using X-ray images through deep CNN models

Develop a system to improve the diagnostic performance from Posterior-Anterior (PA) X-ray images of lungs with COVID-19 cases

Forecast of the COVID-19 to estimate size, lengths and ending time of COVID-19 across China

Analyze the correlation among environmental factors and confirmed cases of COVID-19

Rapid AI Development Cycle for the Coronavirus (COVID-19) Pandemic: Initial Results for Automated Detection & Patient Monitoring using Deep Learning CT Image Analysis

Abstract: Purpose: Develop AI-based automated CT image analysis tools for detection, quantification, and tracking of Coronavirus; demonstrate they can differentiate coronavirus patients from non-patients. Materials and Methods: Multiple international datasets, including from Chinese disease-infected areas were included. We present a system that utilizes robust 2D and 3D deep learning models, modifying and adapting existing AI models and combining them with clinical understanding. We conducted multiple retrospective experiments to analyze the performance of the system in the detection of suspected COVID-19 thoracic CT features and to evaluate evolution of the disease in each patient over time using a 3D volume review, generating a Corona score. The study includes a testing set of 157 international patients (China and U.S). Results: Classification results for Coronavirus vs Non-coronavirus cases per thoracic CT studies were 0.996 AUC (95%CI: 0.989-1.00) ; on datasets of Chinese control and infected patients. Possible working point: 98.2% sensitivity, 92.2% specificity. For time analysis of Coronavirus patients, the system output enables quantitative measurements for smaller opacities (volume, diameter) and visualization of the larger opacities in a slice-based heat map or a 3D volume display. Our suggested Corona score measures the progression of disease over time. Conclusion: This initial study, which is currently being expanded to a larger population, demonstrated that rapidly developed AI-based image analysis can achieve high accuracy in detection of Coronavirus as well as quantification and tracking of disease burden.

A Novel AI-enabled Framework to Diagnose Coronavirus COVID-19 using Smartphone Embedded Sensors: Design Study

Abstract: Coronaviruses are a famous family of viruses that cause illness in both humans and animals. The new type of coronavirus COVID-19 was firstly discovered in Wuhan, China. However, recently, the virus has widely spread in most of the world and causing a pandemic according to the World Health Organization (WHO). Further, nowadays, all the world countries are striving to control the COVID-19. There are many mechanisms to detect coronavirus including clinical analysis of chest CT scan images and blood test results. The confirmed COVID-19 patient manifests as fever, tiredness, and dry cough. Particularly, several techniques can be used to detect the initial results of the virus such as medical detection Kits. However, such devices are incurring huge cost, taking time to install them and use. Therefore, in this paper, a new framework is proposed to detect COVID-19 using built-in smartphone sensors. The proposal provides a low-cost solution, since most of radiologists have already held smartphones for different daily-purposes. Not only that but also ordinary people can use the framework on their smartphones for the virus detection purposes. Today’s smartphones are powerful with existing computation-rich processors, memory space, and large number of sensors including cameras, microphone, temperature sensor, inertial sensors, proximity, colour-sensor, humidity-sensor, and wireless chipsets/sensors. The designed Artificial Intelligence (AI) enabled framework reads the smartphone sensors’ signal measurements to predict the grade of severity of the pneumonia as well as predicting the result of the disease.

COVID-Net: A Tailored Deep Convolutional Neural Network Design for Detection of COVID-19 Cases from Chest X-Ray Images

Abstract: The COVID-19 pandemic continues to have a devastating effect on the health and well-being of the global population. A critical step in the fight against COVID-19 is effective screening of infected patients, with one of the key screening approaches being radiology examination using chest radiography. Motivated by this and inspired by the open source efforts of the research community, in this study we introduce COVID-Net, a deep convolutional neural network design tailored for the detection of COVID-19 cases from chest X-ray (CXR) images that is open source and available to the general public. To the best of the authors' knowledge, COVID-Net is one of the first open source network designs for COVID-19 detection from CXR images at the time of initial release. We also introduce COVIDx, an open access benchmark dataset that we generated comprising of 13,975 CXR images across 13,870 patient patient cases, with the largest number of publicly available COVID-19 positive cases to the best of the authors' knowledge. Furthermore, we investigate how COVID-Net makes predictions using an explainability method in an attempt to not only gain deeper insights into critical factors associated with COVID cases, which can aid clinicians in improved screening, but also audit COVID-Net in a responsible and transparent manner to validate that it is making decisions based on relevant information from the CXR images. By no means a production-ready solution, the hope is that the open access COVID-Net, along with the description on constructing the open source COVIDx dataset, will be leveraged and build upon by both researchers and citizen data scientists alike to accelerate the development of highly accurate yet practical deep learning solutions for detecting COVID-19 cases and accelerate treatment of those who need it the most.

Covid-19: Automatic detection from X-Ray images utilizing Transfer Learning with Convolutional Neural Networks

Abstract: In this study, a dataset of X-Ray images from patients with common pneumonia, Covid-19, and normal incidents was utilized for the automatic detection of the Coronavirus. The aim of the study is to evaluate the performance of state-of-the-art Convolutional Neural Network architectures proposed over recent years for medical image classification. Specifically, the procedure called transfer learning was adopted. With transfer learning, the detection of various abnormalities in small medical image datasets is an achievable target, often yielding remarkable results. The dataset utilized in this experiment is a collection of 1427 X-Ray images. 224 images with confirmed Covid-19, 700 images with confirmed common pneumonia, and 504 images of normal conditions are included. The data was collected from the available X-Ray images on public medical repositories. With transfer learning, an overall accuracy of 97.82% in the detection of Covid-19 is achieved.

Automatic Detection of Coronavirus Disease (COVID-19) Using X-ray Images and Deep Convolutional Neural Networks

Abstract: The 2019 novel coronavirus disease (COVID-19), with a starting point in China, has spread rapidly among people living in other countries, and is approaching approximately 34,986,502 cases worldwide according to the statistics of European Centre for Disease Prevention and Control. There are a limited number of COVID-19 test kits available in hospitals due to the increasing cases daily. Therefore, it is necessary to implement an automatic detection system as a quick alternative diagnosis option to prevent COVID-19 spreading among people. In this study, five pre-trained convolutional neural network based models (ResNet50, ResNet101, ResNet152, InceptionV3 and Inception-ResNetV2) have been proposed for the detection of coronavirus pneumonia infected patient using chest X-ray radiographs. We have implemented three different binary classifications with four classes (COVID-19, normal (healthy), viral pneumonia and bacterial pneumonia) by using 5-fold cross validation. Considering the performance results obtained, it has seen that the pre-trained ResNet50 model provides the highest classification performance (96.1% accuracy for Dataset-1, 99.5% accuracy for Dataset-2 and 99.7% accuracy for Dataset-3) among other four used models.

CoroNet: A deep neural network for detection and diagnosis of COVID-19 from chest x-ray images

Abstract: Background and Objective The novel Coronavirus also called COVID-19 originated in Wuhan, China in December 2019 and has now spread across the world. It has so far infected around 1.8 million people and claimed approximately 114,698 lives overall. As the number of cases are rapidly increasing, most of the countries are facing shortage of testing kits and resources. The limited quantity of testing kits and increasing number of daily cases encouraged us to come up with a Deep Learning model that can aid radiologists and clinicians in detecting COVID-19 cases using chest X-rays. Methods In this study, we propose CoroNet, a Deep Convolutional Neural Network model to automatically detect COVID-19 infection from chest X-ray images. The proposed model is based on Xception architecture pre-trained on ImageNet dataset and trained end-to-end on a dataset prepared by collecting COVID-19 and other chest pneumonia X-ray images from two different publically available databases. Results and Conclusion CoroNet has been trained and tested on the prepared dataset and the experimental results show that our proposed model achieved an overall accuracy of 89.6%, and more importantly the precision and recall rate for COVID-19 cases are 93% and 98.2% for 4-class cases (COVID vs Pneumonia bacterial vs pneumonia viral vs normal). For 3-class classification (COVID vs Pneumonia vs normal), the proposed model produced a classification accuracy of 95%. The preliminary results of this study look promising which can be further improved as more training data becomes available. Overall, the proposed model substantially advances the current radiology based methodology and during COVID-19 pandemic, it can be very helpful tool for clinical practitioners and radiologists to aid them in diagnosis, quantification and follow-up of COVID-19 cases.

Within the Lack of Chest COVID-19 X-ray Dataset: A Novel Detection Model Based on GAN and Deep Transfer Learning

Abstract: The coronavirus (COVID-19) pandemic is putting healthcare systems across the world under unprecedented and increasing pressure according to the World Health Organization (WHO). With the advances in computer algorithms and especially Artificial Intelligence, the detection of this type of virus in the early stages will help in fast recovery and help in releasing the pressure off healthcare systems. In this paper, a GAN with deep transfer learning for coronavirus detection in chest X-ray images is presented. The lack of datasets for COVID-19 especially in chest X-rays images is the main motivation of this scientific study. The main idea is to collect all the possible images for COVID-19 that exists until the writing of this research and use the GAN network to generate more images to help in the detection of this virus from the available X-rays images with the highest accuracy possible. The dataset used in this research was collected from different sources and it is available for researchers to download and use it. The number of images in the collected dataset is 307 images for four different types of classes. The classes are the COVID-19, normal, pneumonia bacterial, and pneumonia virus. Three deep transfer models are selected in this research for investigation. The models are the Alexnet, Googlenet, and Restnet18. Those models are selected for investigation through this research as it contains a small number of layers on their architectures, this will result in reducing the complexity, the consumed memory and the execution time for the proposed model. Three case scenarios are tested through the paper, the first scenario includes four classes from the dataset, while the second scenario includes 3 classes and the third scenario includes two classes. All the scenarios include the COVID-19 class as it is the main target of this research to be detected. In the first scenario, the Googlenet is selected to be the main deep transfer model as it achieves 80.6% in testing accuracy. In the second scenario, the Alexnet is selected to be the main deep transfer model as it achieves 85.2% in testing accuracy, while in the third scenario which includes two classes (COVID-19, and normal), Googlenet is selected to be the main deep transfer model as it achieves 100% in testing accuracy and 99.9% in the validation accuracy. All the performance measurement strengthens the obtained results through the research.

COVIDX-Net: A Framework of Deep Learning Classifiers to Diagnose COVID-19 in X-Ray Images

Abstract: Background and Purpose: Coronaviruses (CoV) are perilous viruses that may cause Severe Acute Respiratory Syndrome (SARS-CoV), Middle East Respiratory Syndrome (MERS-CoV). The novel 2019 Coronavirus disease (COVID-19) was discovered as a novel disease pneumonia in the city of Wuhan, China at the end of 2019. Now, it becomes a Coronavirus outbreak around the world, the number of infected people and deaths are increasing rapidly every day according to the updated reports of the World Health Organization (WHO). Therefore, the aim of this article is to introduce a new deep learning framework; namely COVIDX-Net to assist radiologists to automatically diagnose COVID-19 in X-ray images. Materials and Methods: Due to the lack of public COVID-19 datasets, the study is validated on 50 Chest X-ray images with 25 confirmed positive COVID-19 cases. The COVIDX-Net includes seven different architectures of deep convolutional neural network models, such as modified Visual Geometry Group Network (VGG19) and the second version of Google MobileNet. Each deep neural network model is able to analyze the normalized intensities of the X-ray image to classify the patient status either negative or positive COVID-19 case. Results: Experiments and evaluation of the COVIDX-Net have been successfully done based on 80-20% of X-ray images for the model training and testing phases, respectively. The VGG19 and Dense Convolutional Network (DenseNet) models showed a good and similar performance of automated COVID-19 classification with f1-scores of 0.89 and 0.91 for normal and COVID-19, respectively. Conclusions: This study demonstrated the useful application of deep learning models to classify COVID-19 in X-ray images based on the proposed COVIDX-Net framework. Clinical studies are the next milestone of this research work.

Estimating Uncertainty and Interpretability in Deep Learning for Coronavirus (COVID-19) Detection

Abstract: Deep Learning has achieved state of the art performance in medical imaging. However, these methods for disease detection focus exclusively on improving the accuracy of classification or predictions without quantifying uncertainty in a decision. Knowing how much confidence there is in a computer-based medical diagnosis is essential for gaining clinicians trust in the technology and therefore improve treatment. Today, the 2019 Coronavirus (SARS-CoV-2) infections are a major healthcare challenge around the world. Detecting COVID-19 in X-ray images is crucial for diagnosis, assessment and treatment. However, diagnostic uncertainty in the report is a challenging and yet inevitable task for radiologist. In this paper, we investigate how drop-weights based Bayesian Convolutional Neural Networks (BCNN) can estimate uncertainty in Deep Learning solution to improve the diagnostic performance of the human-machine team using publicly available COVID-19 chest X-ray dataset and show that the uncertainty in prediction is highly correlates with accuracy of prediction. We believe that the availability of uncertainty-aware deep learning solution will enable a wider adoption of Artificial Intelligence (AI) in a clinical setting.

Artificial Intelligence Forecasting of Covid-19 in China

Abstract: BACKGROUND An alternative to epidemiological models for transmission dynamics of Covid-19 in China, we propose the artificial intelligence (AI)-inspired methods for real-time forecasting of Covid-19 to estimate the size, lengths and ending time of Covid-19 across China. METHODS We developed a modified stacked auto-encoder for modeling the transmission dynamics of the epidemics. We applied this model to real-time forecasting the confirmed cases of Covid-19 across China. The data were collected from January 11 to February 27, 2020 by WHO. We used the latent variables in the auto-encoder and clustering algorithms to group the provinces/cities for investigating the transmission structure. RESULTS We forecasted curves of cumulative confirmed cases of Covid-19 across China from Jan 20, 2020 to April 20, 2020. Using the multiple-step forecasting, the estimated average errors of 6-step, 7-step, 8-step, 9-step and 10-step forecasting were 1.64%, 2.27%, 2.14%, 2.08%, 0.73%, respectively. We predicted that the time points of the provinces/cities entering the plateau of the forecasted transmission dynamic curves varied, ranging from Jan 21 to April 19, 2020. The 34 provinces/cities were grouped into 9 clusters. CONCLUSIONS The accuracy of the AI-based methods for forecasting the trajectory of Covid-19 was high. We predicted that the epidemics of Covid-19 will be over by the middle of April. If the data are reliable and there are no second transmissions, we can accurately forecast the transmission dynamics of the Covid-19 across the provinces/cities in China. The AI-inspired methods are a powerful tool for helping public health planning and policymaking.

Investigating a Serious Challenge in the Sustainable Development Process: Analysis of Confirmed cases of COVID-19 (New Type of Coronavirus) Through a Binary Classification Using Artificial Intelligence and Regression Analysis

Abstract: Nowadays, sustainable development is considered a key concept and solution in creating a promising and prosperous future for human societies. Nevertheless, there are some predicted and unpredicted problems that epidemic diseases are real and complex problems. Hence, in this research work, a serious challenge in the sustainable development process was investigated using the classification of confirmed cases of COVID-19 (new version of Coronavirus) as one of the epidemic diseases. Hence, binary classification modeling was used by the group method of data handling (GMDH) type of neural network as one of the artificial intelligence methods. For this purpose, the Hubei province in China was selected as a case study to construct the proposed model, and some important factors, namely maximum, minimum, and average daily temperature, the density of a city, relative humidity, and wind speed, were considered as the input dataset, and the number of confirmed cases was selected as the output dataset for 30 days. The proposed binary classification model provides higher performance capacity in predicting the confirmed cases. In addition, regression analysis has been done and the trend of confirmed cases compared with the fluctuations of daily weather parameters (wind, humidity, and average temperature). The results demonstrated that the relative humidity and maximum daily temperature had the highest impact on the confirmed cases. The relative humidity in the main case study, with an average of 77.9%, affected positively, and maximum daily temperature, with an average of 15.4 °C, affected negatively, the confirmed cases.

A summary of the literature based on their purposes

We synthesise the existing research in terms of their purposes and aims to explore the contribution of AI to fight against the COVID-19 pandemic. A summary of the synthesized data is presented in Table {{table:b3056ba3-0308-4ca2-83ec-fac234bbfc82}}  to show the research scopes and purposes of the original research. Most of the articles (n =16, 48%) were published focusing to detect the COVID-19 infected patients using different AI-based algorithms that include, for example, the Convoluted Neural Network (CNN) model, Support Vector Machine (SVM), generative adversarial network (GAN), and the transfer learning. The chest X-ray images, CT images, mobile sensors data, and COVID-19 symptoms were used to predict/detect the COVID-19 patients. These researches aimed to identify, screen and detect the COVID-19 patients; and also to predict, differentiate, or classify the patients into COVID-19 infection, no infection, and other viral or bacterial infection. For example, Wang et al. {{cite:dc38c15}} proposed a CNN based prediction system named COVID-Net that identifies non-COVID-19 infected, COVID-19 infected, and no infected patients using chest X-ray images. The proposed model was pre-trained on ImageNet (open source) dataset and then trained on the COVIDx (author- created) dataset that includes 13,800 chest X-ray images of 13,725 patients that includes 183 images from 121 COVID-19 positive patients, 8066 images are of healthy patients and 5538 images are of non-COVID19 patients.

The other seven articles (20%) used non-image data, predominantly in the form of text and numbers with the purpose of disease detection, epidemic forecasting, sustainable development, introducing advanced concepts, and disease diagnosis and progression (Table {{table:b3056ba3-0308-4ca2-83ec-fac234bbfc82}} ). Thousands of data points were collected through a mobile survey for a study {{cite:3fe6c15}} that included information related to location, age, gender, race, travel and close contact to any affected person. Furthermore, the study collected health data related to COVID-19 symptoms during a period of 14 days. Clinical data including information on baseline characteristics, medical history, COVID-19 diagnosis from hospitals in Wuhan and other provinces in China were collected for disease prediction and progression purposes. The study {{cite:e4aaa0a}} used environmental and urban data accumulated from 42 different provinces in China, Japan, South Korea, Italy to analyze the correlation among environmental factors (low, high and average temperature, humidity, wind flow) and confirmed cases of COVID-19. The study here {{cite:126fd53}} used migration data and 2003 SARS epidemic for epidemic forecasting of COVID-19. Finally, the study{{cite:9c59be1}} used various research papers and scholarly articles for the purpose of proposing potential ideas to combat COVID-19 using AI.

Most of the research papers (n=29, 83%) aim to use AI to do some kind of classification (COVID-19 detection, differentiate COVID-19 from other respiratory diseases) forecasting, and prediction (Table {{table:b3056ba3-0308-4ca2-83ec-fac234bbfc82}} ). The cost and time associated with the gold standard of testing COVID-19: PCR - takes up to two to three days to get the results, drive researchers to find an easier, cheaper and faster way to detect COVID-19 using computational technique. Therefore, our study found most of the research work (65%) aimed to detect and diagnose COVID-19. Table {{table:7a7c2f4a-e0dc-435d-ad87-6702c99d392c}}  briefly presents the objective, scope and the results of using different AI algorithms.


The Multilingual Amazon Reviews Corpus

Abstract: We present the Multilingual Amazon Reviews Corpus (MARC), a large-scale collection of Amazon reviews for multilingual text classification. The corpus contains reviews in English, Japanese, German, French, Spanish, and Chinese, which were collected between 2015 and 2019. Each record in the dataset contains the review text, the review title, the star rating, an anonymized reviewer ID, an anonymized product ID, and the coarse-grained product category (e.g., 'books', 'appliances', etc.) The corpus is balanced across the 5 possible star ratings, so each rating constitutes 20% of the reviews in each language. For each language, there are 200,000, 5,000, and 5,000 reviews in the training, development, and test sets, respectively. We report baseline results for supervised text classification and zero-shot cross-lingual transfer learning by fine-tuning a multilingual BERT model on reviews data. We propose the use of mean absolute error (MAE) instead of classification accuracy for this task, since MAE accounts for the ordinal nature of the ratings.

SemEval-2017 Task 4: Sentiment Analysis in Twitter

Abstract: This paper describes the fifth year of the Sentiment Analysis in Twitter task. SemEval-2017 Task 4 continues with a rerun of the subtasks of SemEval-2016 Task 4, which include identifying the overall sentiment of the tweet, sentiment towards a topic with classification on a two-point and on a five-point ordinal scale, and quantification of the distribution of sentiment towards a topic across a number of tweets: again on a two-point and on a five-point ordinal scale. Compared to 2016, we made two changes: (i) we introduced a new language, Arabic, for all subtasks, and (ii) we made available information from the profiles of the Twitter users who posted the target tweets. The task continues to be very popular, with a total of 48 teams participating this year.

SemEval-2020 Task 9: Overview of Sentiment Analysis of Code-Mixed Tweets

Abstract: In this paper, we present the results of the SemEval-2020 Task 9 on Sentiment Analysis of Code-Mixed Tweets (SentiMix 2020). We also release and describe our Hinglish (Hindi-English)and Spanglish (Spanish-English) corpora annotated with word-level language identification and sentence-level sentiment labels. These corpora are comprised of 20K and 19K examples, respectively. The sentiment labels are - Positive, Negative, and Neutral. SentiMix attracted 89 submissions in total including 61 teams that participated in the Hinglish contest and 28 submitted systems to the Spanglish competition. The best performance achieved was 75.0% F1 score for Hinglish and 80.6% F1 for Spanglish. We observe that BERT-like models and ensemble methods are the most common and successful approaches among the participants.

Multilingual Twitter Sentiment Classification: The Role of Human Annotators

Abstract: What are the limits of automated Twitter sentiment classification? We analyze a large set of manually labeled tweets in different languages, use them as training data, and construct automated classification models. It turns out that the quality of classification models depends much more on the quality and size of training data than on the type of the model trained. Experimental results indicate that there is no statistically significant difference between the performance of the top classification models. We quantify the quality of training data by applying various annotator agreement measures, and identify the weakest points of different datasets. We show that the model performance approaches the inter-annotator agreement when the size of the training set is sufficiently large. However, it is crucial to regularly monitor the self- and inter-annotator agreements since this improves the training datasets and consequently the model performance. Finally, we show that there is strong evidence that humans perceive the sentiment classes (negative, neutral, and positive) as ordered.

Assessment of Massively Multilingual Sentiment Classifiers

Abstract: Models are increasing in size and complexity in the hunt for SOTA. But what if those 2\% increase in performance does not make a difference in a production use case? Maybe benefits from a smaller, faster model outweigh those slight performance gains. Also, equally good performance across languages in multilingual tasks is more important than SOTA results on a single one. We present the biggest, unified, multilingual collection of sentiment analysis datasets. We use these to assess 11 models and 80 high-quality sentiment datasets (out of 342 raw datasets collected) in 27 languages and included results on the internally annotated datasets. We deeply evaluate multiple setups, including fine-tuning transformer-based models for measuring performance. We compare results in numerous dimensions addressing the imbalance in both languages coverage and dataset sizes. Finally, we present some best practices for working with such a massive collection of datasets and models from a multilingual perspective.

List of all multilingual datasets used in experiments. Category (Cat.): R - Reviews, SM - Social Media, C - Chats, N - News, P - Poems, M - Mixed. HL - human labeled

Presenting statistics of 80 datasets across 27 languages could be challenging. We checked different aggregating and sorting of datasets to make their statistics as readable as possible and easily usable for results discussion. We decided to group datasets by their language and next sorted them based on the number of examples in every aggregate - Table {{table:bd8bb77a-dfba-4c0e-893e-12004513ec5d}} .
In total, we selected 80 datasets containing 6,164,942 text samples. Most of the texts in the datasets are in English (2,330,486 samples across 17 datasets), Arabic (932,075 samples across 9 datasets), and Spanish (418,892 samples across 5 datasets). The datasets contain text from various categories: social media (44 datasets), reviews (24 datasets), news (5 datasets), and others (7 datasets). They also differ in the mean number of words and characters in examples. See the detailed information of datasets used is in Tables {{table:9d4d3a8d-bb25-4c95-b20b-587f59b33751}}  and {{table:a70d891c-f2ab-45bc-abb9-2b09803a1435}} .

We present detailed lists of datasets included in our research in Tables {{table:9d4d3a8d-bb25-4c95-b20b-587f59b33751}}  and {{table:a70d891c-f2ab-45bc-abb9-2b09803a1435}} . They include language, category, dataset size, class balance and basic texts characteristics.


The iNaturalist Challenge 2017 Dataset

Abstract:
Existing image classification datasets used in computer vision tend to have an even number of images for each object category. In contrast, the natural world is heavily imbalanced, as some species are more abundant and easier to photograph than others. To encourage further progress in challenging real world conditions we present the iNaturalist Challenge 2017 dataset - an image classification benchmark consisting of 675,000 images with over 5,000 different species of plants and animals. It features many visually similar species, captured in a wide variety of situations, from all over the world. Images were collected with different camera types, have varying image quality, have been verified by multiple citizen scientists, and feature a large class imbalance.
We discuss the collection of the dataset and present baseline results for state-of-the-art computer vision classification models. Results show that current non-ensemble based methods achieve only 64% top one classification accuracy, illustrating the difficulty of the dataset. Finally, we report results from a competition that was held with the data.

The iNaturalist Challenge 2017 Dataset

Large-Scale Image Retrieval with Attentive Deep Local Features

Abstract: We propose an attentive local feature descriptor suitable for large-scale image retrieval, referred to as DELE (DEep Local Feature). The new feature is based on convolutional neural networks, which are trained only with image-level annotations on a landmark image dataset. To identify semantically useful local features for image retrieval, we also propose an attention mechanism for key point selection, which shares most network layers with the descriptor. This frame-work can be used for image retrieval as a drop-in replacement for other keypoint detectors and descriptors, enabling more accurate feature matching and geometric verification. Our system produces reliable confidence scores to reject false positives–in particular, it is robust against queries that have no correct match in the database. To evaluate the proposed descriptor, we introduce a new large-scale dataset, referred to as Google-Landmarks dataset, which involves challenges in both database and query such as background clutter, partial occlusion, multiple landmarks, objects in variable scales, etc. We show that DELE outperforms the state-of-the-art global and local descriptors in the large-scale setting by significant margins.

The Semi-ViT-Base results on other datasets.

The generalization of Semi-ViT is evaluated on datasets including Food-101 {{cite:97e902f}}, iNaturalist {{cite:5f2b5b0}} and GoogleLandmark {{cite:96b3377}}. Since these datasets are beyond ImageNet, we assume that the ImageNet dataset is available and the model is already supervised pretrained on ImageNet, and then the model is finetuned to different target datasets with a few labels. The results are shown in Table {{table:95741dda-d1cc-4806-a4e3-cce8ee67d183}} . On these dataset, our Semi-ViT can improve over the fine-tuning baselines by 13-21 (7-10) points on 1% (10%) labels. Note that on Food-101, Semi-ViT on 1% (10%) labels is close to fine-tuning baseline on 10% (100%) labels, i.e., 82.1 v.s. 84.5 (91.3 v.s. 93.1), indicating that using Semi-ViT can help to save annotation costs by about 10 times on this dataset.