Simple and Deep Graph Convolutional Networks

Abstract: Graph convolutional networks (GCNs) are a powerful deep learning approach for graph-structured data. Recently, GCNs and subsequent variants have shown superior performance in various application areas on real-world datasets. Despite their success, most of the current GCN models are shallow, due to the {\em over-smoothing} problem. In this paper, we study the problem of designing and analyzing deep graph convolutional networks. We propose the GCNII, an extension of the vanilla GCN model with two simple yet effective techniques: {\em Initial residual} and {\em Identity mapping}. We provide theoretical and empirical evidence that the two techniques effectively relieves the problem of over-smoothing. Our experiments show that the deep GCNII model outperforms the state-of-the-art methods on various semi- and full-supervised tasks. Code is available at this https URL .

Predict then Propagate: Graph Neural Networks meet Personalized PageRank

Abstract: Neural message passing algorithms for semi-supervised classification on graphs have recently achieved great success. However, for classifying a node these methods only consider nodes that are a few propagation steps away and the size of this utilized neighborhood is hard to extend. In this paper, we use the relationship between graph convolutional networks (GCN) and PageRank to derive an improved propagation scheme based on personalized PageRank. We utilize this propagation procedure to construct a simple model, personalized propagation of neural predictions (PPNP), and its fast approximation, APPNP. Our model's training time is on par or faster and its number of parameters on par or lower than previous models. It leverages a large, adjustable neighborhood for classification and can be easily combined with any neural network. We show that this model outperforms several recently proposed methods for semi-supervised classification in the most thorough study done so far for GCN-like models. Our implementation is available online.

Revisiting Graph Convolutional Network on Semi-Supervised Node Classification from an Optimization Perspective

Abstract: Graph convolutional networks (GCNs) have achieved promising performance on various graph-based tasks. However they suffer from over-smoothing when stacking more layers. In this paper, we present a quantitative study on this observation and develop novel insights towards the deeper GCN. First, we interpret the current graph convolutional operations from an optimization perspective and argue that over-smoothing is mainly caused by the naive first-order approximation of the solution to the optimization problem. Subsequently, we introduce two metrics to measure the over-smoothing on node-level tasks. Specifically, we calculate the fraction of the pairwise distance between connected and disconnected nodes to the overall distance respectively. Based on our theoretical and empirical analysis, we establish a universal theoretical framework of GCN from an optimization perspective and derive a novel convolutional kernel named GCN+ which has lower parameter amount while relieving the over-smoothing inherently. Extensive experiments on real-world datasets demonstrate the superior performance of GCN+ over state-of-the-art baseline methods on the node classification tasks.

Break the Ceiling: Stronger Multi-scale Deep Graph Convolutional Networks

Abstract: Recently, neural network based approaches have achieved significant improvement for solving large, complex, graph-structured problems. However, their bottlenecks still need to be addressed, and the advantages of multi-scale information and deep architectures have not been sufficiently exploited. In this paper, we theoretically analyze how existing Graph Convolutional Networks (GCNs) have limited expressive power due to the constraint of the activation functions and their architectures. We generalize spectral graph convolution and deep GCN in block Krylov subspace forms and devise two architectures, both with the potential to be scaled deeper but each making use of the multi-scale information in different ways. We further show that the equivalence of these two architectures can be established under certain conditions. On several node classification tasks, with or without the help of validation, the two new architectures achieve better performance compared to many state-of-the-art methods.

Towards Deeper Graph Neural Networks

Abstract: Graph neural networks have shown significant success in the field of graph representation learning. Graph convolutions perform neighborhood aggregation and represent one of the most important graph operations. Nevertheless, one layer of these neighborhood aggregation methods only consider immediate neighbors, and the performance decreases when going deeper to enable larger receptive fields. Several recent studies attribute this performance deterioration to the over-smoothing issue, which states that repeated propagation makes node representations of different classes indistinguishable. In this work, we study this observation systematically and develop new insights towards deeper graph neural networks. First, we provide a systematical analysis on this issue and argue that the key factor compromising the performance significantly is the entanglement of representation transformation and propagation in current graph convolution operations. After decoupling these two operations, deeper graph neural networks can be used to learn graph node representations from larger receptive fields. We further provide a theoretical analysis of the above observation when building very deep models, which can serve as a rigorous and gentle description of the over-smoothing issue. Based on our theoretical and empirical analysis, we propose Deep Adaptive Graph Neural Network (DAGNN) to adaptively incorporate information from large receptive fields. A set of experiments on citation, co-authorship, and co-purchase datasets have confirmed our analysis and insights and demonstrated the superiority of our proposed methods.

PairNorm: Tackling Oversmoothing in GNNs

Abstract: The performance of graph neural nets (GNNs) is known to gradually decrease with increasing number of layers. This decay is partly attributed to oversmoothing, where repeated graph convolutions eventually make node embeddings indistinguishable. We take a closer look at two different interpretations, aiming to quantify oversmoothing. Our main contribution is PairNorm, a novel normalization layer that is based on a careful analysis of the graph convolution operator, which prevents all node embeddings from becoming too similar. What is more, PairNorm is fast, easy to implement without any change to network architecture nor any additional parameters, and is broadly applicable to any GNN. Experiments on real-world graphs demonstrate that PairNorm makes deeper GCN, GAT, and SGC models more robust against oversmoothing, and significantly boosts performance for a new problem setting that benefits from deeper GNNs. Code is available at this https URL.

Towards Deeper Graph Neural Networks with Differentiable Group Normalization

Abstract: Graph neural networks (GNNs), which learn the representation of a node by aggregating its neighbors, have become an effective computational tool in downstream applications. Over-smoothing is one of the key issues which limit the performance of GNNs as the number of layers increases. It is because the stacked aggregators would make node representations converge to indistinguishable vectors. Several attempts have been made to tackle the issue by bringing linked node pairs close and unlinked pairs distinct. However, they often ignore the intrinsic community structures and would result in sub-optimal performance. The representations of nodes within the same community/class need be similar to facilitate the classification, while different classes are expected to be separated in embedding space. To bridge the gap, we introduce two over-smoothing metrics and a novel technique, i.e., differentiable group normalization (DGN). It normalizes nodes within the same group independently to increase their smoothness, and separates node distributions among different groups to significantly alleviate the over-smoothing issue. Experiments on real-world datasets demonstrate that DGN makes GNN models more robust to over-smoothing and achieves better performance with deeper GNNs.

DropEdge: Towards Deep Graph Convolutional Networks on Node Classification

Abstract: Over-fitting and over-smoothing are two main obstacles of developing deep Graph Convolutional Networks (GCNs) for node classification. In particular, over-fitting weakens the generalization ability on small dataset, while over-smoothing impedes model training by isolating output representations from the input features with the increase in network depth. This paper proposes DropEdge, a novel and flexible technique to alleviate both issues. At its core, DropEdge randomly removes a certain number of edges from the input graph at each training epoch, acting like a data augmenter and also a message passing reducer. Furthermore, we theoretically demonstrate that DropEdge either retards the convergence speed of over-smoothing or relieves the information loss caused by it. More importantly, our DropEdge is a general skill that can be equipped with many other backbone models (e.g. GCN, ResGCN, GraphSAGE, and JKNet) for enhanced performance. Extensive experiments on several benchmarks verify that DropEdge consistently improves the performance on a variety of both shallow and deep GCNs. The effect of DropEdge on preventing over-smoothing is empirically visualized and validated as well. Codes will be made public upon the publication.

Layer-Dependent Importance Sampling for Training Deep and Large Graph Convolutional Networks

Abstract: Graph convolutional networks (GCNs) have recently received wide attentions, due to their successful applications in different graph tasks and different domains. Training GCNs for a large graph, however, is still a challenge. Original full-batch GCN training requires calculating the representation of all the nodes in the graph per GCN layer, which brings in high computation and memory costs. To alleviate this issue, several sampling-based methods are proposed to train GCNs on a subset of nodes. Among them, the node-wise neighbor-sampling method recursively samples a fixed number of neighbor nodes, and thus its computation cost suffers from exponential growing neighbor size across layers; while the layer-wise importance-sampling method discards the neighbor-dependent constraints, and thus the nodes sampled across layer suffer from sparse connection problem. To deal with the above two problems, we propose a new effective sampling algorithm called LAyer-Dependent ImportancE Sampling (LADIES). Based on the sampled nodes in the upper layer, LADIES selects nodes that are in the neighborhood of these nodes and uses the constructed bipartite graph to compute the importance probability. Then, it samples a fixed number of nodes according to the probability for the whole layer, and recursively conducts such procedure per layer to construct the whole computation graph. We prove theoretically and experimentally, that our proposed sampling algorithm outperforms the previous sampling methods regarding both time and memory. Furthermore, LADIES is shown to have better generalization accuracy than original full-batch GCN, due to its stochastic nature.

Bayesian Graph Neural Networks with Adaptive Connection Sampling

Abstract: We propose a unified framework for adaptive connection sampling in graph neural networks (GNNs) that generalizes existing stochastic regularization methods for training GNNs. The proposed framework not only alleviates over-smoothing and over-fitting tendencies of deep GNNs, but also enables learning with uncertainty in graph analytic tasks with GNNs. Instead of using fixed sampling rates or hand-tuning them as model hyperparameters in existing stochastic regularization methods, our adaptive connection sampling can be trained jointly with GNN model parameters in both global and local fashions. GNN training with adaptive connection sampling is shown to be mathematically equivalent to an efficient approximation of training Bayesian GNNs. Experimental results with ablation studies on benchmark datasets validate that adaptively learning the sampling rate given graph training data is the key to boost the performance of GNNs in semi-supervised node classification, less prone to over-smoothing and over-fitting with more robust prediction.

Configurations adopted to implement different approaches for training deep GNNs on PubMed {{cite:bcb460c}}.

Aiming to establish such a fair benchmark, our first step is to thoroughly investigate the design philosophy and implementation details on dozens of poplular deep GNN training techniques, including various residual connections, graph normalization, and random dropping. The summarization could be founded in Tables {{table:6f8400d1-d594-4f02-88f9-40ca9100b320}} , {{table:2a41a82d-5a98-4b73-8d77-82f36359d0ed}} , {{table:0ea2e031-28aa-4471-bf6a-197f3d6fa2fc}} , and {{table:4acce5ed-493d-43b0-b729-89a55f16605e}} . Somehow unfortunately, we find that even sticking to the same dataset and GNN backbone, the hyperparameter configurations (e.g., hidden dimension, learning rate, weight decay, dropout rate, training epochs, early stopping patience) are highly inconsistently implemented, often varying case-to-case, which make it troubling to draw any fair conclusion.

Prerequisite: Unifying the Hyperparameter Configuration

We carefully examine previous implementations of deep GNNs {{cite:2363f96}}, {{cite:dee7a72}}, {{cite:0812184}}, {{cite:f7be3b7}}, {{cite:f2ad407}}, {{cite:a43ce6a}}, {{cite:cae2619}}, {{cite:82e037d}}, {{cite:35b159a}}, {{cite:b088268}}, and list all their basic hyperparameters in Table {{table:6f8400d1-d594-4f02-88f9-40ca9100b320}} , {{table:2a41a82d-5a98-4b73-8d77-82f36359d0ed}} , {{table:0ea2e031-28aa-4471-bf6a-197f3d6fa2fc}} , and {{table:4acce5ed-493d-43b0-b729-89a55f16605e}} . Those hyperparameters play a significant role in those methods' achievable performance, but their inconsistency challenges fair comparison of training techniques, which has been traditionally somehow overlooked in literature.

Diverse hyperparameter configurations of existing deep GNNs.

As shown in Table {{table:2a41a82d-5a98-4b73-8d77-82f36359d0ed}} , {{table:0ea2e031-28aa-4471-bf6a-197f3d6fa2fc}} , and {{table:4acce5ed-493d-43b0-b729-89a55f16605e}} , we can see that these highly inconsistent hyperparameter settings pose severely challenges to fairly compare the existing training tricks for deep GNNs.


Based on the annotation scheme, sentences labelled into 7 categories then grouped into 2 categories

A Benchmark Dataset of Check-worthy Factual Claims

Abstract: In this paper we present the ClaimBuster dataset of 23,533 statements extracted from all U.S. general election presidential debates and annotated by human coders. The ClaimBuster dataset can be leveraged in building computational methods to identify claims that are worth fact-checking from the myriad of sources of digital or traditional media. The ClaimBuster dataset is publicly available to the research community, and it can be found at http://doi.org/10.5281/zenodo.3609356.

Towards Automated Factchecking: Developing an Annotation Schema and Benchmark for Consistent Automated Claim Detection

Abstract: In an effort to assist factcheckers in the process of factchecking, we tackle the claim detection task, one of the necessary stages prior to determining the veracity of a claim. It consists of identifying the set of sentences, out of a long text, deemed capable of being factchecked. This paper is a collaborative work between Full Fact, an independent factchecking charity, and academic partners. Leveraging the expertise of professional factcheckers, we develop an annotation schema and a benchmark for automated claim detection that is more consistent across time, topics and annotators than previous approaches. Our annotation schema has been used to crowdsource the annotation of a dataset with sentences from UK political TV shows. We introduce an approach based on universal sentence representations to perform the classification, achieving an F1 score of 0.83, with over 5% relative improvement over the state-of-the-art methods ClaimBuster and ClaimRank. The system was deployed in production and received positive user feedback.

Overview of the CLEF-2018 CheckThat! Lab on Automatic Identification and Verification of Political Claims. Task 1: Check-Worthiness

Abstract: We present an overview of the CLEF-2018 CheckThat! Lab on Automatic Identification and Verification of Political Claims, with focus on Task 1: Check-Worthiness. The task asks to predict which claims in a political debate should be prioritized for fact-checking. In particular, given a debate or a political speech, the goal was to produce a ranked list of its sentences based on their worthiness for fact checking. We offered the task in both English and Arabic, based on debates from the 2016 US Presidential Campaign, as well as on some speeches during and after the campaign. A total of 30 teams registered to participate in the Lab and seven teams actually submitted systems for Task~1. The most successful approaches used by the participants relied on recurrent and multi-layer neural networks, as well as on combinations of distributional representations, on matchings claims' vocabulary against lexicons, and on measures of syntactic dependency. The best systems achieved mean average precision of 0.18 and 0.15 on the English and on the Arabic test datasets, respectively. This leaves large room for further improvement, and thus we release all datasets and the scoring scripts, which should enable further research in check-worthiness estimation.

Check-worthiness claim detection Datasets

In recent studies, several datasets were built with the purpose of enabling training machine learning models to predict check-worthy claims, as shown in Table {{table:d07c5eb5-48ab-4ad7-b45b-8fbe7c7ce3f4}} . The vast majority of datasets cover sentences pertaining to the political domain, as a result of events that synchronously occur with the US elections. In contrast, the CheckThat! Lab released English and Arabic datasets that contain a small number of instances related to COVID-19 in early 2020.

Labelling of sentences as claims or non-claims is generally done manually by non-experts (see Table {{table:d07c5eb5-48ab-4ad7-b45b-8fbe7c7ce3f4}} ). An alternative to this is to derive labels from previously fact-checked claims collected from fact-checking websites. The main caveat of this approach is that fact-checking websites only list claims, rather than non-claims, which means that one needs to develop models that only leveraged instances of the positive class, i.e. positive unlabelled learning {{cite:c4bca0a}} .


Current status of the DARPA Quantum Network

Abstract: This paper reports the current status of the DARPA Quantum Network, which became fully operational in BBN's laboratory in October 2003, and has been continuously running in 6 nodes operating through telecommunications fiber between Harvard University, Boston University, and BBN since June 2004. The DARPA Quantum Network is the world's first quantum cryptography network, and perhaps also the first QKD systems providing continuous operation across a metropolitan area. Four more nodes are now being added to bring the total to 10 QKD nodes. This network supports a variety of QKD technologies, including phase-modulated lasers through fiber, entanglement through fiber, and freespace QKD. We provide a basic introduction and rational for this network, discuss the February 2005 status of the various QKD hardware suites and software systems in the network, and describe our operational experience with the DARPA Quantum Network to date. We conclude with a discussion of our ongoing work.

A trusted node–free eight-user metropolitan quantum communication network

Abstract: An eight-user fully connected and scalable city-wide quantum communication network can be a basis for the future quantum internet. Quantum communication is rapidly gaining popularity due to its high security and technological maturity. However, most implementations are limited to just two communicating parties (users). Quantum communication networks aim to connect a multitude of users. Here, we present a fully connected quantum communication network on a city-wide scale without active switching or trusted nodes. We demonstrate simultaneous and secure connections between all 28 pairings of eight users. Our novel network topology is easily scalable to many users, allows traffic management features, and minimizes the infrastructure as well as the user hardware needed.

QKD networks properties

Table {{table:13cf9f98-f723-40b8-8d4b-dc5f9846750c}}  summarises the main features of the major QKD networks. Although the first field-trial of a QKD network was the DARPA QKD demonstrated in 2004 {{cite:4977244}}, it did not provide any practical implementation of the QKD system. The SECOQC was a subsequent demonstration of a QKD network aiming at implementing practical applications of the QKD technologies {{cite:0991b78}}, {{cite:961e2fc}}. Moreover, Tokyo QKD network was similar to the SECOQC in terms of infrastructure and was based on point-point links using trusted nodes topology. However, it included the implementation of the first Key Management Server (KMS) for centralised key management {{cite:c13a844}}. The most recent network demonstration is the integrated space-to-ground quantum communication network in China {{cite:b353a12}}. This QKD network consisted of a long-distance fibre backbone network and two satellite–ground links and four quantum metropolitan area network where the backbone link covers over 2000km. A trusted node–free eight-user metropolitan quantum communication using a polarisation-entangled photon source has been implemented in the city of Bristol {{cite:ec36ebd}}.


ConceptNet 5.5: An Open Multilingual Graph of General Knowledge

Abstract: Machine learning about language can be improved by supplying it with specific knowledge and sources of external information. We present here a new version of the linked open data resource ConceptNet that is particularly well suited to be used with modern NLP techniques such as word embeddings. ConceptNet is a knowledge graph that connects words and phrases of natural language with labeled edges. Its knowledge is collected from many sources that include expert-created resources, crowd-sourcing, and games with a purpose. It is designed to represent the general knowledge involved in understanding language, improving natural language applications by allowing the application to better understand the meanings behind the words people use. When ConceptNet is combined with word embeddings acquired from distributional semantics (such as word2vec), it provides applications with understanding that they would not acquire from distributional semantics alone, nor from narrower resources such as WordNet or DBPedia. We demonstrate this with state-of-the-art results on intrinsic evaluations of word relatedness that translate into improvements on applications of word vectors, including solving SAT-style analogies.

CSKG: The CommonSense Knowledge Graph

Abstract: Sources of commonsense knowledge support applications in natural language understanding, computer vision, and knowledge graphs. Given their complementarity, their integration is desired. Yet, their different foci, modeling approaches, and sparse overlap make integration difficult. In this paper, we consolidate commonsense knowledge by following five principles, which we apply to combine seven key sources into a first integrated CommonSense Knowledge Graph (CSKG). We analyze CSKG and its various text and graph embeddings, showing that CSKG is well-connected and that its embeddings provide a useful entry point to the graph. We demonstrate how CSKG can provide evidence for generalizable downstream reasoning and for pre-training of language models. CSKG and all its embeddings are made publicly available to support further research on commonsense knowledge integration and reasoning.

CSKG: The CommonSense Knowledge Graph

Comparison of some well-known commonsense knowledge bases.

Most commonsense knowledge bases are represented in the form of taxonomies, i.e., the knowledge base is usually a collection of individuals grouped into categories, with certain relations between them. A typical example is shown in Figure {{figure:705f7faf-3480-46a3-8c3a-907c0e53b186}} , which illustrates some entities and relations extracted from ConceptNet {{cite:3deb022}}. To our knowledge, most of the existing commonsense knowledge bases have a mixture of factual knowledge and commonsense knowledge. Table {{table:2586d0e7-699d-428e-9ef5-c452cdf8af56}}  gives a comparison of some well-known commonsense knowledge bases. Next, we are going to briefly introduce these knowledge bases one by one and then take a look at how these knowledge bases are evaluated. For each of the knowledge base, we also denote the year when its development began and the year when its newest version was published.


DCFace: Synthetic Face Generation with Dual Condition Diffusion Model

Abstract: Generating synthetic datasets for training face recognition models is challenging because dataset generation entails more than creating high fidelity images. It involves generating multiple images of same subjects under different factors (e.g., variations in pose, illumination, expression, aging and occlusion) which follows the real image conditional distribution. Previous works have studied the generation of synthetic datasets using GAN or 3D models. In this work, we approach the problem from the aspect of combining subject appearance (ID) and external factor (style) conditions. These two conditions provide a direct way to control the inter-class and intra-class variations. To this end, we propose a Dual Condition Face Generator (DCFace) based on a diffusion model. Our novel Patch-wise style extractor and Time-step dependent ID loss enables DCFace to consistently produce face images of the same subject under different styles with precise control. Face recognition models trained on synthetic images from the proposed DCFace provide higher verification accuracies compared to previous works by 6.11% on average in 4 out of 5 test datasets, LFW, CFP-FP, CPLFW, AgeDB and CALFW. Code Link

GANDiffFace: Controllable Generation of Synthetic Datasets for Face Recognition with Realistic Variations

Abstract: Face recognition systems have significantly advanced in recent years, driven by the availability of large-scale datasets. However, several issues have recently came up, including privacy concerns that have led to the discontinuation of well-established public datasets. Synthetic datasets have emerged as a solution, even though current synthesis methods present other drawbacks such as limited intraclass variations, lack of realism, and unfair representation of demographic groups. This study introduces GAN-DiffFace, a novel framework for the generation of synthetic datasets for face recognition that combines the power of Generative Adversarial Networks (GANs) and Diffusion models to overcome the limitations of existing synthetic datasets. In GANDiffFace, we first propose the use of GANs to synthesize highly realistic identities and meet target demographic distributions. Subsequently, we fine-tune Diffusion models with the images generated with GANs, synthesizing multiple images of the same identity with a variety of accessories, poses, expressions, and contexts. We generate multiple synthetic datasets by changing GANDiffFace settings, and compare their mated and non-mated score distributions with the distributions provided by popular real-world datasets for face recognition, i.e. VGG2 and IJB-C. Our results show the feasibility of the proposed GANDiffFace, in particular the use of Diffusion models to enhance the (limited) intra-class variations provided by GANs towards the level of real-world datasets.

Learning Face Representation from Scratch

Abstract: Pushing by big data and deep convolutional neural network (CNN), the performance of face recognition is becoming comparable to human. Using private large scale training datasets, several groups achieve very high performance on LFW, i.e., 97% to 99%. While there are many open source implementations of CNN, none of large scale face dataset is publicly available. The current situation in the field of face recognition is that data is more important than algorithm. To solve this problem, this paper proposes a semi-automatical way to collect face images from Internet and builds a large scale dataset containing about 10,000 subjects and 500,000 images, called CASIAWebFace. Based on the database, we use a 11-layer CNN to learn discriminative representation and obtain state-of-theart accuracy on LFW and YTF. The publication of CASIAWebFace will attract more research groups entering this field and accelerate the development of face recognition in the wild.

A Style-Based Generator Architecture for Generative Adversarial Networks

Abstract: We propose an alternative generator architecture for generative adversarial networks, borrowing from style transfer literature. The new architecture leads to an automatically learned, unsupervised separation of high-level attributes (e.g., pose and identity when trained on human faces) and stochastic variation in the generated images (e.g., freckles, hair), and it enables intuitive, scale-specific control of the synthesis. The new generator improves the state-of-the-art in terms of traditional distribution quality metrics, leads to demonstrably better interpolation properties, and also better disentangles the latent factors of variation. To quantify interpolation quality and disentanglement, we propose two new, automated methods that are applicable to any generator architecture. Finally, we introduce a new, highly varied and high-quality dataset of human faces.

Mitigate Bias in Face Recognition using Skewness-Aware Reinforcement Learning

Abstract: Racial equality is an important theme of international human rights law, but it has been largely obscured when the overall face recognition accuracy is pursued blindly. More facts indicate racial bias indeed degrades the fairness of recognition system and the error rates on non-Caucasians are usually much higher than Caucasians. To encourage fairness, we introduce the idea of adaptive margin to learn balanced performance for different races based on large margin losses. A reinforcement learning based race balance network (RL-RBN) is proposed. We formulate the process of finding the optimal margins for non-Caucasians as a Markov decision process and employ deep Q-learning to learn policies for an agent to select appropriate margin by approximating the Q-value function. Guided by the agent, the skewness of feature scatter between races can be reduced. Besides, we provide two ethnicity aware training datasets, called BUPT-Globalface and BUPT-Balancedface dataset, which can be utilized to study racial bias from both data and algorithm aspects. Extensive experiments on RFW database show that RL-RBN successfully mitigates racial bias and learns more balanced performance for different races.

Details of the databases considered in the FRCSyn Challenge. Id = Identities, Img = Images.

Table {{table:291cb715-2f79-4c53-8a4c-061d0ce9b716}}  provides details of the public databases considered in the FRCSyn Challenge. Participants were instructed to download all necessary databases for the FRCSyn Challenge upon registration. Permission for redistributing these databases was obtained from the owners.


Artificial intelligence and Machine Learning (ML) for UAV communications

UAV characteristics

Communication issues in ML-Enhanced UAV networks

UAV communication security

Mathematical tools for designing UAV communication systems.

Cellular-Connected drones

Flying Ad-hoc Networks with UAVs

Trajectory Optimization

A Survey on Machine-Learning Techniques for UAV-Based Communications

Abstract: Unmanned aerial vehicles (UAVs) will be an integral part of the next generation wireless communication networks. Their adoption in various communication-based applications is expected to improve coverage and spectral efficiency, as compared to traditional ground-based solutions. However, this new degree of freedom that will be included in the network will also add new challenges. In this context, the machine-learning (ML) framework is expected to provide solutions for the various problems that have already been identified when UAVs are used for communication purposes. In this article, we provide a detailed survey of all relevant research works, in which ML techniques have been used on UAV-based communications for improving various design and functional aspects such as channel modeling, resource management, positioning, and security.

A Tutorial on UAVs for Wireless Networks: Applications, Challenges, and Open Problems

Abstract: The use of flying platforms such as unmanned aerial vehicles (UAVs), popularly known as drones, is rapidly growing. In particular, with their inherent attributes such as mobility, flexibility, and adaptive altitude, UAVs admit several key potential applications in wireless systems. On the one hand, UAVs can be used as aerial base stations to enhance coverage, capacity, reliability, and energy efficiency of wireless networks. On the other hand, UAVs can operate as flying mobile terminals within a cellular network. Such cellular-connected UAVs can enable several applications ranging from real-time video streaming to item delivery. In this paper, a comprehensive tutorial on the potential benefits and applications of UAVs in wireless communications is presented. Moreover, the important challenges and the fundamental tradeoffs in UAV-enabled wireless networks are thoroughly investigated. In particular, the key UAV challenges such as 3D deployment, performance analysis, channel modeling, and energy efficiency are explored along with representative results. Then, open problems and potential research directions pertaining to UAV communications are introduced. Finally, various analytical frameworks and mathematical tools, such as optimization theory, machine learning, stochastic geometry, transport theory, and game theory are described. The use of such tools for addressing unique UAV problems is also presented. In a nutshell, this tutorial provides key guidelines on how to analyze, optimize, and design UAV-based wireless communication systems.

Comparison of Existing Survey Articles

blueIn addition to the growing number of new solutions for UAV communication networks in recent years, a number of surveys have been published focusing on UAV communication. These surveys suggested different types of technologies to improve the performance of UAV communication. A summary of these existing survey and tutorial articles is provided in Table {{table:684df6fb-423e-466b-b76d-68ffe06f2a0f}} . The authors of references {{cite:d013ff0}} and {{cite:4d77691}} provided a comprehensive study on the use of UAVs in wireless networks. In addition, two main UAV applications were investigated, namely, UAV-assisted aerial base stations and cellular-connected UAVs. Especially, reference {{cite:4d77691}} presented research based on the cyber-physical security of UAV-assisted cellular communications. In {{cite:e3c9418}}, the authors conducted a comprehensive survey and analysis of air-to-ground channel measurements and channel model for the UAV communication. In addition, they analyzed the link budget for UAV communications, presented the design guideline for managing the link budget, taking into account spread losses and link fading. UAV communication research in the areas of routing, seamless handover and energy efficiency have been discussed in {{cite:8c3cfbf}}. In addition, reference {{cite:9f24aee}} offered a detailed summary of relevant studies, ML-based UAV communication strategies to optimize various model and functional aspects such as UAV channel modeling, resource management, positioning and security.


Monant Medical Misinformation Dataset: Mapping Articles to Fact-Checked Claims

Abstract: False information has a significant negative influence on individuals as well as on the whole society. Especially in the current COVID-19 era, we witness an unprecedented growth of medical misinformation. To help tackle this problem with machine learning approaches, we are publishing a feature-rich dataset of approx. 317k medical news articles/blogs and 3.5k fact-checked claims. It also contains 573 manually and more than 51k automatically labelled mappings between claims and articles. Mappings consist of claim presence, i.e., whether a claim is contained in a given article, and article stance towards the claim. We provide several baselines for these two tasks and evaluate them on the manually labelled part of the dataset. The dataset enables a number of additional tasks related to medical misinformation, such as misinformation characterisation studies or studies of misinformation diffusion between sources.

LESA: Linguistic Encapsulation and Semantic Amalgamation Based Generalised Claim Detection from Online Content

Abstract: The conceptualization of a claim lies at the core of argument mining. The segregation of claims is complex, owing to the divergence in textual syntax and context across different distributions. Another pressing issue is the unavailability of labeled unstructured text for experimentation. In this paper, we propose LESA, a framework which aims at advancing headfirst into expunging the former issue by assembling a source-independent generalized model that captures syntactic features through part-of-speech and dependency embeddings, as well as contextual features through a fine-tuned language model. We resolve the latter issue by annotating a Twitter dataset which aims at providing a testing ground on a large unstructured dataset. Experimental results show that LESA improves upon the state-of-the-art performance across six benchmark claim datasets by an average of 3 claim-F1 points for in-domain experiments and by 2 claim-F1 points for general-domain experiments. On our dataset too, LESA outperforms existing baselines by 1 claim-F1 point on the in-domain experiments and 2 claim-F1 points on the general-domain experiments. We also release comprehensive data annotation guidelines compiled during the annotation phase (which was missing in the current literature).

Is it indeed bigger better? The comprehensive study of claim detection LMs applied for disinformation tackling

Abstract: This study compares the performance of (1) fine-tuned models and (2) extremely large language models on the task of check-worthy claim detection. For the purpose of the comparison we composed a multilingual and multi-topical dataset comprising texts of various sources and styles. Building on this, we performed a benchmark analysis to determine the most general multilingual and multi-topical claim detector. We chose three state-of-the-art models in the check-worthy claim detection task and fine-tuned them. Furthermore, we selected three state-of-the-art extremely large language models without any fine-tuning. We made modifications to the models to adapt them for multilingual settings and through extensive experimentation and evaluation. We assessed the performance of all the models in terms of accuracy, recall, and F1-score in in-domain and cross-domain scenarios. Our results demonstrate that despite the technological progress in the area of natural language processing, the models fine-tuned for the task of check-worthy claim detection still outperform the zero-shot approaches in a cross-domain settings.

Dataset selection and group assignment - seven public datasets and Custom Combined dataset.

We divided the seven publicly available datasets into two groups: the train group, which comprised five datasets used for creating the custom dataset, and the test group, which consisted of two unseen datasets. Additionally, we incorporated our custom dataset into the train group as described in table {{table:ea32867d-c770-44a6-902b-d8209c0d1057}} .


Dataset description for machine learning

Classification of crowdsourced evaluation schemes

Datasheets for datasets

Abstract: Documentation to facilitate communication between dataset creators and consumers.

Towards A Rigorous Science of Interpretable Machine Learning

Abstract: As machine learning systems become ubiquitous, there has been a surge of interest in interpretable machine learning: systems that provide explanation for their outputs. These explanations are often used to qualitatively assess other criteria such as safety or non-discrimination. However, despite the interest in interpretability, there is very little consensus on what interpretable machine learning is and how it should be measured. In this position paper, we first define interpretability and describe when interpretability is needed (and when it is not). Next, we suggest a taxonomy for rigorous evaluation and expose open questions towards a more rigorous science of interpretable machine learning.

Studies on transparency in human computation.

Transparency of human computation systems

To ensure the transparency of human computation systems, in addition to the explanation of AI, profiles and characteristics of data produced by workers is important. Besides, explanation of AI should be understandabe and justified for general users. This section covers studies on explanation of the data and evaluation of interoperability by humans. The related studies are summarized in Table {{table:bf76e11c-23b7-42a2-8b76-d93acf44a7d4}} .


Others (Isolation based feature importance)

Applicable to any AD; Considers both numeric and categorical features

Easy to evaluate the explanation quality

Provides quantitative evaluation of explanation quality

KDE does not work well in high-dimensional subspace

Outlying Property Detection with Numerical Attributes

Abstract: The outlying property detection problem is the problem of discovering the properties distinguishing a given object, known in advance to be an outlier in a database, from the other database objects. In this paper, we analyze the problem within a context where numerical attributes are taken into account, which represents a relevant case left open in the literature. We introduce a measure to quantify the degree the outlierness of an object, which is associated with the relative likelihood of the value, compared to the to the relative likelihood of other objects in the database. As a major contribution, we present an efficient algorithm to compute the outlierness relative to significant subsets of the data. The latter subsets are characterized in a "rule-based" fashion, and hence the basis for the underlying explanation of the outlierness.

Active Anomaly Detection via Ensembles: Insights, Algorithms, and Interpretability

Abstract:
Anomaly detection (AD) task corresponds to identifying the true anomalies from a given set of data instances. AD algorithms score the data instances and produce a ranked list of candidate anomalies, which are then analyzed by a human to discover the true anomalies. However, this process can be laborious for the human analyst when the number of false-positives is very high. Therefore, in many real-world AD applications including computer security and fraud prevention, the anomaly detector must be configurable by the human analyst to minimize the effort on false positives.
In this paper, we study the problem of active learning to automatically tune ensemble of anomaly detectors to maximize the number of true anomalies discovered. We make four main contributions towards this goal. First, we present an important insight that explains the practical successes of AD ensembles and how ensembles are naturally suited for active learning. Second, we present several algorithms for active learning with tree-based AD ensembles. These algorithms help us to improve the diversity of discovered anomalies, generate rule sets for improved interpretability of anomalous instances, and adapt to streaming data settings in a principled manner. Third, we present a novel algorithm called GLocalized Anomaly Detection (GLAD) for active learning with generic AD ensembles. GLAD allows end-users to retain the use of simple and understandable global anomaly detectors by automatically learning their local relevance to specific data instances using label feedback. Fourth, we present extensive experiments to evaluate our insights and algorithms. Our results show that in addition to discovering significantly more anomalies than state-of-the-art unsupervised baselines, our active learning algorithms under the streaming-data setup are competitive with the batch setup.

Rule Extraction in Unsupervised Anomaly Detection for Model Explainability: Application to OneClass SVM

Abstract: OneClass SVM is a popular method for unsupervised anomaly detection. As many other methods, it suffers from the black box problem: it is difficult to justify, in an intuitive and simple manner, why the decision frontier is identifying data points as anomalous or non anomalous. Such type of problem is being widely addressed for supervised models. However, it is still an uncharted area for unsupervised learning. In this paper, we evaluate several rule extraction techniques over OneClass SVM models, as well as present alternative designs for some of those algorithms. Together with that, we propose algorithms to compute metrics related with eXplainable Artificial Intelligence (XAI) regarding the "comprehensibility", "representativeness", "stability" and "diversity" of the extracted rules. We evaluate our proposals with different datasets, including real-world data coming from industry. With this, our proposal contributes to extend XAI techniques to unsupervised machine learning models.

Sequential Feature Explanations for Anomaly Detection

Abstract: In many applications, an anomaly detection system presents the most anomalous data instance to a human analyst, who then must determine whether the instance is truly of interest (e.g., a threat in a security setting). Unfortunately, most anomaly detectors provide no explanation about why an instance was considered anomalous, leaving the analyst with no guidance about where to begin the investigation. To address this issue, we study the problems of computing and evaluating sequential feature explanations (SFEs) for anomaly detectors. An SFE of an anomaly is a sequence of features, which are presented to the analyst one at a time (in order) until the information contained in the highlighted features is enough for the analyst to make a confident judgement about the anomaly. Since analyst effort is related to the amount of information that they consider in an investigation, an explanation’s quality is related to the number of features that must be revealed to attain confidence. In this article, we first formulate the problem of optimizing SFEs for a particular density-based anomaly detector. We then present both greedy algorithms and an optimal algorithm, based on branch-and-bound search, for optimizing SFEs. Finally, we provide a large scale quantitative evaluation of these algorithms using a novel framework for evaluating explanations. The results show that our algorithms are quite effective and that our best greedy algorithm is competitive with optimal solutions.

Why Are You Weird? Infusing Interpretability in Isolation Forest for Anomaly Detection

Abstract: Anomaly detection is concerned with identifying examples in a dataset that do not conform to the expected behaviour. While a vast amount of anomaly detection algorithms exist, little attention has been paid to explaining why these algorithms flag certain examples as anomalies. However, such an explanation could be extremely useful to anyone interpreting the algorithms' output. This paper develops a method to explain the anomaly predictions of the state-of-the-art Isolation Forest anomaly detection algorithm. The method outputs an explanation vector that captures how important each attribute of an example is to identifying it as anomalous. A thorough experimental evaluation on both synthetic and real-world datasets shows that our method is more accurate and more efficient than most contemporary state-of-the-art explainability methods.

Explainable Anomaly Detection Framework for Maritime Main Engine Sensor Data

Abstract: In this study, we proposed a data-driven approach to the condition monitoring of the marine engine. Although several unsupervised methods in the maritime industry have existed, the common limitation was the interpretation of the anomaly; they do not explain why the model classifies specific data instances as an anomaly. This study combines explainable AI techniques with anomaly detection algorithm to overcome the limitation above. As an explainable AI method, this study adopts Shapley Additive exPlanations (SHAP), which is theoretically solid and compatible with any kind of machine learning algorithm. SHAP enables us to measure the marginal contribution of each sensor variable to an anomaly. Thus, one can easily specify which sensor is responsible for the specific anomaly. To illustrate our framework, the actual sensor stream obtained from the cargo vessel collected over 10 months was analyzed. In this analysis, we performed hierarchical clustering analysis with transformed SHAP values to interpret and group common anomaly patterns. We showed that anomaly interpretation and segmentation using SHAP value provides more useful interpretation compared to the case without using SHAP value.

Summary of surveyed shallow post-model techniques. Spec indicates whether a method is model-agnostic (A) or model-specific (S). Pers specifies whether a method is feature-based (F), sample-based (S) or pattern-based (P). Tech indicates the techniques used in each method. Data represents the type of data to which each method can be applied. Moreover, Loc shows whether a method provides a local explanation (L) or global explanation (G). Pros and Cons describe the advantages and disadvantages of each method, respectively.

While Table {{table:8642670a-56b4-4cae-b3b3-db0ffd5f5025}}  provides the full characterization for all methods discussed in this section based on the six criteria of our taxonomy, we here make some general observations on shallow post-model XAD techniques.


SimCLR advocates for a learnable nonlinear transformation bridging the representation and the contrastive loss to enhance the quality of learned representations, and highlights the significance of larger batch sizes, more training steps, and the composition of data augmentations.

MoCo maintains a dynamic queue to enrich the set of negative samples, and proposes a slowly progressing key encoder, implemented as a momentum-based moving average of the query encoder, to preserve the consistency of key representations.

BYOL removes the need of using negative samples, and employs a predictor on top of the online network to learn the mapping from the online encoder to the target encoder, which helps prevent mode collapse.

CPC extracts compact latent representations to encode predictions over the future observations by combining autoregressive modeling and noise-contrastive estimation with intuitions from predictive coding.

Simsiam learns representations by using Siamese architectures without negative sample pairs, large batches and momentum encoders. SimSiam tackles the mode collapse problem by using the stop-gradient mechanism.

A Simple Framework for Contrastive Learning of Visual Representations

Abstract: This paper presents SimCLR: a simple framework for contrastive learning of visual representations. We simplify recently proposed contrastive self-supervised learning algorithms without requiring specialized architectures or a memory bank. In order to understand what enables the contrastive prediction tasks to learn useful representations, we systematically study the major components of our framework. We show that (1) composition of data augmentations plays a critical role in defining effective predictive tasks, (2) introducing a learnable nonlinear transformation between the representation and the contrastive loss substantially improves the quality of the learned representations, and (3) contrastive learning benefits from larger batch sizes and more training steps compared to supervised learning. By combining these findings, we are able to considerably outperform previous methods for self-supervised and semi-supervised learning on ImageNet. A linear classifier trained on self-supervised representations learned by SimCLR achieves 76.5% top-1 accuracy, which is a 7% relative improvement over previous state-of-the-art, matching the performance of a supervised ResNet-50. When fine-tuned on only 1% of the labels, we achieve 85.8% top-5 accuracy, outperforming AlexNet with 100X fewer labels.

Momentum Contrast for Unsupervised Visual Representation Learning

Abstract: We present Momentum Contrast (MoCo) for unsupervised visual representation learning. From a perspective on contrastive learning as dictionary look-up, we build a dynamic dictionary with a queue and a moving-averaged encoder. This enables building a large and consistent dictionary on-the-fly that facilitates contrastive unsupervised learning. MoCo provides competitive results under the common linear protocol on ImageNet classification. More importantly, the representations learned by MoCo transfer well to downstream tasks. MoCo can outperform its supervised pre-training counterpart in 7 detection/segmentation tasks on PASCAL VOC, COCO, and other datasets, sometimes surpassing it by large margins. This suggests that the gap between unsupervised and supervised representation learning has been largely closed in many vision tasks.

Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning

Abstract: We introduce Bootstrap Your Own Latent (BYOL), a new approach to self-supervised image representation learning. BYOL relies on two neural networks, referred to as online and target networks, that interact and learn from each other. From an augmented view of an image, we train the online network to predict the target network representation of the same image under a different augmented view. At the same time, we update the target network with a slow-moving average of the online network. While state-of-the art methods rely on negative pairs, BYOL achieves a new state of the art without them. BYOL reaches $74.3\%$ top-1 classification accuracy on ImageNet using a linear evaluation with a ResNet-50 architecture and $79.6\%$ with a larger ResNet. We show that BYOL performs on par or better than the current state of the art on both transfer and semi-supervised benchmarks. Our implementation and pretrained models are given on GitHub.

Representation Learning with Contrastive Predictive Coding

Abstract: While supervised learning has enabled great progress in many applications, unsupervised learning has not seen such widespread adoption, and remains an important and challenging endeavor for artificial intelligence. In this work, we propose a universal unsupervised learning approach to extract useful representations from high-dimensional data, which we call Contrastive Predictive Coding. The key insight of our model is to learn such representations by predicting the future in latent space by using powerful autoregressive models. We use a probabilistic contrastive loss which induces the latent space to capture information that is maximally useful to predict future samples. It also makes the model tractable by using negative sampling. While most prior work has focused on evaluating representations for a particular modality, we demonstrate that our approach is able to learn useful representations achieving strong performance on four distinct domains: speech, images, text and reinforcement learning in 3D environments.

Exploring Simple Siamese Representation Learning

Abstract: Siamese networks have become a common structure in various recent models for unsupervised visual representation learning. These models maximize the similarity between two augmentations of one image, subject to certain conditions for avoiding collapsing solutions. In this paper, we report surprising empirical results that simple Siamese networks can learn meaningful representations even using none of the following: (i) negative sample pairs, (ii) large batches, (iii) momentum encoders. Our experiments show that collapsing solutions do exist for the loss and structure, but a stop-gradient operation plays an essential role in preventing collapsing. We provide a hypothesis on the implication of stop-gradient, and further show proof-of-concept experiments verifying it. Our "SimSiam" method achieves competitive results on ImageNet and downstream tasks. We hope this simple baseline will motivate people to rethink the roles of Siamese architectures for unsupervised representation learning. Code is made available.1

Summary of instance-level contrastive methods for time series representation learning. DA indicates whether the model utilizes data augmentations. ACC refers to accuracy, F1 refers to F1 score.

Contrastive methods learn meaningful representations from time series by optimizing self-discrimination tasks. Instead of directly modeling the complex raw data, they employ pretext tasks that leverage the underlying similarity between samples, which eliminates the need for reconstructing the complete input and allows for the discovery of contextualized underlying factors of variations {{cite:51941f9}}. Contrastive methods typically generate augmented views of the raw data through various transformations and then learn representations by contrasting positive samples against negative samples {{cite:274251f}}. Exploring negative samples in a completely unsupervised manner can be challenging for contrastive learning, as it often encounters hard negative samples with features that are highly similar to anchors but have different labels. To address this, researchers have explored various techniques to introduce more negatives, such as increasing the batch size {{cite:b7f4787}} or exploiting external data structures {{cite:8f338a9}}, {{cite:3bec721}}. There are also studies that remove the need of using negative samples and instead use other strategies such as utilizing a predictor {{cite:c57d039}}, a stop-gradient operation {{cite:5565e4d}}, or clustering {{cite:c850344}} to learn the mapping. Within the realm of computer vision, contrastive methods are commonly categorized into instance-level and prototype-level methods, with each approach focusing on different levels of contrast. Additionally, there are specific methods tailored for time series data, which consider the temporal-level contrast. We analyze the similarities and differences between these representative contrastive models in terms of their main contributions, backbone architectures, data augmentations (DA), datasets, evaluation strategies, and performance metrics. Contrastive learning enables the discovery of informative representations in time series data without relying on explicit reconstruction tasks. This analysis provides insights into the key characteristics and approaches of representative contrastive models in the field of representation learning for time series data. The concise overview of instance-level, prototype-level, and temporal-level methods can be found in Table {{table:6c7eb62e-ed78-4d78-897c-524de22a05fd}} , Table {{table:d8fc4852-a4da-4cbb-82d8-8484378451f4}} , and Table {{table:db6f7e97-94ad-457e-822e-6b784da1befd}} , respectively. The following sections provide more in-depth details about these methods.

Instance-level Contrastive Learning Models

Instance-level contrastive learning models treat individual samples independently for the purpose of instance discrimination. They utilize data augmentations to transform original inputs into a new embedding space. Within this space, augmentations derived from the same sample are considered as positive pairs, while those from different samples are treated as negative pairs. During training, these models are optimized by maximizing the similarity between representations of positive pairs, while simultaneously minimizing the similarity between representations of negative pairs. Instance-level contrastive methods achieve significant performance improvement by setting larger batch size {{cite:b7f4787}}, using stronger augmentations {{cite:88881f3}}, {{cite:3cb9950}}, or introducing an additional storage space to store more contrastive candidates {{cite:3b9f1f4}}, {{cite:8f338a9}}. However, these methods can be memory-intensive, and ensuring the consistency of representations is challenging when randomly extracting negative samples from an additional storage space. Recent advancements in non-contrastive learning, which eliminate the need for negative samples, have exhibited promising outcomes {{cite:83c8938}}. BYOL {{cite:c57d039}} and SimSiam {{cite:5565e4d}} serve as exemplars, achieving remarkable results without the reliance on negative samples. These methods leverage an additional learnable predictor and employ a stop-gradient operation to prevent collapsing, contributing to their successful performance {{cite:f8588db}}. Table {{table:6c7eb62e-ed78-4d78-897c-524de22a05fd}}  provides a succinct summary of instance-level contrastive methods for time series representation learning. Further elaboration concerning these methods are explicated below.

A comparative analysis of using different backbones

Table {{table:44b9bdb3-230c-4ef7-9cd1-98b788cb1f83}}  presents a comparative analysis of employing different backbones in the SimCLR model for unsupervised time series representation learning. Typically, we find that most models summarized in Table {{table:6c7eb62e-ed78-4d78-897c-524de22a05fd}} , Table {{table:d8fc4852-a4da-4cbb-82d8-8484378451f4}}  and Table {{table:db6f7e97-94ad-457e-822e-6b784da1befd}}  utilize ResNets {{cite:a30ba3d}} as the backbone and achieve satisfactory performance. Therefore, we compare the results of using ResNet-18 and ResNet-50 as backbones. Furthermore, we evaluate InceptionTime {{cite:653c9a1}}, which is specifically designed for time series data by incorporating the temporal dependencies. As shown in Table {{table:44b9bdb3-230c-4ef7-9cd1-98b788cb1f83}} , InceptionTime outperforms the general-purpose ResNets in most datasets. This observation highlights the capability of InceptionTime to identify both local and global shape patterns (i.e., low-level and high-level features) in time series data. InceptionTime proves to be exceptionally fitting for time series data due to its utilization of 1D convolutions with multiple kernel sizes in parallel. This approach facilitates efficient feature extraction on time series data across various scales, making it well-suited to capture diverse granularities of semantics in time series data. In contrast, ResNets primarily rely on 2D convolutions and are optimized for image data, which may not yield the same level of effectiveness when applied to time series data analysis.


ImageNet(tage label), ILSVRC(full annotation)

ImageNEt(tage label), Youtube-objects (unlabelled)

Initialization based on classification network and subgraph discovery + iterative Fast RCNN learning

Discriminative region minning+transferring tracking object pattern + learn object detector

Simple image initialization + graph-based representation adaptation on hard image

.Max-margin visual concept discovery + Domain-specific detector selection

Revisiting knowledge transfer for training object class detectors

Abstract: We propose to revisit knowledge transfer for training object detectors on target classes from weakly supervised training images, helped by a set of source classes with bounding-box annotations. We present a unified knowledge transfer framework based on training a single neural network multi-class object detector over all source classes, organized in a semantic hierarchy. This generates proposals with scores at multiple levels in the hierarchy, which we use to explore knowledge transfer over a broad range of generality, ranging from class-specific (bicycle to motorbike) to class-generic (objectness to any class). Experiments on the 200 object classes in the ILSVRC 2013 detection dataset show that our technique: (1) leads to much better performance on the target classes (70.3% CorLoc, 36.9% mAP) than a weakly supervised baseline which uses manually engineered objectness [11] (50.5% CorLoc, 25.4% mAP). (2) delivers target object detectors reaching 80% of the mAP of their fully supervised counterparts. (3) outperforms the best reported transfer learning results on this dataset (+41% CorLoc and +3% mAP over [18, 46], +16.2% mAP over [32]). Moreover, we also carry out several across-dataset knowledge transfer experiments [27, 24, 35] and find that (4) our technique outperforms the weakly supervised baseline in all dataset pairs by 1.5x-1.9x, establishing its general applicability.

Deep Self-Taught Learning for Weakly Supervised Object Localization

Abstract: Most existing weakly supervised localization (WSL) approaches learn detectors by finding positive bounding boxes based on features learned with image-level supervision. However, those features do not contain spatial location related information and usually provide poor-quality positive samples for training a detector. To overcome this issue, we propose a deep self-taught learning approach, which makes the detector learn the object-level features reliable for acquiring tight positive samples and afterwards re-train itself based on them. Consequently, the detector progressively improves its detection ability and localizes more informative positive samples. To implement such self-taught learning, we propose a seed sample acquisition method via image-to-object transferring and dense subgraph discovery to find reliable positive samples for initializing the detector. An online supportive sample harvesting scheme is further proposed to dynamically select the most confident tight positive samples and train the detector in a mutual boosting way. To prevent the detector from being trapped in poor optima due to overfitting, we propose a new relative improvement of predicted CNN scores for guiding the self-taught learning process. Extensive experiments on PASCAL 2007 and 2012 show that our approach outperforms the state-of-the-arts, strongly validating its effectiveness.

Track and Transfer: Watching Videos to Simulate Strong Human Supervision for Weakly-Supervised Object Detection

Abstract: The status quo approach to training object detectors requires expensive bounding box annotations. Our framework takes a markedly different direction: we transfer tracked object boxes from weakly-labeled videos to weakly-labeled images to automatically generate pseudo ground-truth boxes, which replace manually annotated bounding boxes. We first mine discriminative regions in the weakly-labeled image collection that frequently/rarely appear in the positive/ negative images. We then match those regions to videos and retrieve the corresponding tracked object boxes. Finally, we design a hough transform algorithm to vote for the best box to serve as the pseudo GT for each image, and use them to train an object detector. Together, these lead to state-of-the-art weakly-supervised detection results on the PASCAL 2007 and 2010 datasets.

Webly Supervised Learning of Convolutional Networks

Abstract: We present an approach to utilize large amounts of web data for learning CNNs. Specifically inspired by curriculum learning, we present a two-step approach for CNN training. First, we use easy images to train an initial visual representation. We then use this initial CNN and adapt it to harder, more realistic images by leveraging the structure of data and categories. We demonstrate that our two-stage CNN outperforms a fine-tuned CNN trained on ImageNet on Pascal VOC 2012. We also demonstrate the strength of webly supervised learning by localizing objects in web images and training a R-CNN style detector. It achieves the best performance on VOC 2007 where no VOC training data is used. Finally, we show our approach is quite robust to noise and performs comparably even when we use image search results from March 2013 (pre-CNN image search era).

ConceptLearner: Discovering Visual Concepts from Weakly Labeled Image Collections

Abstract: Discovering visual knowledge from weakly labeled data is crucial to scale up computer vision recognition system, since it is expensive to obtain fully labeled data for a large number of concept categories. In this paper, we propose ConceptLearner, which is a scalable approach to discover visual concepts from weakly labeled image collections. Thousands of visual concept detectors are learned automatically, without human in the loop for additional annotation. We show that these learned detectors could be applied to recognize concepts at image-level and to detect concepts at image region-level accurately. Under domain-specific supervision, we further evaluate the learned concepts for scene recognition on SUN database and for object detection on Pascal VOC 2007. ConceptLearner shows promising performance compared to fully supervised and weakly supervised methods.

Summary of the approaches with fine-tuned deep models, which is a subcategory in the weakly supervised object localization and detection approaches based on the off-the-shelf deep models. * indicates a certain variation of the corresponding model. An approach is considered for general object category when it is tested for detecting more than five object categories in the corresponding literature.

The methods of this category fine-tune the off-the-shelf DNN models during the weakly supervised learning process to obtain strong object detectors {{cite:16e67f7}}, {{cite:45a1e26}}, {{cite:06f57e3}}.
A brief summary of these approaches is shown in Table {{table:22303f37-37ef-4ba8-ae65-e30fecc5f668}} .


Facial Attribute Capsules for Noise Face Super Resolution

Abstract: Existing face super-resolution (SR) methods mainly assume the input image to be noise-free. Their performance degrades drastically when applied to real-world scenarios where the input image is always contaminated by noise. In this paper, we propose a Facial Attribute Capsules Network (FACN) to deal with the problem of high-scale super-resolution of noisy face image. Capsule is a group of neurons whose activity vector models different properties of the same entity. Inspired by the concept of capsule, we propose an integrated representation model of facial information, which named Facial Attribute Capsule (FAC). In the SR processing, we first generated a group of FACs from the input LR face, and then reconstructed the HR face from this group of FACs. Aiming to effectively improve the robustness of FAC to noise, we generate FAC in semantic, probabilistic and facial attributes manners by means of integrated learning strategy. Each FAC can be divided into two sub-capsules: Semantic Capsule (SC) and Probabilistic Capsule (PC). Them describe an explicit facial attribute in detail from two aspects of semantic representation and probability distribution. The group of FACs model an image as a combination of facial attribute information in the semantic space and probabilistic space by an attribute-disentangling way. The diverse FACs could better combine the face prior information to generate the face images with fine-grained semantic attributes. Extensive benchmark experiments show that our method achieves superior hallucination results and outperforms state-of-the-art for very low resolution (LR) noise face image super resolution.

Learning Spatial Attention for Face Super-Resolution

Abstract: General image super-resolution techniques have difficulties in recovering detailed face structures when applying to low resolution face images. Recent deep learning based methods tailored for face images have achieved improved performance by jointly trained with additional task such as face parsing and landmark prediction. However, multi-task learning requires extra manually labeled data. Besides, most of the existing works can only generate relatively low resolution face images (<italic>e.g</italic>., <inline-formula> <tex-math notation="LaTeX">$128\times 128$ </tex-math></inline-formula>), and their applications are therefore limited. In this paper, we introduce a novel SPatial Attention Residual Network (SPARNet) built on our newly proposed Face Attention Units (FAUs) for face super-resolution. Specifically, we introduce a spatial attention mechanism to the vanilla residual blocks. This enables the convolutional layers to adaptively bootstrap features related to the key face structures and pay less attention to those less feature-rich regions. This makes the training more effective and efficient as the key face structures only account for a very small portion of the face image. Visualization of the attention maps shows that our spatial attention network can capture the key face structures well even for very low resolution faces (<italic>e.g</italic>., <inline-formula> <tex-math notation="LaTeX">$16\times 16$ </tex-math></inline-formula>). Quantitative comparisons on various kinds of metrics (including PSNR, SSIM, identity similarity, and landmark detection) demonstrate the superiority of our method over current state-of-the-arts. We further extend SPARNet with multi-scale discriminators, named as SPARNetHD, to produce high resolution results (<italic>i.e</italic>., <inline-formula> <tex-math notation="LaTeX">$512\times 512$ </tex-math></inline-formula>). We show that SPARNetHD trained with synthetic data can not only produce high quality and high resolution outputs for synthetically degraded face images, but also show good generalization ability to real world low quality face images. Codes are available at <uri>https://github.com/chaofengc/Face-SPARNet</uri>.

Speed and overhead comparison of typical non-blind methods that measured on {{formula:bacdef69-0a73-4338-bbc2-08cca00ef811}}  images. We test all models using an NVIDIA RTX 3090 GPU.

Regarding the non-blind task, we chose to focus on evaluating non-blind super-resolution methods due to their predominant emphasis in the field. TABLE {{table:438c2c9c-6e63-40d5-80ce-3b179d59e075}}  presents a compilation of ten state-of-the-art non-blind methods, including fine-tuned image restoration methods {{cite:ff5162a}}, {{cite:e38300e}}, methods based on attention mechanisms {{cite:d155def}}, {{cite:47e8184}}, {{cite:aeb9c8a}}, {{cite:e41023b}}, and methods relying on various priors {{cite:a2fdad0}}, {{cite:c61e871}}, {{cite:1e53ab8}}. Among these, methods employing hybrid attention mechanisms, namely CTCNet {{cite:47e8184}}, SCTANet {{cite:aeb9c8a}}, and SFMNet {{cite:e41023b}}, achieve either the best or second-best performance across all metrics on both test sets. TABLE {{table:5c26ffd4-6e26-4e9e-80b9-0a7ac52b2cb3}}  provides detailed information about the model characteristics of these methods, including parameters, computation, and inference duration. Furthermore, Fig. {{table:438c2c9c-6e63-40d5-80ce-3b179d59e075}}  visually illustrates the efficiency of these techniques through three perspectives: performance, inference speed, and model size. Notably, attention-based methods, particularly SFMNet {{cite:e41023b}}, stand out as they achieve superior performance while maintaining smaller computational loads. Finally, Fig. {{figure:17b12bbe-9a5a-4a30-ab1b-591cec77003f}}  provides a visual comparison of these methods.

A range of state-of-the-art methods were selected for the blind task, including approaches that do not rely on prior such as network architecture design {{cite:0e5bc63}}, {{cite:211cd89}} and diffusion modeling techniques {{cite:cff1347}}). Additionally, techniques utilizing internally-specific priors such as parsing maps {{cite:f444336}} and 3D face shapes {{cite:7f0e9be}} were considered. Furthermore, methods employing external compensatory prior like pre-trained StyleGAN prior {{cite:935b4b7}}, {{cite:38a4831}}, {{cite:7f0e9be}}, pre-trained VQGAN prior {{cite:0dcfed6}}, {{cite:97510da}}, face dictionary {{cite:ea2fcaa}}, {{cite:72eb0f9}}, and reference prior {{cite:2a4214c}}) were also included. To evaluate the application scope of non-blind and blind methods, we randomly selected several real face photos and restored them using the fine-tuned non-blind method SFMNet {{cite:e41023b}} and the blind method GFPGAN {{cite:38a4831}} respectively. As shown in Fig. {{table:5c26ffd4-6e26-4e9e-80b9-0a7ac52b2cb3}} , it is evident that SFMNet struggles to effectively handle real-world photos, while GFPGAN, despite showing some racial bias in certain images, generally offers superior visual quality. Consequently, the blind method holds greater promise for real-world applications.


FairFuzz: A Targeted Mutation Strategy for Increasing Greybox Fuzz Testing Coverage

Abstract: In recent years, fuzz testing has proven itself to be one of the most effective techniques for finding correctness bugs and security vulnerabilities in practice. One particular fuzz testing tool, American Fuzzy Lop (AFL), has become popular thanks to its ease-of-use and bug-finding power. However, AFL remains limited in the bugs it can find since it simply does not cover large regions of code. If it does not cover parts of the code, it will not find bugs there. We propose a two-pronged approach to increase the coverage achieved by AFL. First, the approach automatically identifies branches exercised by few AFL-produced inputs (rare branches), which often guard code that is empirically hard to cover by naïvely mutating inputs. The second part of the approach is a novel mutation mask creation algorithm, which allows mutations to be biased towards producing inputs hitting a given rare branch. This mask is dynamically computed during fuzz testing and can be adapted to other testing targets. We implement this approach on top of AFL in a tool named FairFuzz. We conduct evaluation on real-world programs against state-of-the-art versions of AFL. We find that on these programs FairFuzz achieves high branch coverage at a faster rate that state-of-the-art versions of AFL. In addition, on programs with nested conditional structure, it achieves sustained increases in branch coverage after 24 hours (average 10.6% increase). In qualitative analysis, we find that FairFuzz has an increased capacity to automatically discover keywords.

Angora: Efficient Fuzzing by Principled Search

Abstract: Fuzzing is a popular technique for finding software bugs. However, the performance of the state-of-the-art fuzzers leaves a lot to be desired. Fuzzers based on symbolic execution produce quality inputs but run slow, while fuzzers based on random mutation run fast but have difficulty producing quality inputs. We propose Angora, a new mutation-based fuzzer that outperforms the state-of-the-art fuzzers by a wide margin. The main goal of Angora is to increase branch coverage by solving path constraints without symbolic execution. To solve path constraints efficiently, we introduce several key techniques: scalable byte-level taint tracking, context-sensitive branch count, search based on gradient descent, and input length exploration. On the LAVA-M data set, Angora found almost all the injected bugs, found more bugs than any other fuzzer that we compared with, and found eight times as many bugs as the second-best fuzzer in the program who. Angora also found 103 bugs that the LAVA authors injected but could not trigger. We also tested Angora on eight popular, mature open source programs. Angora found 6, 52, 29, 40 and 48 new bugs in file, jhead, nm, objdump and size, respectively. We measured the coverage of Angora and evaluated how its key techniques contribute to its impressive performance.

Fuzzers evaluated.

An overview of the eight mutational fuzzers selected for evaluation in this paper are shown in Table {{table:574e624a-a477-48a2-8a85-80648e7d1f48}} . These fuzzers were selected for evaluation using the following criteria.


Pose Guided Person Image Generation

Abstract: This paper proposes the novel Pose Guided Person Generation Network (PG$^2$) that allows to synthesize person images in arbitrary poses, based on an image of that person and a novel pose. Our generation framework PG$^2$ utilizes the pose information explicitly and consists of two key stages: pose integration and image refinement. In the first stage the condition image and the target pose are fed into a U-Net-like network to generate an initial but coarse image of the person with the target pose. The second stage then refines the initial and blurry result by training a U-Net-like generator in an adversarial way. Extensive experimental results on both 128$\times$64 re-identification images and 256$\times$256 fashion photos show that our model generates high-quality person images with convincing details.

Pose Guided Human Video Generation

Abstract: Due to the emergence of Generative Adversarial Networks, video synthesis has witnessed exceptional breakthroughs. However, existing methods lack a proper representation to explicitly control the dynamics in videos. Human pose, on the other hand, can represent motion patterns intrinsically and interpretably, and impose the geometric constraints regardless of appearance. In this paper, we propose a pose guided method to synthesize human videos in a disentangled way: plausible motion prediction and coherent appearance generation. In the first stage, a Pose Sequence Generative Adversarial Network (PSGAN) learns in an adversarial manner to yield pose sequences conditioned on the class label. In the second stage, a Semantic Consistent Generative Adversarial Network (SCGAN) generates video frames from the poses while preserving coherent appearances in the input image. By enforcing semantic consistency between the generated and ground-truth poses at a high feature level, our SCGAN is robust to noisy or abnormal poses. Extensive experiments on both human action and human face datasets manifest the superiority of the proposed method over other state-of-the-arts.

Categories of pose transfer methods. Again, they can be classified depending on whether one model can work for only one person or any persons. Some of the frameworks only focus on generating single images, while others also demonstrate their effectiveness on videos. Works with * do not use GANs in their framework.

Pose transfer techniques aim at transferring the body pose of one person to another person. It can be seen as the whole body counterpart of face reenactment. In contrast to the talking head generation, which usually shares similar motions, body poses have more varieties and are thus much harder to synthesize. Early works focus on simple pose transfers that generate low resolution and lower quality images. They only work on single images instead of videos. Recent works have shown their capability to generate high quality and high-resolution videos for challenging poses but can only work on a particular person per model. Very recently, several works attempt to perform subject-agnostic video synthesis. A summary of the categories is shown in Table {{table:fccdfa22-7deb-47a2-a5a0-54bf0759c083}} . Below we introduce each category in more detail.


The speaker model encodes each individual speaker into a vector to capture individual characteristics; Generating personal responses matching a specific user

Training a response generation model on a small personalized dialogue data, and then training an autoencoder model with non-conversational data; Sharing parameters of the two models to obtain the personalized dialogue model

A Persona-Based Neural Conversation Model

Abstract: We present persona-based models for handling the issue of speaker consistency in neural response generation. A speaker model encodes personas in distributed embeddings that capture individual characteristics such as background information and speaking style. A dyadic speaker-addressee model captures properties of interactions between two interlocutors. Our models yield qualitative performance improvements in both perplexity and BLEU scores over baseline sequence-to-sequence models, with similar gains in speaker consistency as measured by human judges.

Multi-Task Learning for Speaker-Role Adaptation in Neural Conversation Models

Abstract: Building a persona-based conversation agent is challenging owing to the lack of large amounts of speaker-specific conversation data for model training. This paper addresses the problem by proposing a multi-task learning approach to training neural conversation models that leverages both conversation data across speakers and other types of data pertaining to the speaker and speaker roles to be modeled. Experiments show that our approach leads to significant improvements over baseline model quality, generating responses that capture more precisely speakers’ traits and speaking styles. The model offers the benefits of being algorithmically simple and easy to implement, and not relying on large quantities of data representing specific individual speakers.

A summary of personalized text generation methods

Various human characteristics significantly impact interpersonal communication and human writing style. In other words, personalization plays a key role in enhancing the quality of text generation system. On the one hand, personalization is vital for creating truly smart dialogue agents which can be seamlessly incorporated into the lives of human beings. On the other hand, it ensures the generated product review content depends not only on the characteristics of the product, but also on the preferences of specific users, endowing the authenticity for the generated reviews. Several efforts for personalized text generation are conducted, as summarized in Table  {{table:91f04c59-221c-412c-9a30-223a1a13e3f9}} , and we will discuss them in details below.


Federated (differentially-private) image synthesis

Estimating and Improving Fairness with Adversarial Learning

Abstract: Fairness and accountability are two essential pillars for trustworthy Artificial Intelligence (AI) in healthcare. However, the existing AI model may be biased in its decision marking. To tackle this issue, we propose an adversarial multi-task training strategy to simultaneously mitigate and detect bias in the deep learning-based medical image analysis system. Specifically, we propose to add a discrimination module against bias and a critical module that predicts unfairness within the base classification model. We further impose an orthogonality regularization to force the two modules to be independent during training. Hence, we can keep these deep learning tasks distinct from one another, and avoid collapsing them into a singular point on the manifold. Through this adversarial training method, the data from the underprivileged group, which is vulnerable to bias because of attributes such as sex and skin tone, are transferred into a domain that is neutral relative to these attributes. Furthermore, the critical module can predict fairness scores for the data with unknown sensitive attributes. We evaluate our framework on a large-scale public-available skin lesion dataset under various fairness evaluation metrics. The experiments demonstrate the effectiveness of our proposed method for estimating and improving fairness in the deep learning-based medical image analysis system.

Multi-modal AsynDGAN: Learn From Distributed Medical Image Data without Sharing Private Information

Abstract: As deep learning technologies advance, increasingly more data is necessary to generate general and robust models for various tasks. In the medical domain, however, large-scale and multi-parties data training and analyses are infeasible due to the privacy and data security concerns. In this paper, we propose an extendable and elastic learning framework to preserve privacy and security while enabling collaborative learning with efficient communication. The proposed framework is named distributed Asynchronized Discriminator Generative Adversarial Networks (AsynDGAN), which consists of a centralized generator and multiple distributed discriminators. The advantages of our proposed framework are five-fold: 1) the central generator could learn the real data distribution from multiple datasets implicitly without sharing the image data; 2) the framework is applicable for single-modality or multi-modality data; 3) the learned generator can be used to synthesize samples for down-stream learning tasks to achieve close-to-real performance as using actual samples collected from multiple data centers; 4) the synthetic samples can also be used to augment data or complete missing modalities for one single data center; 5) the learning process is more efficient and requires lower bandwidth than other distributed deep learning methods.

No Surprises: Training Robust Lung Nodule Detection for Low-Dose CT Scans by Augmenting with Adversarial Attacks

Abstract: Detecting malignant pulmonary nodules at an early stage can allow medical interventions which may increase the survival rate of lung cancer patients. Using computer vision techniques to detect nodules can improve the sensitivity and the speed of interpreting chest CT for lung cancer screening. Many studies have used CNNs to detect nodule candidates. Though such approaches have been shown to outperform the conventional image processing based methods regarding the detection accuracy, CNNs are also known to be limited to generalize on under-represented samples in the training set and prone to imperceptible noise perturbations. Such limitations can not be easily addressed by scaling up the dataset or the models. In this work, we propose to add adversarial synthetic nodules and adversarial attack samples to the training data to improve the generalization and the robustness of the lung nodule detection systems. To generate hard examples of nodules from a differentiable nodule synthesizer, we use projected gradient descent (PGD) to search the latent code within a bounded neighbourhood that would generate nodules to decrease the detector response. To make the network more robust to unanticipated noise perturbations, we use PGD to search for noise patterns that can trigger the network to give over-confident mistakes. By evaluating on two different benchmark datasets containing consensus annotations from three radiologists, we show that the proposed techniques can improve the detection performance on real CT data. To understand the limitations of both the conventional networks and the proposed augmented networks, we also perform stress-tests on the false positive reduction networks by feeding different types of artificially produced patches. We show that the augmented networks are more robust to both under-represented nodules as well as resistant to noise perturbations.

PathologyGAN: Learning deep representations of cancer tissue

Abstract: Histopathological images of tumors contain abundant information about how tumors grow and how they interact with their micro-environment. Better understanding of tissue phenotypes in these images could reveal novel determinants of pathological processes underlying cancer, and in turn improve diagnosis and treatment options. Advances of Deep learning makes it ideal to achieve those goals, however, its application is limited by the cost of high quality labels from patients data. Unsupervised learning, in particular, deep generative models with representation learning properties provides an alternative path to further understand cancer tissue phenotypes, capturing tissue morphologies. In this paper, we develop a framework which allows GANs to capture key tissue features and uses these characteristics to give structure to its latent space. To this end, we trained our model on two different datasets, an H&E colorectal cancer tissue from the National Center for Tumor diseases (NCT) and an H&E breast cancer tissue from the Netherlands Cancer Institute (NKI) and Vancouver General Hospital (VGH). Composed of 86 slide images and 576 TMAs respectively. We show that our model generates high quality images, with a FID of 16.65 (breast cancer) and 32.05 (colorectal cancer). We further assess the quality of the images with cancer tissue characteristics (e.g. count of cancer, lymphocytes, or stromal cells), using quantitative information to calculate the FID and showing consistent performance of 9.86. Additionally, the latent space of our model shows an interpretable structure and allows semantic vector operations that translate into tissue feature transformations. Furthermore, ratings from two expert pathologists found no significant difference between our generated tissue images from real ones. The code, images, and pretrained models are available at https://github.com/AdalbertoCq/Pathology-GAN

mycorrectSelection of studies that employ data synthesis and adversarial networks methodology curated based on their promising potential towards solving the cancer imaging challenges surveyed in sections -. Each of the studies represents one concrete proposed solution to one of the challenges.

mycorrectIn our analysis we first sample the present-day challenges in cancer imaging that were surveyed in Section  and summarised in Figure {{figure:5725370d-41bf-4252-9b46-d09a5f19fcd9}} . Next, we carefully select representative adversarial network publications to represent a particular challenge and its solution. This selection is based on the criteria that the publication (a) proposes a particularly promising solution to its respective challenge, (b) contributes a methodology that is generally-applicable across domains and (c) report promising results. Most of the sampled publications further (d) have shown more impact and were referenced in other relevant studies. The selected studies are displayed in Table {{table:ed4929cb-3102-4b02-9af2-3e7ace645355}}  together with their representative solution and associated cancer imaging challenge.