MesoNet: a Compact Facial Video Forgery Detection Network

Abstract: This paper presents a method to automatically and efficiently detect face tampering in videos, and particularly focuses on two recent techniques used to generate hyper-realistic forged videos: Deepfake and Face2Face. Traditional image forensics techniques are usually not well suited to videos due to the compression that strongly degrades the data. Thus, this paper follows a deep learning approach and presents two networks, both with a low number of layers to focus on the mesoscopic properties of images. We evaluate those fast networks on both an existing dataset and a dataset we have constituted from online videos. The tests demonstrate a very successful detection rate with more than 98% for Deepfake and 95% for Face2Face.

CNN-Generated Images Are Surprisingly Easy to Spot… for Now

Abstract: In this work we ask whether it is possible to create a "universal" detector for telling apart real images from these generated by a CNN, regardless of architecture or dataset used. To test this, we collect a dataset consisting of fake images generated by 11 different CNN-based image generator models, chosen to span the space of commonly used architectures today (ProGAN, StyleGAN, BigGAN, CycleGAN, StarGAN, GauGAN, DeepFakes, cascaded refinement networks, implicit maximum likelihood estimation, second-order attention super-resolution, seeing-in-the-dark). We demonstrate that, with careful pre- and post-processing and data augmentation, a standard image classifier trained on only one specific CNN generator (ProGAN) is able to generalize surprisingly well to unseen architectures, datasets, and training methods (including the just released StyleGAN2). Our findings suggest the intriguing possibility that today's CNN-generated images share some common systematic flaws, preventing them from achieving realistic image synthesis.

EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks

Abstract:
Convolutional Neural Networks (ConvNets) are commonly developed at a fixed resource budget, and then scaled up for better accuracy if more resources are available. In this paper, we systematically study model scaling and identify that carefully balancing network depth, width, and resolution can lead to better performance. Based on this observation, we propose a new scaling method that uniformly scales all dimensions of depth/width/resolution using a simple yet highly effective compound coefficient. We demonstrate the effectiveness of this method on scaling up MobileNets and ResNet.
To go even further, we use neural architecture search to design a new baseline network and scale it up to obtain a family of models, called EfficientNets, which achieve much better accuracy and efficiency than previous ConvNets. In particular, our EfficientNet-B7 achieves state-of-the-art 84.4% top-1 / 97.1% top-5 accuracy on ImageNet, while being 8.4x smaller and 6.1x faster on inference than the best existing ConvNet. Our EfficientNets also transfer well and achieve state-of-the-art accuracy on CIFAR-100 (91.7%), Flowers (98.8%), and 3 other transfer learning datasets, with an order of magnitude fewer parameters. Source code is at this https URL.

Capsule-forensics: Using Capsule Networks to Detect Forged Images and Videos

Abstract: Recent advances in media generation techniques have made it easier for attackers to create forged images and videos. State-of-the-art methods enable the real-time creation of a forged version of a single video obtained from a social network. Although numerous methods have been developed for detecting forged images and videos, they are generally targeted at certain domains and quickly become obsolete as new kinds of attacks appear. The method introduced in this paper uses a capsule network to detect various kinds of spoofs, from replay attacks using printed images or recorded videos to computer-generated videos using deep convolutional neural networks. It extends the application of capsule networks beyond their original intention to the solving of inverse graphics problems.

Exposing DeepFake Videos By Detecting Face Warping Artifacts

Abstract: In this work, we describe a new deep learning based method that can effectively distinguish AI-generated fake videos (referred to as {\em DeepFake} videos hereafter) from real videos. Our method is based on the observations that current DeepFake algorithm can only generate images of limited resolutions, which need to be further warped to match the original faces in the source video. Such transforms leave distinctive artifacts in the resulting DeepFake videos, and we show that they can be effectively captured by convolutional neural networks (CNNs). Compared to previous methods which use a large amount of real and DeepFake generated images to train CNN classifier, our method does not need DeepFake generated images as negative training examples since we target the artifacts in affine face warping as the distinctive feature to distinguish real and fake images. The advantages of our method are two-fold: (1) Such artifacts can be simulated directly using simple image processing operations on a image to make it as negative example. Since training a DeepFake model to generate negative examples is time-consuming and resource-demanding, our method saves a plenty of time and resources in training data collection; (2) Since such artifacts are general existed in DeepFake videos from different sources, our method is more robust compared to others. Our method is evaluated on two sets of DeepFake video datasets for its effectiveness in practice.

Face X-Ray for More General Face Forgery Detection

Abstract: In this paper we propose a novel image representation called face X-ray for detecting forgery in face images. The face X-ray of an input face image is a greyscale image that reveals whether the input image can be decomposed into the blending of two images from different sources. It does so by showing the blending boundary for a forged image and the absence of blending for a real image. We observe that most existing face manipulation methods share a common step: blending the altered face into an existing background image. For this reason, face X-ray provides an effective way for detecting forgery generated by most existing face manipulation algorithms. Face X-ray is general in the sense that it only assumes the existence of a blending step and does not rely on any knowledge of the artifacts associated with a specific face manipulation technique. Indeed, the algorithm for computing face X-ray can be trained without fake images generated by any of the state-of-the-art face manipulation methods. Extensive experiments show that face X-ray remains effective when applied to forgery generated by unseen face manipulation techniques, while most existing face forgery detection or deepfake detection algorithms experience a significant performance drop.

On the Detection of Digital Face Manipulation

Abstract: Detecting manipulated facial images and videos is an increasingly important topic in digital media forensics. As advanced face synthesis and manipulation methods are made available, new types of fake face representations are being created which have raised significant concerns for their use in social media. Hence, it is crucial to detect manipulated face images and localize manipulated regions. Instead of simply using multi-task learning to simultaneously detect manipulated images and predict the manipulated mask (regions), we propose to utilize an attention mechanism to process and improve the feature maps for the classification task. The learned attention maps highlight the informative regions to further improve the binary classification (genuine face v. fake face), and also visualize the manipulated regions. To enable our study of manipulated face detection and localization, we collect a large-scale database that contains numerous types of facial forgeries. With this dataset, we perform a thorough analysis of data-driven fake face detection. We show that the use of an attention mechanism improves facial forgery detection and manipulated region localization.

CORE: Consistent Representation Learning for Face Forgery Detection

Abstract: Face manipulation techniques develop rapidly and arouse widespread public concerns. Despite that vanilla convolutional neural networks achieve acceptable performance, they suffer from the overfitting issue. To relieve this issue, there is a trend to introduce some erasing-based augmentations. We find that these methods indeed attempt to implicitly induce more consistent representations for different augmentations via assigning the same label for different augmented images. However, due to the lack of explicit regularization, the consistency between different representations is less satisfactory. Therefore, we constrain the consistency of different representations explicitly and propose a simple yet effective framework, COnsistent REpresentation Learning (CORE). Specifically, we first capture the different representations with different augmentations, then regularize the cosine distance of the representations to enhance the consistency. Extensive experiments (in-dataset and cross-dataset) demonstrate that CORE performs favorably against state-of-the-art face forgery detection methods. Our code is available at https://github.com/niyunsheng/CORE.

UCF: Uncovering Common Features for Generalizable Deepfake Detection

Abstract: Deepfake detection remains a challenging task due to the difficulty of generalizing to new types of forgeries. This problem primarily stems from the overfitting of existing detection methods to forgery-irrelevant features and method-specific patterns. The latter has been rarely studied and not well addressed by previous works. This paper presents a novel approach to address the two types of overfitting issues by uncovering common forgery features. Specifically, we first propose a disentanglement framework that decomposes image information into three distinct components: forgery-irrelevant, method-specific forgery, and common forgery features. To ensure the decoupling of method-specific and common forgery features, a multi-task learning strategy is employed, including a multi-class classification that predicts the category of the forgery method and a binary classification that distinguishes the real from the fake. Additionally, a conditional decoder is designed to utilize forgery features as a condition along with forgery-irrelevant features to generate reconstructed images. Furthermore, a contrastive regularization technique is proposed to encourage the disentanglement of the common and specific forgery features. Ultimately, we only utilize the common forgery features for the purpose of generalizable deepfake detection. Extensive evaluations demonstrate that our framework can perform superior generalization than current state-of-the-art methods.

Spatial-Phase Shallow Learning: Rethinking Face Forgery Detection in Frequency Domain

Abstract: The remarkable success in face forgery techniques has received considerable attention in computer vision due to security concerns. We observe that up-sampling is a necessary step of most face forgery techniques, and cumulative up-sampling will result in obvious changes in the frequency domain, especially in the phase spectrum. According to the property of natural images, the phase spectrum preserves abundant frequency components that provide extra information and complement the loss of the amplitude spectrum. To this end, we present a novel Spatial-Phase Shallow Learning (SPSL) method, which combines spatial image and phase spectrum to capture the up-sampling artifacts of face forgery to improve the transferability, for face forgery detection. And we also theoretically analyze the validity of utilizing the phase spectrum. Moreover, we notice that local texture information is more crucial than high-level semantic information for the face forgery detection task. So we reduce the receptive fields by shallowing the network to suppress high-level features and focus on the local region. Extensive experiments show that SPSL can achieve the state-of-the-art performance on cross-datasets evaluation as well as multi-class classification and obtain comparable results on single dataset evaluation.

Generalizing Face Forgery Detection with High-frequency Features

Abstract: Current face forgery detection methods achieve high accuracy under the within-database scenario where training and testing forgeries are synthesized by the same algorithm. However, few of them gain satisfying performance under the cross-database scenario where training and testing forgeries are synthesized by different algorithms. In this paper, we find that current CNN-based detectors tend to overfit to method-specific color textures and thus fail to generalize. Observing that image noises remove color textures and expose discrepancies between authentic and tampered regions, we propose to utilize the high-frequency noises for face forgery detection. We carefully devise three functional modules to take full advantage of the high-frequency features. The first is the multi-scale high-frequency feature extraction module that extracts high-frequency noises at multiple scales and composes a novel modality. The second is the residual-guided spatial attention module that guides the low-level RGB feature extractor to concentrate more on forgery traces from a new perspective. The last is the cross-modality attention module that leverages the correlation between the two complementary modalities to promote feature learning for each other. Comprehensive evaluations on several benchmark databases corroborate the superior generalization performance of our proposed method.

Summary of the compared deepfake detectors. For detectors without publicly available repositories, we undertake careful re-implementation, adhering to the instructions specified in the original papers.

Our benchmark has implemented a total of 15 established deepfake detection algorithms, as detailed in Tab. {{table:deae7984-cb96-4be5-bbd4-61be1d454163}} . The selection of these algorithms is guided by three criteria.
First, we prioritize methods that hold a classic status ({{formula:331aeca2-8764-427c-b534-30075d83ce4c}} , Xception), or those considered advanced, typically published in recent top-tier conferences or journals in computer vision or machine learning.
Second, our benchmark classifies detectors into three categories: naive detectors, spatial detectors, and frequency detectors. Our primary emphasis is on image forgery detection, hence, temporal-based detectors have not yet been incorporated. Moreover, we have refrained from including traditional detectors ({{formula:004ee701-de0f-4fe1-a33d-096675355cc8}} , Headpose {{cite:c473b8c}}) due to their limited scalability to large-scale datasets, making them less suitable for our benchmark's objectives.
Third, we aim to include methods that are straightforward to implement and reproduce. We notice that several existing methods involve a series of steps, some of which are reliant on third-party algorithms or heuristic strategies. These methods usually have numerous hyper-parameters and are fraught with uncertainty, making their implementation and reproduction challenging. Therefore, these methods without open-source codes are intentionally excluded from our benchmark.
However, it is important to note that there are also some non-open-source methods we employed that are derived from the code directly provided by their respective authors.

The Training Module currently accommodates 15 detectors across three categories: naive detector, spatial detector, and frequency detector, all of which are shown in Tab. {{table:deae7984-cb96-4be5-bbd4-61be1d454163}} .
1) Naive Detector leverages various CNN architectures to directly detect forgeries without relying on additional manually designed features.
2) Spatial Detector builds upon the backbone of CNNs used in the Naive Detector and further explores manual-designed algorithms to detect deepfake.
3) Frequency Detector focuses on utilizing information from the frequency domain and extracting frequency artifacts for detection.
Each detector implemented in our benchmark is managed in a streamlined and efficient way, with a YAML config file created for each one. This allows users to easily set their desired parameters, {{formula:8d661185-c74a-4c1f-971d-a7145731066e}} , batch size, learning rate, {{formula:9177337a-6f86-49f2-8d84-676f8159a7a1}} .
These detectors are trained on a unified trainer that records the metrics and losses during the training and evaluation process.
Thus, the training and evaluation processes, logging, and visualization are handled automatically, eliminating the need for manual specification.


The STRING database in 2023: protein–protein association networks and functional enrichment analyses for any sequenced genome of interest

Abstract: Abstract Much of the complexity within cells arises from functional and regulatory interactions among proteins. The core of these interactions is increasingly known, but novel interactions continue to be discovered, and the information remains scattered across different database resources, experimental modalities and levels of mechanistic detail. The STRING database (https://string-db.org/) systematically collects and integrates protein–protein interactions—both physical interactions as well as functional associations. The data originate from a number of sources: automated text mining of the scientific literature, computational interaction predictions from co-expression, conserved genomic context, databases of interaction experiments and known complexes/pathways from curated sources. All of these interactions are critically assessed, scored, and subsequently automatically transferred to less well-studied organisms using hierarchical orthology information. The data can be accessed via the website, but also programmatically and via bulk downloads. The most recent developments in STRING (version 12.0) are: (i) it is now possible to create, browse and analyze a full interaction network for any novel genome of interest, by submitting its complement of encoded proteins, (ii) the co-expression channel now uses variational auto-encoders to predict interactions, and it covers two new sources, single-cell RNA-seq and experimental proteomics data and (iii) the confidence in each experimentally derived interaction is now estimated based on the detection method used, and communicated to the user in the web-interface. Furthermore, STRING continues to enhance its facilities for functional enrichment analysis, which are now fully available also for user-submitted genomes.

Prediction of Compound Synthesis Accessibility Based on Reaction Knowledge Graph

Abstract: With the increasing application of deep-learning-based generative models for de novo molecule design, the quantitative estimation of molecular synthetic accessibility (SA) has become a crucial factor for prioritizing the structures generated from generative models. It is also useful for helping in the prioritization of hit/lead compounds and guiding retrosynthesis analysis. In this study, based on the USPTO and Pistachio reaction datasets, a chemical reaction network was constructed for the identification of the shortest reaction paths (SRP) needed to synthesize compounds, and different SRP cut-offs were then used as the threshold to distinguish a organic compound as either an easy-to-synthesize (ES) or hard-to-synthesize (HS) class. Two synthesis accessibility models (DNN-ECFP model and graph-based CMPNN model) were built using deep learning/machine learning algorithms. Compared to other existing synthesis accessibility scoring schemes, such as SYBA, SCScore, and SAScore, our results show that CMPNN (ROC AUC: 0.791) performs better than SYBA (ROC AUC: 0.76), albeit marginally, and outperforms SAScore and SCScore. Our prediction models based on historical reaction knowledge could be a potential tool for estimating molecule SA.

Drug Repurposing for Parkinson's Disease by Integrating Knowledge Graph Completion Model and Knowledge Fusion of Medical Literature

Abstract: The prevalence of Parkinson’s disease increases a tremendous medical and economic burden to society. Therefore, the effective drugs are urgently required. However, the traditional development of effective drugs is costly and risky. Drug repurposing, which identifies new applications for existing drugs, is a feasible strategy for discovering new drugs for Parkinson’s disease. Drug repurposing is based on sufficient medical knowledge. The local medical knowledge base with manually labeled data contains a large number of accurate, but not novel, medical knowledge, while the medical literature containing the latest knowledge is difficult to utilize, because of unstructured data. This paper proposes a framework, named Drug Repurposing for Parkinson’s disease by integrating Knowledge Graph Completion method and Knowledge Fusion of medical literature data (DRKF) in order to make full use of a local medical knowledge base containing accurate knowledge and medical literature with novel knowledge. DRKF first extracts the relations that are related to Parkinson’s disease from medical literature and builds a medical literature knowledge graph. After that, the literature knowledge graph is fused with a local medical knowledge base that integrates several specific medical knowledge sources in order to construct a fused medical knowledge graph. Subsequently, knowledge graph completion methods are leveraged to predict the drug candidates for Parkinson’s disease by using the fused knowledge graph. Finally, we employ classic machine learning methods to repurpose the drug for Parkinson’s disease and compare the results with the method only using the literature-based knowledge graph in order to confirm the effectiveness of knowledge fusion. The experiment results demonstrate that our framework can achieve competitive performance, which confirms the effectiveness of our proposed DRKF for drug repurposing against Parkinson’s disease. It could be a supplement to traditional drug discovery methods.

Disease Ontology: a backbone for disease semantic integration

Abstract: The Disease Ontology (DO) database (http://disease-ontology.org) represents a comprehensive knowledge base of 8043 inherited, developmental and acquired human diseases (DO version 3, revision 2510). The DO web browser has been designed for speed, efficiency and robustness through the use of a graph database. Full-text contextual searching functionality using Lucene allows the querying of name, synonym, definition, DOID and cross-reference (xrefs) with complex Boolean search strings. The DO semantically integrates disease and medical vocabularies through extensive cross mapping and integration of MeSH, ICD, NCI's thesaurus, SNOMED CT and OMIM disease-specific terms and identifiers. The DO is utilized for disease annotation by major biomedical databases (e.g. Array Express, NIF, IEDB), as a standard representation of human disease in biomedical ontologies (e.g. IDO, Cell line ontology, NIFSTD ontology, Experimental Factor Ontology, Influenza Ontology), and as an ontological cross mappings resource between DO, MeSH and OMIM (e.g. GeneWiki). The DO project (http://diseaseontology.sf.net) has been incorporated into open source tools (e.g. Gene Answers, FunDO) to connect gene and disease biomedical data through the lens of human disease. The next iteration of the DO web browser will integrate DO's extended relations and logical definition representation along with these biomedical resource cross-mappings.

DrugBank 5.0: a major update to the DrugBank database for 2018

Abstract: Abstract DrugBank (www.drugbank.ca) is a web-enabled database containing comprehensive molecular information about drugs, their mechanisms, their interactions and their targets. First described in 2006, DrugBank has continued to evolve over the past 12 years in response to marked improvements to web standards and changing needs for drug research and development. This year’s update, DrugBank 5.0, represents the most significant upgrade to the database in more than 10 years. In many cases, existing data content has grown by 100% or more over the last update. For instance, the total number of investigational drugs in the database has grown by almost 300%, the number of drug-drug interactions has grown by nearly 600% and the number of SNP-associated drug effects has grown more than 3000%. Significant improvements have been made to the quantity, quality and consistency of drug indications, drug binding data as well as drug-drug and drug-food interactions. A great deal of brand new data have also been added to DrugBank 5.0. This includes information on the influence of hundreds of drugs on metabolite levels (pharmacometabolomics), gene expression levels (pharmacotranscriptomics) and protein expression levels (pharmacoprotoemics). New data have also been added on the status of hundreds of new drug clinical trials and existing drug repurposing trials. Many other important improvements in the content, interface and performance of the DrugBank website have been made and these should greatly enhance its ease of use, utility and potential applications in many areas of pharmacological research, pharmaceutical science and drug education.

Resource of Existing Healthcare Knowledge Graphs (HKGs).

In this section, we aim to provide a detailed resource overview of the current state of healthcare knowledge graphs (HKGs), as a reference for researchers and healthcare professionals interested in developing and applying HKGs. A vast range of HKG resources from various domains is organized in Table {{table:fe5f73d1-309a-4e59-9934-0fdc6f7d40c1}} , with attribute information including HKG name, node types, edge types, statistics, and their applications.


OGB-LSC: A Large-Scale Challenge for Machine Learning on Graphs

Abstract: Enabling effective and efficient machine learning (ML) over large-scale graph data (e.g., graphs with billions of edges) can have a great impact on both industrial and scientific applications. However, existing efforts to advance large-scale graph ML have been largely limited by the lack of a suitable public benchmark. Here we present OGB Large-Scale Challenge (OGB-LSC), a collection of three real-world datasets for facilitating the advancements in large-scale graph ML. The OGB-LSC datasets are orders of magnitude larger than existing ones, covering three core graph learning tasks -- link prediction, graph regression, and node classification. Furthermore, we provide dedicated baseline experiments, scaling up expressive graph ML models to the massive datasets. We show that expressive models significantly outperform simple scalable baselines, indicating an opportunity for dedicated efforts to further improve graph ML at scale. Moreover, OGB-LSC datasets were deployed at ACM KDD Cup 2021 and attracted more than 500 team registrations globally, during which significant performance improvements were made by a variety of innovative techniques. We summarize the common techniques used by the winning solutions and highlight the current best practices in large-scale graph ML. Finally, we describe how we have updated the datasets after the KDD Cup to further facilitate research advances. The OGB-LSC datasets, baseline code, and all the information about the KDD Cup are available at https://ogb.stanford.edu/docs/lsc/ .

Benchmarking Graph Neural Networks

Abstract: Graph neural networks (GNNs) have become the standard toolkit for analyzing and learning from data on graphs. As the field grows, it becomes critical to identify key architectures and validate new ideas that generalize to larger, more complex datasets. Unfortunately, it has been increasingly difficult to gauge the effectiveness of new models in the absence of a standardized benchmark with consistent experimental settings. In this paper, we introduce a reproducible GNN benchmarking framework, with the facility for researchers to add new models conveniently for arbitrary datasets. We demonstrate the usefulness of our framework by presenting a principled investigation into the recent Weisfeiler-Lehman GNNs (WL-GNNs) compared to message passing-based graph convolutional networks (GCNs) for a variety of graph tasks, i.e. graph regression/classification and node/link prediction, with medium-scale datasets.

Benchmarking Graph Neural Networks

Benchmarking Graph Neural Networks

Benchmarking Graph Neural Networks

Overview of the datasets used in our experiments, covering graph-level, node-level, and edge-level tasks.

In Table {{table:a69858d8-2fc3-4529-9755-1ca48e6ea9b4}} , we list the information of the datasets used for training and evaluation. Next, we describe them in detail.


Robust Individual Circadian Parameter Estimation for Biosignal-based Personalisation of Cancer Chronotherapy

Abstract: In cancer treatment, chemotherapy is administered according a constant schedule. The chronotherapy approach, considering chronobiological drug delivery, adapts the chemotherapy profile to the circadian rhythms of the human organism. This reduces toxicity effects and at the same time enhances efficiency of chemotherapy. To personalize cancer treatment, chemotherapy profiles have to be further adapted to individual patients. Therefore, we present a new model to represent cycle phenomena in circadian rhythms. The model enables a more precise modelling of the underlying circadian rhythms. In comparison with the standard model, our model delivers better results in all defined quality indices. The new model can be used to adapt the chemotherapy profile efficiently to individual patients. The adaption to individual patients contributes to the aim of personalizing cancer therapy.

A Benchmark Data Set to Evaluate the Illumination Robustness of Image Processing Algorithms for Object Segmentation and Classification

Abstract: Developers of image processing routines rely on benchmark data sets to give qualitative comparisons of new image analysis algorithms and pipelines. Such data sets need to include artifacts in order to occlude and distort the required information to be extracted from an image. Robustness, the quality of an algorithm related to the amount of distortion is often important. However, using available benchmark data sets an evaluation of illumination robustness is difficult or even not possible due to missing ground truth data about object margins and classes and missing information about the distortion. We present a new framework for robustness evaluation. The key aspect is an image benchmark containing 9 object classes and the required ground truth for segmentation and classification. Varying levels of shading and background noise are integrated to distort the data set. To quantify the illumination robustness, we provide measures for image quality, segmentation and classification success and robustness. We set a high value on giving users easy access to the new benchmark, therefore, all routines are provided within a software package, but can as well easily be replaced to emphasize other aspects.

Nearest-Neighbor Based Non-Parametric Probabilistic Forecasting with Applications in Photovoltaic Systems

Abstract: The present contribution offers a simple methodology for the obtainment of data-driven interval forecasting models by combining pairs of quantile regressions. Those regressions are created without the usage of the non-differentiable pinball-loss function, but through a k-nearest-neighbors based training set transformation and traditional regression approaches. By leaving the underlying training algorithms of the data mining techniques unchanged, the presented approach simplifies the creation of quantile regressions with more complex techniques (e.g. artificial neural networks). The quality of the presented methodology is tested on the usecase of photovoltaic power forecasting, for which quantile regressions using polynomial models as well as artificial neural networks and support vector regressions are created. From the resulting evaluation values it can be concluded that acceptable interval forecasting models are created.

Selected SciXMiner applications

Selected SciXMiner applications are listed in Table {{table:6a12b4a1-b493-4551-b850-a141bcb54a72}} .


Siamese Transformer Pyramid Networks for Real-Time UAV Tracking

Abstract: Recent object tracking methods depend upon deep networks or convoluted architectures. Most of those trackers can hardly meet real-time processing requirements on mobile platforms with limited computing resources. In this work, we introduce the Siamese Transformer Pyramid Network (SiamTPN), which inherits the advantages from both CNN and Transformer architectures. Specifically, we exploit the inherent feature pyramid of a lightweight network (ShuffleNetV2) and reinforce it with a Transformer to construct a robust target-specific appearance model. A centralized architecture with lateral cross attention is developed for building augmented high-level feature maps. To avoid the computation and memory intensity while fusing pyramid representations with the Transformer, we further introduce the pooling attention module, which significantly reduces memory and time complexity while improving the robustness. Comprehensive experiments on both aerial and prevalent tracking benchmarks achieve competitive results while operating at high speed, demonstrating the effectiveness of SiamTPN. Moreover, our fastest variant tracker operates over 30 Hz on a single CPU-core and obtaining an AUC score of 58.1% on the LaSOT dataset. Source codes are available at https://github.com/RISC-NYUAD/SiamTPNTracker

Efficient Visual Tracking with Exemplar Transformers

Abstract: The design of more complex and powerful neural network models has significantly advanced the state-of-the-art in visual object tracking. These advances can be attributed to deeper networks, or the introduction of new building blocks, such as transformers. However, in the pursuit of increased tracking performance, runtime is often hindered. Furthermore, efficient tracking architectures have received surprisingly little attention. In this paper, we introduce the Exemplar Transformer, a transformer module utilizing a single instance level attention layer for realtime visual object tracking. E.T.Track, our visual tracker that incorporates Exemplar Transformer modules, runs at 47 FPS on a CPU. This is up to 8× faster than other transformer-based models. When compared to lightweight trackers that can operate in realtime on standard CPUs, E.T.Track consistently outperforms all other methods on the LaSOT [16], OTB-100 [52], NFS [27], TrackingNet [36], and VOTST2020 [29] datasets. Code and models are available at https://github.com/pblatter/ettrack.

Summary of Siamese Network Related Papers

Similarity information between two frames helps a lot in object tracking. Thus the Siamese network tries to learn the similarities and differentiate the inputs. This network has two parallel sub networks sharing the same weight and parameter space. Finally, the parameters between the twin networks are tied up and then trained on a certain loss function to measure the semantic similarity between them. The summary of applying Siamese network in MOT task is given in Table {{table:20adede6-ea15-4f2e-b20e-f73610567f06}} .


SlideImages: A Dataset for Educational Image Classification

Abstract: In the past few years, convolutional neural networks (CNNs) have achieved impressive results in computer vision tasks, which however mainly focus on photos with natural scene content. Besides, non-sensor derived images such as illustrations, data visualizations, figures, etc. are typically used to convey complex information or to explore large datasets. However, this kind of images has received little attention in computer vision. CNNs and similar techniques use large volumes of training data. Currently, many document analysis systems are trained in part on scene images due to the lack of large datasets of educational image data. In this paper, we address this issue and present SlideImages, a dataset for the task of classifying educational illustrations. SlideImages contains training data collected from various sources, e.g., Wikimedia Commons and the AI2D dataset, and test data collected from educational slides. We have reserved all the actual educational images as a test dataset in order to ensure that the approaches using this dataset generalize well to new educational images, and potentially other domains. Furthermore, we present a baseline system using a standard deep neural architecture and discuss dealing with the challenge of limited training data.

FigureQA: An Annotated Figure Dataset for Visual Reasoning

Abstract: We introduce FigureQA, a visual reasoning corpus of over one million question-answer pairs grounded in over 100,000 images. The images are synthetic, scientific-style figures from five classes: line plots, dot-line plots, vertical and horizontal bar graphs, and pie charts. We formulate our reasoning task by generating questions from 15 templates; questions concern various relationships between plot elements and examine characteristics like the maximum, the minimum, area-under-the-curve, smoothness, and intersection. To resolve, such questions often require reference to multiple plot elements and synthesis of information distributed spatially throughout a figure. To facilitate the training of machine learning systems, the corpus also includes side data that can be used to formulate auxiliary objectives. In particular, we provide the numerical data used to generate each figure as well as bounding-box annotations for all plot elements. We study the proposed visual reasoning task by training several models, including the recently proposed Relation Network as a strong baseline. Preliminary results indicate that the task poses a significant machine learning challenge. We envision FigureQA as a first step towards developing models that can intuitively recognize patterns from visual representations of data.

DVQA: Understanding Data Visualizations via Question Answering

Abstract: Bar charts are an effective way to convey numeric information, but today's algorithms cannot parse them. Existing methods fail when faced with even minor variations in appearance. Here, we present DVQA, a dataset that tests many aspects of bar chart understanding in a question answering framework. Unlike visual question answering (VQA), DVQA requires processing words and answers that are unique to a particular bar chart. State-of-the-art VQA algorithms perform poorly on DVQA, and we propose two strong baselines that perform considerably better. Our work will enable algorithms to automatically extract numeric and semantic information from vast quantities of bar charts found in scientific publications, Internet articles, business reports, and many other areas.

SciCap: Generating Captions for Scientific Figures

Abstract: Researchers use figures to communicate rich, complex information in scientific papers. The captions of these figures are critical to conveying effective messages. However, low-quality figure captions commonly occur in scientific articles and may decrease understanding. In this paper, we propose an end-to-end neural framework to automatically generate informative, high-quality captions for scientific figures. To this end, we introduce SCICAP, a large-scale figure-caption dataset based on computer science arXiv papers published between 2010 and 2020. After pre-processing - including figure-type classification, sub-figure identification, text normalization, and caption text selection - SCICAP contained more than two million figures extracted from over 290,000 papers. We then established baseline models that caption graph plots, the dominant (19.2%) figure type. The experimental results showed both opportunities and steep challenges of generating captions for scientific figures.

Datasets for scientific figure understanding

Automated methods for scientific figure understanding

Scientific figure understanding tasks such as figure classification, visual question-answering (VQA), or image captioning have received significant attention from the AI community in recent years {{cite:d8a24da}}, {{cite:bf245ac}}, {{cite:05d890b}}, {{cite:a679382}}, {{cite:8328e3f}}, {{cite:b6a8e62}}, {{cite:1c11ccd}}, {{cite:cb13ee0}}, {{cite:f0c79ab}}, {{cite:1bd1485}}, {{cite:74f2746}}, {{cite:6185a52}}. In Table {{table:b97ec2d4-73b6-4c17-96cb-5a80e833bce0}} , we describe a number of datasets that have been introduced to train models and evaluate their performance on these tasks.


Domain Adaptation of Recurrent Neural Networks for Natural Language Understanding

Abstract: The goal of this paper is to use multi-task learning to efficiently scale slot filling models for natural language understanding to handle multiple target tasks or domains. The key to scalability is reducing the amount of training data needed to learn a model for a new task. The proposed multi-task model delivers better performance with less data by leveraging patterns that it learns from the other tasks. The approach supports an open vocabulary, which allows the models to generalize to unseen words, which is particularly important when very little training data is used. A newly collected crowd-sourced data set, covering four different domains, is used to demonstrate the effectiveness of the domain adaptation and open vocabulary techniques.

Span-ConveRT: Few-shot Span Extraction for Dialog with Pretrained Conversational Representations

Abstract: We introduce Span-ConveRT, a light-weight model for dialog slot-filling which frames the task as a turn-based span extraction task. This formulation allows for a simple integration of conversational knowledge coded in large pretrained conversational models such as ConveRT (Henderson et al., 2019). We show that leveraging such knowledge in Span-ConveRT is especially useful for few-shot learning scenarios: we report consistent gains over 1) a span extractor that trains representations from scratch in the target domain, and 2) a BERT-based span extractor. In order to inspire more work on span extraction for the slot-filling task, we also release RESTAURANTS-8K, a new challenging data set of 8,198 utterances, compiled from actual conversations in the restaurant booking domain.

Cross-TOP: Zero-Shot Cross-Schema Task-Oriented Parsing

Abstract: t

Abstract: t

Abstract: t

Abstract: t

Summary of slot-filling datasets. All datasets listed in this table consist of English utterances.

This section surveys slot-filling datasets, which are summarized in Table {{table:91d2ef00-1017-4d10-a4fc-e0ccd628d38e}} .
These are typically datasets that consist of utterances with only slot-level annotations.


Large Scale Crowdsourcing and Characterization of Twitter Abusive Behavior

Abstract: In recent years online social networks have suffered an increase in sexism, racism, and other types of aggressive and cyberbullying behavior, often manifesting itself through offensive, abusive, or hateful language. Past scientific work focused on studying these forms of abusive activity in popular online social networks, such as Facebook and Twitter. Building on such work, we present an eight month study of the various forms of abusive behavior on Twitter, in a holistic fashion. Departing from past work, we examine a wide variety of labeling schemes, which cover different forms of abusive behavior. We propose an incremental and iterative methodology that leverages the power of crowdsourcing to annotate a large collection of tweets with a set of abuse-related labels. By applying our methodology and performing statistical analysis for label merging or elimination, we identify a reduced but robust set of labels to characterize abuse-related tweets. Finally, we offer a characterization of our annotated dataset of 80 thousand tweets, which we make publicly available for further scientific exploration.

Detecting and Monitoring Hate Speech in Twitter

Abstract: Social Media are sensors in the real world that can be used to measure the pulse of societies. However, the massive and unfiltered feed of messages posted in social media is a phenomenon that nowadays raises social alarms, especially when these messages contain hate speech targeted to a specific individual or group. In this context, governments and non-governmental organizations (NGOs) are concerned about the possible negative impact that these messages can have on individuals or on the society. In this paper, we present HaterNet, an intelligent system currently being used by the Spanish National Office Against Hate Crimes of the Spanish State Secretariat for Security that identifies and monitors the evolution of hate speech in Twitter. The contributions of this research are many-fold: (1) It introduces the first intelligent system that monitors and visualizes, using social network analysis techniques, hate speech in Social Media. (2) It introduces a novel public dataset on hate speech in Spanish consisting of 6000 expert-labeled tweets. (3) It compares several classification approaches based on different document representation strategies and text classification models. (4) The best approach consists of a combination of a LTSM+MLP neural network that takes as input the tweet’s word, emoji, and expression tokens’ embeddings enriched by the tf-idf, and obtains an area under the curve (AUC) of 0.828 on our dataset, outperforming previous methods presented in the literature.

Data sets details.

We study two widely used English abusive data sets (WASEEM and FOUNTA). We also study a Chinese data set (JIANG), a Spanish data set (PEREIRA), and an Italian data set (SANGUINETTI), in order to explore the impact of temporality on different languages. We choose these data sets because the creation time of each post is provided or accessible (via tweet IDs). Details of the data sets are shown in Table {{table:d57d9247-4a0e-49dd-937b-794a8fee69e4}} .


Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales

Abstract: We address the rating-inference problem, wherein rather than simply decide whether a review is "thumbs up" or "thumbs down", as in previous sentiment analysis work, one must determine an author's evaluation with respect to a multi-point scale (e.g., one to five "stars"). This task represents an interesting twist on standard multi-class text categorization because there are several different degrees of similarity between class labels; for example, "three stars" is intuitively closer to "four stars" than to "one star". We first evaluate human performance at the task. Then, we apply a meta-algorithm, based on a metric labeling formulation of the problem, that alters a given n-ary classifier's output in an explicit attempt to ensure that similar items receive similar labels. We show that the meta-algorithm can provide significant improvements over both multi-class and regression versions of SVMs when we employ a novel similarity measure appropriate to the problem.

A Sentimental Education: Sentiment Analysis Using Subjectivity Summarization Based on Minimum Cuts

Abstract: Sentiment analysis seeks to identify the viewpoint(s) underlying a text span; an example application is classifying a movie review as "thumbs up" or "thumbs down". To determine this sentiment polarity, we propose a novel machine-learning method that applies text-categorization techniques to just the subjective portions of the document. Extracting these portions can be implemented using efficient techniques for finding minimum cuts in graphs; this greatly facilitates incorporation of cross-sentence contextual constraints.

Sampled SentEval and IMDb datasets used for our evaluation.

Additionally, we have examined the performance of our method when the number of training examples is limited to 1,000 and 10,000 observations on sampled datasets according to Table {{table:c9655846-d60d-4937-b167-1fcd882f5fe3}} .


Representation Learning on Graphs with Jumping Knowledge Networks

Abstract: Recent deep learning approaches for representation learning on graphs follow a neighborhood aggregation procedure. We analyze some important properties of these models, and propose a strategy to overcome those. In particular, the range of "neighboring" nodes that a node's representation draws from strongly depends on the graph structure, analogous to the spread of a random walk. To adapt to local neighborhood properties and tasks, we explore an architecture -- jumping knowledge (JK) networks -- that flexibly leverages, for each node, different neighborhood ranges to enable better structure-aware representation. In a number of experiments on social, bioinformatics and citation networks, we demonstrate that our model achieves state-of-the-art performance. Furthermore, combining the JK framework with models like Graph Convolutional Networks, GraphSAGE and Graph Attention Networks consistently improves those models' performance.

Simple and Deep Graph Convolutional Networks

Abstract: Graph convolutional networks (GCNs) are a powerful deep learning approach for graph-structured data. Recently, GCNs and subsequent variants have shown superior performance in various application areas on real-world datasets. Despite their success, most of the current GCN models are shallow, due to the {\em over-smoothing} problem. In this paper, we study the problem of designing and analyzing deep graph convolutional networks. We propose the GCNII, an extension of the vanilla GCN model with two simple yet effective techniques: {\em Initial residual} and {\em Identity mapping}. We provide theoretical and empirical evidence that the two techniques effectively relieves the problem of over-smoothing. Our experiments show that the deep GCNII model outperforms the state-of-the-art methods on various semi- and full-supervised tasks. Code is available at this https URL .

Revisiting Graph Convolutional Network on Semi-Supervised Node Classification from an Optimization Perspective

Abstract: Graph convolutional networks (GCNs) have achieved promising performance on various graph-based tasks. However they suffer from over-smoothing when stacking more layers. In this paper, we present a quantitative study on this observation and develop novel insights towards the deeper GCN. First, we interpret the current graph convolutional operations from an optimization perspective and argue that over-smoothing is mainly caused by the naive first-order approximation of the solution to the optimization problem. Subsequently, we introduce two metrics to measure the over-smoothing on node-level tasks. Specifically, we calculate the fraction of the pairwise distance between connected and disconnected nodes to the overall distance respectively. Based on our theoretical and empirical analysis, we establish a universal theoretical framework of GCN from an optimization perspective and derive a novel convolutional kernel named GCN+ which has lower parameter amount while relieving the over-smoothing inherently. Extensive experiments on real-world datasets demonstrate the superior performance of GCN+ over state-of-the-art baseline methods on the node classification tasks.

Towards Deeper Graph Neural Networks

Abstract: Graph neural networks have shown significant success in the field of graph representation learning. Graph convolutions perform neighborhood aggregation and represent one of the most important graph operations. Nevertheless, one layer of these neighborhood aggregation methods only consider immediate neighbors, and the performance decreases when going deeper to enable larger receptive fields. Several recent studies attribute this performance deterioration to the over-smoothing issue, which states that repeated propagation makes node representations of different classes indistinguishable. In this work, we study this observation systematically and develop new insights towards deeper graph neural networks. First, we provide a systematical analysis on this issue and argue that the key factor compromising the performance significantly is the entanglement of representation transformation and propagation in current graph convolution operations. After decoupling these two operations, deeper graph neural networks can be used to learn graph node representations from larger receptive fields. We further provide a theoretical analysis of the above observation when building very deep models, which can serve as a rigorous and gentle description of the over-smoothing issue. Based on our theoretical and empirical analysis, we propose Deep Adaptive Graph Neural Network (DAGNN) to adaptively incorporate information from large receptive fields. A set of experiments on citation, co-authorship, and co-purchase datasets have confirmed our analysis and insights and demonstrated the superiority of our proposed methods.

Understanding and Resolving Performance Degradation in Deep Graph Convolutional Networks

Abstract: A Graph Convolutional Network (GCN) stacks several layers and in each layer performs a PROPagation operation~(PROP) and a TRANsformation operation~(TRAN) for learning node representations over graph-structured data. Though powerful, GCNs tend to suffer performance drop when the model gets deep. Previous works focus on PROPs to study and mitigate this issue, but the role of TRANs is barely investigated. In this work, we study performance degradation of GCNs by experimentally examining how stacking only TRANs or PROPs works. We find that TRANs contribute significantly, or even more than PROPs, to declining performance, and moreover that they tend to amplify node-wise feature variance in GCNs, causing variance inflammation that we identify as a key factor for causing performance drop. Motivated by such observations, we propose a variance-controlling technique termed Node Normalization (NodeNorm), which scales each node's features using its own standard deviation. Experimental results validate the effectiveness of NodeNorm on addressing performance degradation of GCNs. Specifically, it enables deep GCNs to outperform shallow ones in cases where deep models are needed, and to achieve comparable results with shallow ones on 6 benchmark datasets. NodeNorm is a generic plug-in and can well generalize to other GNN architectures. Code is publicly available at https://github.com/miafei/NodeNorm.

Masked Label Prediction: Unified Massage Passing Model for Semi-Supervised Classification

Abstract: Graph neural network (GNN) and label propagation algorithm (LPA) are both message passing algorithms, which have achieved superior performance in semi-supervised classification. GNN performs feature propagation by a neural network to make predictions, while LPA uses label propagation across graph adjacency matrix to get results. However, there is still no effective way to directly combine these two kinds of algorithms. To address this issue, we propose a novel Unified Message Passaging Model (UniMP) that can incorporate feature and label propagation at both training and inference time. First, UniMP adopts a Graph Transformer network, taking feature embedding and label embedding as input information for propagation. Second, to train the network without overfitting in self-loop input label information, UniMP introduces a masked label prediction strategy, in which some percentage of input label information are masked at random, and then predicted. UniMP conceptually unifies feature propagation and label propagation and is empirically powerful. It obtains new state-of-the-art semi-supervised classification results in Open Graph Benchmark (OGB).

Configurations adopted to implement approaches for training deep GNNs on Ogbn-ArXiv {{cite:ee14788}}.

Aiming to establish such a fair benchmark, our first step is to thoroughly investigate the design philosophy and implementation details on dozens of poplular deep GNN training techniques, including various residual connections, graph normalization, and random dropping. The summarization could be founded in Tables {{table:6f8400d1-d594-4f02-88f9-40ca9100b320}} , {{table:2a41a82d-5a98-4b73-8d77-82f36359d0ed}} , {{table:0ea2e031-28aa-4471-bf6a-197f3d6fa2fc}} , and {{table:4acce5ed-493d-43b0-b729-89a55f16605e}} . Somehow unfortunately, we find that even sticking to the same dataset and GNN backbone, the hyperparameter configurations (e.g., hidden dimension, learning rate, weight decay, dropout rate, training epochs, early stopping patience) are highly inconsistently implemented, often varying case-to-case, which make it troubling to draw any fair conclusion.

Prerequisite: Unifying the Hyperparameter Configuration

We carefully examine previous implementations of deep GNNs {{cite:2363f96}}, {{cite:dee7a72}}, {{cite:0812184}}, {{cite:f7be3b7}}, {{cite:f2ad407}}, {{cite:a43ce6a}}, {{cite:cae2619}}, {{cite:82e037d}}, {{cite:35b159a}}, {{cite:b088268}}, and list all their basic hyperparameters in Table {{table:6f8400d1-d594-4f02-88f9-40ca9100b320}} , {{table:2a41a82d-5a98-4b73-8d77-82f36359d0ed}} , {{table:0ea2e031-28aa-4471-bf6a-197f3d6fa2fc}} , and {{table:4acce5ed-493d-43b0-b729-89a55f16605e}} . Those hyperparameters play a significant role in those methods' achievable performance, but their inconsistency challenges fair comparison of training techniques, which has been traditionally somehow overlooked in literature.

Diverse hyperparameter configurations of existing deep GNNs.

As shown in Table {{table:2a41a82d-5a98-4b73-8d77-82f36359d0ed}} , {{table:0ea2e031-28aa-4471-bf6a-197f3d6fa2fc}} , and {{table:4acce5ed-493d-43b0-b729-89a55f16605e}} , we can see that these highly inconsistent hyperparameter settings pose severely challenges to fairly compare the existing training tricks for deep GNNs.


BioGRID: a general repository for interaction datasets

Abstract: Access to unified datasets of protein and genetic interactions is critical for interrogation of gene/protein function and analysis of global network properties. BioGRID is a freely accessible database of physical and genetic interactions available at . BioGRID release version 2.0 includes >116 000 interactions from Saccharomyces cerevisiae, Caenorhabditis elegans, Drosophila melanogaster and Homo sapiens. Over 30 000 interactions have recently been added from 5778 sources through exhaustive curation of the Saccharomyces cerevisiae primary literature. An internally hyper-linked web interface allows for rapid search and retrieval of interaction data. Full or user-defined datasets are freely downloadable as tab-delimited text files and PSI-MI XML. Pre-computed graphical layouts of interactions are available in a variety of file formats. User-customized graphs with embedded protein, gene and interaction attributes can be constructed with a visualization system called Osprey that is dynamically linked to the BioGRID.

Characterizing interactions in online social networks during exceptional events

Abstract: Nowadays, millions of people interact on a daily basis on online social media like Facebook and Twitter, where they share and discuss information about a wide variety of topics. In this paper, we focus on a specific online social network, Twitter, and we analyze multiple datasets each one consisting of individuals' online activity before, during and after an exceptional event in terms of volume of the communications registered. We consider important events that occurred in different arenas that range from policy to culture or science. For each dataset, the users' online activities are modeled by a multilayer network in which each layer conveys a different kind of interaction, specifically: retweeting, mentioning and replying. This representation allows us to unveil that these distinct types of interaction produce networks with different statistical properties, in particular concerning the degree distribution and the clustering structure. These results suggests that models of online activity cannot discard the information carried by this multilayer representation of the system, and should account for the different processes generated by the different kinds of interactions. Secondly, our analysis unveils the presence of statistical regularities among the different events, suggesting that the non-trivial topological patterns that we observe may represent universal features of the social dynamics on online social networks during exceptional events.

Combinatorial Analysis of Multiple Networks

Abstract: The study of complex networks has been historically based on simple graph data models representing relationships between individuals. However, often reality cannot be accurately captured by a flat graph model. This has led to the development of multi-layer networks. These models have the potential of becoming the reference tools in network data analysis, but require the parallel development of specific analysis methods explicitly exploiting the information hidden in-between the layers and the availability of a critical mass of reference data to experiment with the tools and investigate the real-world organization of these complex systems. In this work we introduce a real-world layered network combining different kinds of online and offline relationships, and present an innovative methodology and related analysis tools suggesting the existence of hidden motifs traversing and correlating different representation layers. We also introduce a notion of betweenness centrality for multiple networks. While some preliminary experimental evidence is reported, our hypotheses are still largely unverified, and in our opinion this calls for the availability of new analysis methods but also new reference multi-layer social network data.

Navigability of interconnected networks under random failures

Abstract: Significance Network theory has been exploited in the last decades to deepen our comprehension of complex systems. However, real-world complex systems exhibit multiple levels of relationships and require modeling by interconnected networks, characterizing interactions on several levels simultaneously. Questions such as “what is the efficiency of exploration of a city using the multiple transportation layers, like subway and bus?” and “what is its resilience to failures?” have to be answered using the multiplex framework. Here, we introduce fundamental mechanisms to perform such exploration, using random walks on multilayer networks, and we show how the topological structure, together with the navigation strategy, influences the efficiency in exploring the whole structure. Assessing the navigability of interconnected networks (transporting information, people, or goods) under eventual random failures is of utmost importance to design and protect critical infrastructures. Random walks are a good proxy to determine this navigability, specifically the coverage time of random walks, which is a measure of the dynamical functionality of the network. Here, we introduce the theoretical tools required to describe random walks in interconnected networks accounting for structure and dynamics inherent to real systems. We develop an analytical approach for the covering time of random walks in interconnected networks and compare it with extensive Monte Carlo simulations. Generally speaking, interconnected networks are more resilient to random failures than their individual layers per se, and we are able to quantify this effect. As an application––which we illustrate by considering the public transport of London––we show how the efficiency in exploring the multiplex critically depends on layers’ topology, interconnection strengths, and walk strategy. Our findings are corroborated by data-driven simulations, where the empirical distribution of check-ins and checks-out is considered and passengers travel along fastest paths in a network affected by real disruptions. These findings are fundamental for further development of searching and navigability strategies in real interconnected systems.

Real multiplex networks and their properties. The ID in the second column corresponds to Figure .

Multiplex network comparison across disciplines

We now consider real world static networks.
All considered networks are summarised with their main characteristics in Table  {{table:0723c308-b9f4-4f7c-9917-615727afc628}}The networks are hosted at https://comunelab.fbk.eu/data.php. Unfortunately, we have not found a real case with a large number of transition layer edges, so we limit this evaluation to elementary layer entanglement. For each network, we computed elementary layer homogeneity and intensity, for all connected components.


Describing Textures in the Wild

Abstract: Patterns and textures are key characteristics of many natural objects: a shirt can be striped, the wings of a butterfly can be veined, and the skin of an animal can be scaly. Aiming at supporting this dimension in image understanding, we address the problem of describing textures with semantic attributes. We identify a vocabulary of forty-seven texture terms and use them to describe a large dataset of patterns collected "in the wild". The resulting Describable Textures Dataset (DTD) is a basis to seek the best representation for recognizing describable texture attributes in images. We port from object recognition to texture recognition the Improved Fisher Vector (IFV) and Deep Convolutional-network Activation Features (DeCAF), and show that surprisingly, they both outperform specialized texture descriptors not only on our problem, but also in established material recognition datasets. We also show that our describable attributes are excellent texture descriptors, transferring between datasets and tasks, in particular, combined with IFV and DeCAF, they significantly outperform the state-of-the-art by more than 10% on both FMD and KTH-TIPS-2b benchmarks. We also demonstrate that they produce intuitive descriptions of materials and Internet images.

Progressive Growing of GANs for Improved Quality, Stability, and Variation

Abstract: We describe a new training methodology for generative adversarial networks. The key idea is to grow both the generator and discriminator progressively: starting from a low resolution, we add new layers that model increasingly fine details as training progresses. This both speeds the training up and greatly stabilizes it, allowing us to produce images of unprecedented quality, e.g., CelebA images at 1024^2. We also propose a simple way to increase the variation in generated images, and achieve a record inception score of 8.80 in unsupervised CIFAR10. Additionally, we describe several implementation details that are important for discouraging unhealthy competition between the generator and discriminator. Finally, we suggest a new metric for evaluating GAN results, both in terms of image quality and variation. As an additional contribution, we construct a higher-quality version of the CelebA dataset.

Training and test splits of four datasets.

Datasets  We conduct experiments on four datasets with different characteristics as below (details in Table {{table:94b72968-09b8-4271-8e9d-8af7cd81cd7d}} ):


Random Variable Learning Rate Back-propagation Neural Network (RVLRBPNN)

Multi-input and Multi-output neural network optimized by modified differential evolution algorithm

Quantum based evolutionary neural network optimized by Self-balanced adaptive differential evolution algorithm

Multiple related attributes are predicted concurrently

works on single solution, slow convergence

Higher complexity because of generation and processing of qubits

RVLBPNN: A Workload Forecasting Model for Smart Cloud Computing

Abstract: Given the increasing deployments of Cloud datacentres and the excessive usage of server resources, their associated energy and environmental implications are also increasing at an alarming rate. Cloud service providers are under immense pressure to significantly reduce both such implications for promoting green computing. Maintaining the desired level of Quality of Service (QoS) without violating the Service Level Agreement (SLA), whilst attempting to reduce the usage of the datacentre resources is an obvious challenge for the Cloud service providers. Scaling the level of active server resources in accordance with the predicted incoming workloads is one possible way of reducing the undesirable energy consumption of the active resources without affecting the performance quality. To this end, this paper analyzes the dynamic characteristics of the Cloud workloads and defines a hierarchy for the latency sensitivity levels of the Cloud workloads. Further, a novel workload prediction model for energy efficient Cloud Computing is proposed, named RVLBPNN (Rand Variable Learning Rate Backpropagation Neural Network) based on BPNN (Backpropagation Neural Network) algorithm. Experiments evaluating the prediction accuracy of the proposed prediction model demonstrate that RVLBPNN achieves an improved prediction accuracy compared to the HMM and Naive Bayes Classifier models by a considerable margin.

A proactive autoscaling and energy-efficient VM allocation framework using online multi-resource neural network for cloud data center

Abstract: This work proposes an energy-efficient resource provisioning and allocation framework to meet the dynamic demands of future applications. The frequent variations in a cloud user's resource demand lead 'to the problem of excess power consumption, resource wastage, performance, and Quality-of-Service degradation. The proposed framework addresses these challenges by matching the application's predicted resource requirement with the resource capacity of VMs precisely and thereby consolidating the entire load on the minimum number of energy-efficient physical machines. The three consecutive contributions of the proposed work are: Online Multi-Resource Feed-forward Neural Network to forecast the multiple resource demands concurrently for future applications; autoscaling of VMs based on the clustering of the predicted resource requirements; allocation of the scaled VMs on the energy-efficient PMs. The integrated approach successively optimizes resource utilization, saves energy and automatically adapts to the changes in future application resource demand. The proposed framework is evaluated by using real workload traces of the benchmark Google Cluster Dataset and compared against different scenarios including energy-efficient VM placement with resource prediction only, VMP without resource prediction and autoscaling, and optimal VMP with autoscaling based on actual resource utilization. The observed results demonstrate that the proposed integrated approach achieves near-optimal performance against optimal VMP and outperforms rest of the VMPs in terms of power saving and resource utilization up to 88.5% and 21.12% respectively. In addition, the OM-FNN predictor shows better accuracy, lesser time and space complexity over a traditional single-input and single-output feed-forward neural network predictor.

A Quantum Approach Towards the Adaptive Prediction of Cloud Workloads

Abstract: This work presents a novel Evolutionary Quantum Neural Network (EQNN) based workload prediction model for Cloud datacenter. It exploits the computational efficiency of quantum computing by encoding workload information into qubits and propagating this information through the network to estimate the workload or resource demands with enhanced accuracy proactively. The rotation and reverse rotation effects of the Controlled-NOT (C-NOT) gate serve activation function at the hidden and output layers to adjust the qubit weights. In addition, a Self Balanced Adaptive Differential Evolution (SB-ADE) algorithm is developed to optimize qubit network weights. The accuracy of the EQNN prediction model is extensively evaluated and compared with seven state-of-the-art methods using eight real world benchmark datasets of three different categories. Experimental results reveal that the use of the quantum approach to evolutionary neural network substantially improves the prediction accuracy up to 91.6% over the existing approaches.

Summary of Machine Learning based Workload Prediction

Table {{table:e375b774-aa5f-4776-8f7f-2d10a49718b6}}  entails the summary of aforementioned machine learning based workload prediction approaches.


Wide Residual Networks

Abstract: Deep residual networks were shown to be able to scale up to thousands of layers and still have improving performance. However, each fraction of a percent of improved accuracy costs nearly doubling the number of layers, and so training very deep residual networks has a problem of diminishing feature reuse, which makes these networks very slow to train. To tackle these problems, in this paper we conduct a detailed experimental study on the architecture of ResNet blocks, based on which we propose a novel architecture where we decrease depth and increase width of residual networks. We call the resulting network structures wide residual networks (WRNs) and show that these are far superior over their commonly used thin and very deep counterparts. For example, we demonstrate that even a simple 16-layer-deep wide residual network outperforms in accuracy and efficiency all previous deep residual networks, including thousand-layer-deep networks, achieving new state-of-the-art results on CIFAR, SVHN, COCO, and significant improvements on ImageNet. Our code and models are available at this https URL

RoBERTa: A Robustly Optimized BERT Pretraining Approach

Abstract: Language model pretraining has led to significant performance gains but careful comparison between different approaches is challenging. Training is computationally expensive, often done on private datasets of different sizes, and, as we will show, hyperparameter choices have significant impact on the final results. We present a replication study of BERT pretraining (Devlin et al., 2019) that carefully measures the impact of many key hyperparameters and training data size. We find that BERT was significantly undertrained, and can match or exceed the performance of every model published after it. Our best model achieves state-of-the-art results on GLUE, RACE and SQuAD. These results highlight the importance of previously overlooked design choices, and raise questions about the source of recently reported improvements. We release our models and code.

CamemBERT: a Tasty French Language Model

Abstract: Pretrained language models are now ubiquitous in Natural Language Processing. Despite their success, most available models have either been trained on English data or on the concatenation of data in multiple languages. This makes practical use of such models –in all languages except English– very limited. In this paper, we investigate the feasibility of training monolingual Transformer-based language models for other languages, taking French as an example and evaluating our language models on part-of-speech tagging, dependency parsing, named entity recognition and natural language inference tasks. We show that the use of web crawled data is preferable to the use of Wikipedia data. More surprisingly, we show that a relatively small web crawled dataset (4GB) leads to results that are as good as those obtained using larger datasets (130+GB). Our best performing model CamemBERT reaches or improves the state of the art in all four downstream tasks.

Cross-lingual Language Model Pretraining

Abstract: Recent studies have demonstrated the efficiency of generative pretraining for English natural language understanding. In this work, we extend this approach to multiple languages and show the effectiveness of cross-lingual pretraining. We propose two methods to learn cross-lingual language models (XLMs): one unsupervised that only relies on monolingual data, and one supervised that leverages parallel data with a new cross-lingual language model objective. We obtain state-of-the-art results on cross-lingual classification, unsupervised and supervised machine translation. On XNLI, our approach pushes the state of the art by an absolute gain of 4.9% accuracy. On unsupervised machine translation, we obtain 34.3 BLEU on WMT’16 German-English, improving the previous state of the art by more than 9 BLEU. On supervised machine translation, we obtain a new state of the art of 38.5 BLEU on WMT’16 Romanian-English, outperforming the previous best approach by more than 4 BLEU. Our code and pretrained models will be made publicly available.

Deep Learning Recommendation Model for Personalization and Recommendation Systems

Abstract: With the advent of deep learning, neural network-based recommendation models have emerged as an important tool for tackling personalization and recommendation tasks. These networks differ significantly from other deep learning networks due to their need to handle categorical features and are not well studied or understood. In this paper, we develop a state-of-the-art deep learning recommendation model (DLRM) and provide its implementation in both PyTorch and Caffe2 frameworks. In addition, we design a specialized parallelization scheme utilizing model parallelism on the embedding tables to mitigate memory constraints while exploiting data parallelism to scale-out compute from the fully-connected layers. We compare DLRM against existing recommendation models and characterize its performance on the Big Basin AI platform, demonstrating its usefulness as a benchmark for future algorithmic experimentation and system co-design.

DNN models used in our experiments.

DNN Workloads. We experiment with 13 popular DNN models: VGG11 {{cite:5c8a8ae}}, VGG16 {{cite:8dc6a8e}}, VGG19 {{cite:87bf6b4}}, ResNet50 {{cite:93c6649}}, WideResNet101 {{cite:acbb1c3}}, BERT {{cite:944c11c}}, RoBERTa {{cite:6efc83a}}, XLM {{cite:4ee6d5f}}, CamemBERT {{cite:06445a3}}, GPT-1 {{cite:b24d55a}}, GPT-2 {{cite:e915d9c}}, GPT-3 {{cite:ecc4466}}, and DLRM {{cite:7caffcd}}. All Models have an equal probability of occurrence and the training duration time is randomly selected between 200 - 1,000 iterations. Table {{table:98f6f443-03a5-48ac-a4c8-96b84ed83c7b}}  (Appendix REF ) provides details about model configurations and batch sizes used in this paper.

As mentioned in Section REF , we run our experiment with 13 popular DNN models: VGG11 {{cite:5c8a8ae}}, VGG16 {{cite:8dc6a8e}}, VGG19 {{cite:87bf6b4}}, ResNet50 {{cite:93c6649}}, WideResNet101 {{cite:acbb1c3}}, BERT {{cite:944c11c}}, RoBERTa {{cite:6efc83a}}, XLM {{cite:4ee6d5f}}, CamemBERT {{cite:06445a3}}, GPT1 {{cite:b24d55a}}, GPT2 {{cite:e915d9c}}, GPT3 {{cite:ecc4466}}, and DLRM {{cite:7caffcd}}. Table {{table:98f6f443-03a5-48ac-a4c8-96b84ed83c7b}}  summarizes the parameters of each model and batch sizes. Note that the batch sizes are provided as a range because the number of workers and hyper-parameters change during scheduling epochs. In particular, in different experiments, we select the batch size according to the hyper-parameters used in prior work {{cite:85921c7}}, {{cite:650215e}}, {{cite:363250a}}, {{cite:3ee8125}}, {{cite:773affd}}. The memory requirements of each model reflects the amount of memory each model occupies in the GPU memory. We adjust the model sizes for different models depending on the parallelization strategy.


FaceNet: A unified embedding for face recognition and clustering

Abstract: Despite significant recent advances in the field of face recognition [10, 14, 15, 17], implementing face verification and recognition efficiently at scale presents serious challenges to current approaches. In this paper we present a system, called FaceNet, that directly learns a mapping from face images to a compact Euclidean space where distances directly correspond to a measure offace similarity. Once this space has been produced, tasks such as face recognition, verification and clustering can be easily implemented using standard techniques with FaceNet embeddings asfeature vectors. Our method uses a deep convolutional network trained to directly optimize the embedding itself, rather than an intermediate bottleneck layer as in previous deep learning approaches. To train, we use triplets of roughly aligned matching / non-matching face patches generated using a novel online triplet mining method. The benefit of our approach is much greater representational efficiency: we achieve state-of-the-artface recognition performance using only 128-bytes perface. On the widely used Labeled Faces in the Wild (LFW) dataset, our system achieves a new record accuracy of 99.63%. On YouTube Faces DB it achieves 95.12%. Our system cuts the error rate in comparison to the best published result [15] by 30% on both datasets.

Web-scale training for face identification

Abstract: Scaling machine learning methods to very large datasets has attracted considerable attention in recent years, thanks to easy access to ubiquitous sensing and data from the web. We study face recognition and show that three distinct properties have surprising effects on the transferability of deep convolutional networks (CNN): (1) The bottleneck of the network serves as an important transfer learning regularizer, and (2) in contrast to the common wisdom, performance saturation may exist in CNN's (as the number of training samples grows); we propose a solution for alleviating this by replacing the naive random subsampling of the training set with a bootstrapping process. Moreover, (3) we find a link between the representation norm and the ability to discriminate in a target domain, which sheds lights on how such networks represent faces. Based on these discoveries, we are able to improve face recognition accuracy on the widely used LFW benchmark, both in the verification (1:1) and identification (1:N) protocols, and directly compare, for the first time, with the state of the art Commercially-Off-The-Shelf system and show a sizable leap in performance.

Level Playing Field for Million Scale Face Recognition

Abstract: Face recognition has the perception of a solved problem, however when tested at the million-scale exhibits dramatic variation in accuracies across the different algorithms [11]. Are the algorithms very different? Is access to good/big training data their secret weapon? Where should face recognition improve? To address those questions, we created a benchmark, MF2, that requires all algorithms to be trained on same data, and tested at the million scale. MF2 is a public large-scale set with 672K identities and 4.7M photos created with the goal to level playing field for large scale face recognition. We contrast our results with findings from the other two large-scale benchmarks MegaFace Challenge and MS-Celebs-1M where groups were allowed to train on any private/public/big/small set. Some key discoveries: 1) algorithms, trained on MF2, were able to achieve state of the art and comparable results to algorithms trained on massive private sets, 2) some outperformed themselves once trained on MF2, 3) invariance to aging suffers from low accuracies as in MegaFace, identifying the need for larger age variations possibly within identities or adjustment of algorithms in future testing.

UMDFaces: An annotated face dataset for training deep networks

Abstract: Recent progress in face detection (including keypoint detection), and recognition is mainly being driven by (i) deeper convolutional neural network architectures, and (ii) larger datasets. However, most of the large datasets are maintained by private companies and are not publicly available. The academic computer vision community needs larger and more varied datasets to make further progress. In this paper, we introduce a new face dataset, called UMDFaces, which has 367,888 annotated faces of 8,277 subjects. We also introduce a new face recognition evaluation protocol which will help advance the state-of-the-art in this area. We discuss how a large dataset can be collected and annotated using human annotators and deep networks. We provide human curated bounding boxes for faces. We also provide estimated pose (roll, pitch and yaw), locations of twenty-one key-points and gender information generated by a pre-trained neural network. In addition, the quality of keypoint annotations has been verified by humans for about 115,000 images. Finally, we compare the quality of the dataset with other publicly available face datasets at similar scales.

VGGFace2: A Dataset for Recognising Faces across Pose and Age

Abstract: In this paper, we introduce a new large-scale face dataset named VGGFace2. The dataset contains 3.31 million images of 9131 subjects, with an average of 362.6 images for each subject. Images are downloaded from Google Image Search and have large variations in pose, age, illumination, ethnicity and profession (e.g. actors, athletes, politicians). The dataset was collected with three goals in mind: (i) to have both a large number of identities and also a large number of images for each identity; (ii) to cover a large range of pose, age and ethnicity; and (iii) to minimise the label noise. We describe how the dataset was collected, in particular the automated and manual filtering stages to ensure a high accuracy for the images of each identity. To assess face recognition performance using the new dataset, we train ResNet-50 (with and without Squeeze-and-Excitation blocks) Convolutional Neural Networks on VGGFace2, on MS-Celeb-1M, and on their union, and show that training on VGGFace2 leads to improved recognition performance over pose and age. Finally, using the models trained on these datasets, we demonstrate state-of-the-art performance on the IJB-A and IJB-B face recognition benchmarks, exceeding the previous state-of-the-art by a large margin. The dataset and models are publicly available.

WebFace260M: A Benchmark Unveiling the Power of Million-Scale Deep Face Recognition

Abstract: In this paper, we contribute a new million-scale face benchmark containing noisy 4M identities/260M faces (WebFace260M) and cleaned 2M identities/42M faces (WebFace42M) training data, as well as an elaborately designed time-constrained evaluation protocol. Firstly, we collect 4M name list and download 260M faces from the Internet. Then, a Cleaning Automatically utilizing Self-Training (CAST) pipeline is devised to purify the tremendous WebFace260M, which is efficient and scalable. To the best of our knowledge, the cleaned WebFace42M is the largest public face recognition training set and we expect to close the data gap between academia and industry. Referring to practical scenarios, Face Recognition Under Inference Time conStraint (FRUITS) protocol and a test set are constructed to comprehensively evaluate face matchers.Equipped with this benchmark, we delve into million-scale face recognition problems. A distributed framework is developed to train face recognition models efficiently without tampering with the performance. Empowered by Web-Face42M, we reduce relative 40% failure rate on the challenging IJB-C set, and rank the 3rd among 430 entries on NIST-FRVT. Even 10% data (WebFace4M) shows superior performance compared with public training set. Furthermore, comprehensive baselines are established on our rich-attribute test set under FRUITS-100ms/500ms/1000ms protocol, including MobileNet, EfficientNet, AttentionNet, ResNet, SENet, ResNeXt and RegNet families. Benchmark website is https://www.face-benchmark.org.

SynFace: Face Recognition with Synthetic Data

Abstract: With the recent success of deep neural networks, remarkable progress has been achieved on face recognition. However, collecting large-scale real-world training data for face recognition has turned out to be challenging, especially due to the label noise and privacy issues. Meanwhile, existing face recognition datasets are usually collected from web images, lacking detailed annotations on attributes (e.g., pose and expression), so the influences of different attributes on face recognition have been poorly investigated. In this paper, we address the above-mentioned issues in face recognition using synthetic face images, i.e., SynFace. Specifically, we first explore the performance gap between recent state-of-the-art face recognition models trained with synthetic and real face images. We then analyze the underlying causes behind the performance gap, e.g., the poor intraclass variations and the domain gap between synthetic and real face images. Inspired by this, we devise the SynFace with identity mixup (IM) and domain mixup (DM) to mitigate the above performance gap, demonstrating the great potentials of synthetic data for face recognition. Furthermore, with the controllable face synthesis model, we can easily manage different factors of synthetic face generation, including pose, expression, illumination, the number of identities, and samples per identity. Therefore, we also perform a systematically empirical analysis on synthetic face images to provide some insights on how to effectively utilize synthetic data for face recognition.

DigiFace-1M: 1 Million Digital Face Images for Face Recognition

Abstract: State-of-the-art face recognition models show impressive accuracy, achieving over 99.8% on Labeled Faces in the Wild (LFW) dataset. Such models are trained on large-scale datasets that contain millions of real human face images collected from the internet. Web-crawled face images are severely biased (in terms of race, lighting, makeup, etc) and often contain label noise. More importantly, the face images are collected without explicit consent, raising ethical concerns. To avoid such problems, we introduce a large-scale synthetic dataset for face recognition, obtained by rendering digital faces using a computer graphics pipeline1. We first demonstrate that aggressive data augmentation can significantly reduce the synthetic-to-real domain gap. Having full control over the rendering pipeline, we also study how each attribute (e.g., variation in facial pose, accessories and textures) affects the accuracy. Compared to Syn-Face, a recent method trained on GAN-generated synthetic faces, we reduce the error rate on LFW by 52.5% (accuracy from 91.93% to 96.17%). By fine-tuning the network on a smaller number of real face images that could reason-ably be obtained with consent, we achieve accuracy that is comparable to the methods trained on millions of real face images.

SFace: Privacy-friendly and Accurate Face Recognition using Synthetic Data

Abstract: Recent deep face recognition models proposed in the literature utilized large-scale public datasets such as MS-Celeb-1M and VGGFace2 for training very deep neural networks, achieving state-of-the-art performance on mainstream benchmarks. Recently, many of these datasets, e.g., MS-Celeb-1M and VGGFace2, are retracted due to credible privacy and ethical concerns. This motivates this work to propose and investigate the feasibility of using a privacy-friendly synthetically generated face dataset to train face recognition models. Towards this end, we utilize a class-conditional generative adversarial network to generate class-labeled synthetic face images, namely SFace. To address the privacy aspect of using such data to train a face recognition model, we provide extensive evaluation experiments on the identity relation between the synthetic dataset and the original authentic dataset used to train the generative model. Our reported evaluation proved that associating an identity of the authentic dataset to one with the same class label in the synthetic dataset is hardly possible. We also propose to train face recognition on our privacy-friendly dataset, SFace, using three different learning strategies, multi-class classification, label-free knowledge transfer, and combined learning of multi-class classification and knowledge transfer. The reported evaluation results on five authentic face benchmarks demonstrated that the privacy-friendly synthetic dataset has a high potential to be used for training face recognition models, achieving, for example, a verification accuracy of 91.87% on LFW using multi-class classification and 99.13% using the combined learning strategy. The training code and the synthetic face image dataset are publicly released11https://github.com/fdbtrs/SFace-Privacy-friendly-and-Accurate-Face-Recognition-using-Synthetic-Data.

Unsupervised Face Recognition using Unlabeled Synthetic Data

Abstract: Over the past years, the main research innovations in face recognition focused on training deep neural networks on large-scale identity-labeled datasets using variations of multi-class classification losses. However, many of these datasets are retreated by their creators due to increased privacy and ethical concerns. Very recently, privacy-friendly synthetic data has been proposed as an alternative to privacy-sensitive authentic data to comply with privacy regulations and to ensure the continuity of face recognition research. In this paper, we propose an unsupervised face recognition model based on unlabeled synthetic data (USynthFace). Our proposed USynthFace learns to maximize the similarity between two augmented images of the same synthetic instance. We enable this by a large set of geometric and color transformations in addition to GAN-based augmentation that contributes to the USynthFace model training. We also conduct numerous empirical studies on different components of our USynthFace. With the proposed set of augmentation operations, we proved the effectiveness of our USynthFace in achieving relatively high recognition accuracies using unlabeled synthetic data. The training code and pretrained model are publicly available under https://github.com/fdbtrs/Unsupervised-Face-Recognition-using-Unlabeled-Synthetic-Data.

Overview of the most widely used authentic and synthetic facial datasets commonly used to train FR models, along with the number of images, identities, images per identity, and the fact that each database is public and/or still accessible. Note that many of the public databases are not accessible (raising a practical problem for researchers and developers) anymore based on legal and ethical concerns and even those that are available are ethically questioned as the individual consent of the data subjects is not always insured.

The breakthroughs of deep neural networks and their training optimizations as well as the availability of large-scale identity-labeled face datasets have reshaped the research landscape of face recognition (FR) over the past years. These emerging technologies have dramatically improved FR performances leading to the wider integration of FR in a variety of applications from logical access control and consumer low-end devices to automated border control.
State-of-the-Art (SOTA) FR models {{cite:5e6b2e8}}, {{cite:0e8c82a}} utilized large-scale face datasets e.g. CASIA-WebFace {{cite:99d6d19}}, MS-Celeb-1M {{cite:b1f425f}}, or VGGFace2 {{cite:27fd80a}} to train deep neural networks (DNN) with millions of trainable parameters, where the goal is to optimize the empirical risk minimization function given input training samples, their corresponding labels, and DNN trainable parameters. Achieving such a goal without being over-optimized, i.e. overfitted, requires that training datasets are of large scale (massive number of images of many identities) and representative of various variations that exist in the real world.
Large and representative data is also required to evaluate FR accuracies against different variations that present in real operation scenarios e.g. pose, aging, occlusion, or lighting. Data is required to evaluate the vulnerability of FR against different types of attacks such as morphing, presentation, master-face, and deep fake attacks.
FR components, face processing models, attack detectors, and face image quality estimation models are not different as they require face data for training and evaluation.
Besides the technical limitation of collecting large-scale data with realistic variations, there are increased concerns about collecting, maintaining, redistributing, and using biometric data due to legal, ethical, and privacy concerns {{cite:b5457af}}. Consequently, many widely used datasets for FR development such as VGGFace2 {{cite:27fd80a}} and MS-Celeb-1M {{cite:b1f425f}} have been retracted by their creator. Table {{table:c149ce0b-8bcb-4ce6-b32c-1cfe239a0f29}}  summarizes the most widely used datasets to train FR models. Even though many of these datasets have been publically released, there are not any more accessible.