Lower overloading probability and job latency

Optimal placement considering QoS & running cost

Avoid pre-training/retraining and consider unbalanced scenarios

Avoid pre-training/retraining and consider multiple applications

Managing Fog Networks using Reinforcement Learning Based Load Balancing Algorithm

Abstract: The powerful paradigm of Fog computing is currently receiving major interest, as it provides the possibility to integrate virtualized servers into networks and brings cloud service closer to end devices. To support this distributed intelligent platform, Software-Defined Network (SDN) has emerged as a viable network technology in the Fog computing environment. However, uncertainties related to task demands and the different computing capacities of Fog nodes, inquire an effective load balancing algorithm. In this paper, the load balancing problem has been addressed under the constraint of achieving the minimum latency in Fog networks. To handle this problem, a reinforcement learning based decision-making process has been proposed to find the optimal offloading decision with unknown reward and transition functions. The proposed process allows Fog nodes to offload an optimal number of tasks among incoming tasks by selecting an available neighboring Fog node under their respective resource capabilities with the aim to minimize the processing time and the overall overloading probability. Compared with the traditional approaches, the proposed scheme not only simplifies the algorithmic framework without imposing any specific assumption on the network model but also guarantees convergence in polynomial time. The results show that, during average delays, the proposed reinforcement learning-based offloading method achieves significant performance improvements over the variation of service rate and traffic arrival rate. The proposed algorithm achieves 1.17%, 1.02%, and 3.21% lower overload probability relative to random, least-queue and nearest offloading selection schemes, respectively.

Context-aware Distribution of Fog Applications Using Deep Reinforcement Learning

Abstract: Fog computing is an emerging paradigm that aims to meet the increasing computation demands arising from the billions of devices connected to the Internet. Offloading services of an application from the Cloud to the edge of the network can improve the overall Quality-of-Service (QoS) of the application since it can process data closer to user devices. Diverse Fog nodes ranging from Wi-Fi routers to mini-clouds with varying resource capabilities makes it challenging to determine which services of an application need to be offloaded. In this paper, a context-aware mechanism for distributing applications across the Cloud and the Fog is proposed. The mechanism dynamically generates (re)deployment plans for the application to maximise the performance efficiency of the application by taking the QoS and running costs into account. The mechanism relies on deep Q-networks to generate a distribution plan without prior knowledge of the available resources on the Fog node, the network condition and the application. The feasibility of the proposed context-aware distribution mechanism is demonstrated on two use-cases, namely a face detection application and a location-based mobile game. The benefits are increased utility of dynamic distribution in both use cases, when compared to a static distribution approach used in existing research.

Comparison between existing load balancing approaches for Fog Networks

There are many limitations in existing load balancing approaches in the literature (summarized in Table {{table:bc4d01fb-6b0d-479a-b0c2-d847d4f4b437}} ). That motivated us to propose a solution that mitigates these limitations. Our proposed load balancing solution optimizes multiple objectives simultaneously using an MCDA-based method to improve the overall system performance. MCDA-based methods were used to make resource selection decisions in different domains, including Mobile Crowd Computing systems, where smart mobile devices are utilized as computing resources {{cite:059df57}}. These methods achieved satisfactory performance and provided quality of service by selecting the most suitable resources. Furthermore, MCDA methods were also used to efficiently solve the service selection problem in Cloud Computing environments {{cite:13807dc}}.

The varying significance of the selection criteria makes MCDA methods suitable for load balancing problems in the Fog. ELECTRE, for example, is an effective outranking MCDA method that was used to solve the Cloud service selection {{cite:13807dc}} and the Fog service placement {{cite:bef946f}} problems. However, to the best of our knowledge, MCDA-based methods have never been used to balance the load in Fog environments using service selection decisions of stateless micro Fog service replicas. To fill this gap, we propose in this work an MCDA-based load balancing solution using the ELECTRE algorithm, where we simultaneously minimize the hop count, propagation delay, processing delay, execution delay, and waiting delay to improve the overall system performance. We compare the performance of our approach against traditional service selection methods, i.e., random, RR, nearest node, and fastest service selection. Evaluating load balancing algorithms against such baselines is a common practice in the literature (see Table {{table:bc4d01fb-6b0d-479a-b0c2-d847d4f4b437}} ). To advance the state of the art, the proposed solutions must mitigate the limitations of existing approaches in the literature before improving the system performance in terms of execution delay and resource utilization.

Table {{table:bc4d01fb-6b0d-479a-b0c2-d847d4f4b437}}  shows that our proposed solution addresses a number of limitations in existing work in the literature. For example, our solution provides a near-optimal solution for the load balancing problem in Fog environments. Unlike randomized methods (e.g., {{cite:ea27f6b}}{{cite:e469a30}}{{cite:d14420f}}{{cite:df8e523}}{{cite:6f3b6f8}}), which do not guarantee optimal load distribution, and can unnecessarily waste compute and network resources. In addition, our proposed approach can easily adapt to dynamic changes in the topology of the system, availability of compute and network resources, as well as the changes in load distribution and its generation rate. In contrast to static allocation solutions (e.g., {{cite:3280f3f}}{{cite:53ba3e2}}), with initial service placement of Fog services, which can only perform well in unrealistic static environments that never change overtime. Our proposed solution also avoids predefined priorities and thresholds that are common in existing solutions (e.g., {{cite:bd9b964}}), which are hard to be estimated even by experts. Avoiding such predetermined parameters allows for dynamic load balancing solutions that can easily adjust to system changes.


Appearance prior (spatial distribution, size, saliency)

Mapping relationship between the box overlap and appearance similarity

Bayesian Joint Topic Modelling for Weakly Supervised Object Localisation

Abstract: We address the problem of localisation of objects as bounding boxes in images with weak labels. This weakly supervised object localisation problem has been tackled in the past using discriminative models where each object class is localised independently from other classes. We propose a novel framework based on Bayesian joint topic modelling. Our framework has three distinctive advantages over previous works: (1) All object classes and image backgrounds are modelled jointly together in a single generative model so that "explaining away" inference can resolve ambiguity and lead to better learning and localisation. (2) The Bayesian formulation of the model enables easy integration of prior knowledge about object appearance to compensate for limited supervision. (3) Our model can be learned with a mixture of weakly labelled and unlabelled data, allowing the large volume of unlabelled images on the Internet to be exploited for learning. Extensive experiments on the challenging VOC dataset demonstrate that our approach outperforms the state-of-the-art competitors.

Transfer Learning by Ranking for Weakly Supervised Object Annotation

Abstract: Most existing approaches to training object detectors rely on fully supervised learning, which requires the tedious manual annotation of object location in a training set. Recently there has been an increasing interest in developing weakly supervised approach to detector training where the object location is not manually annotated but automatically determined based on binary (weak) labels indicating if a training image contains the object. This is a challenging problem because each image can contain many candidate object locations which partially overlaps the object of interest. Existing approaches focus on how to best utilise the binary labels for object location annotation. In this paper we propose to solve this problem from a very different perspective by casting it as a transfer learning problem. Specifically, we formulate a novel transfer learning based on learning to rank, which effectively transfers a model for automatic annotation of object location from an auxiliary dataset to a target dataset with completely unrelated object categories. We show that our approach outperforms existing state-of-the-art weakly supervised approach to annotating objects in the challenging VOC dataset.

Summary of the approaches for initialization, which is a subcategory in the weakly supervised object localization and detection approaches that learn by classic models. An approach is considered for general object category when it is tested for detecting more than five object categories in the corresponding literature. The approaches with None detector indicate the weakly supervised object localization approaches.

Numerous methods have been developed to mine reliable instance samples, using prior knowledge, as weak supervision for the following processes.
A brief summary of these approaches are shown in Table {{table:66cd985b-f584-4b2b-a9e0-a506c5330fb8}} .


Open Graph Benchmark: Datasets for Machine Learning on Graphs

Abstract: We present the Open Graph Benchmark (OGB), a diverse set of challenging and realistic benchmark datasets to facilitate scalable, robust, and reproducible graph machine learning (ML) research. OGB datasets are large-scale (up to 100+ million nodes and 1+ billion edges), encompass multiple important graph ML tasks, and cover a diverse range of domains, ranging from social and information networks to biological networks, molecular graphs, source code ASTs, and knowledge graphs. For each dataset, we provide a unified evaluation protocol using meaningful application-specific data splits and evaluation metrics. In addition to building the datasets, we also perform extensive benchmark experiments for each dataset. Our experiments suggest that OGB datasets present significant challenges of scalability to large-scale graphs and out-of-distribution generalization under realistic data splits, indicating fruitful opportunities for future research. Finally, OGB provides an automated end-to-end graph ML pipeline that simplifies and standardizes the process of graph data loading, experimental setup, and model evaluation. OGB will be regularly updated and welcomes inputs from the community. OGB datasets as well as data loaders, evaluation scripts, baseline code, and leaderboards are publicly available at this https URL .

struc2vec: Learning Node Representations from Structural Identity

Abstract: Structural identity is a concept of symmetry in which network nodes are identified according to the network structure and their relationship to other nodes. Structural identity has been studied in theory and practice over the past decades, but only recently has it been addressed with representational learning techniques. This work presents struc2vec, a novel and flexible framework for learning latent representations for the structural identity of nodes. struc2vec uses a hierarchy to measure node similarity at different scales, and constructs a multilayer graph to encode structural similarities and generate structural context for nodes. Numerical experiments indicate that state-of-the-art techniques for learning node representations fail in capturing stronger notions of structural identity, while struc2vec exhibits much superior performance in this task, as it overcomes limitations of prior approaches. As a consequence, numerical experiments indicate that struc2vec improves performance on classification tasks that depend more on structural identity.

Statistics of the datasets used for link prediction experiments. All numbers are counted in the training dataset without valid and test edges.

Data.
We evaluate our framework on four real-world datasets from multiple domains. Ogbl-Collab and Ogbl-Citation2 {{cite:e00465e}} are the author collaboration and citation networks.
USAir {{cite:1982559}} is a network of US Air lines.
We notice that these datasets are relatively dense, with average node degree of 8-20.
In real-world deployed systems, the observed data is typically more sparse, with many low-degree nodes.
Hence, we adapt a sparse graph dataset from E-commerce {{cite:7f31a25}}.
E-commerce is a real-world dataset constructed according to shopping queries and related products in Amazon Search, which captures whether the queries are exact matches of the products.
We give the details of dataset construction in Appendix  REF . Detailed statistics of all datasets are shown in Tab. {{table:b13149ba-738b-4f83-a9d0-cc8d23baef56}}  .


Gait Recognition via Disentangled Representation Learning

Abstract: Gait, the walking pattern of individuals, is one of the most important biometrics modalities. Most of the existing gait recognition methods take silhouettes or articulated body models as the gait features. These methods suffer from degraded recognition performance when handling confounding variables, such as clothing, carrying and view angle. To remedy this issue, we propose a novel AutoEncoder framework to explicitly disentangle pose and appearance features from RGB imagery and the LSTM-based integration of pose features over time produces the gait feature. In addition, we collect a Frontal-View Gait (FVG) dataset to focus on gait recognition from frontal-view walking, which is a challenging problem since it contains minimal gait cues compared to other views. FVG also includes other important variations, e.g., walking speed, carrying, and clothing. With extensive experiments on CASIA-B, USF and FVG datasets, our method demonstrates superior performance to the state of the arts quantitatively, the ability of feature disentanglement qualitatively, and promising computational efficiency.

GaitSet: Regarding Gait as a Set for Cross-View Gait Recognition

Abstract: As a unique biometric feature that can be recognized at a distance, gait has broad applications in crime prevention, forensic identification and social security. To portray a gait, existing gait recognition methods utilize either a gait template, where temporal information is hard to preserve, or a gait sequence, which must keep unnecessary sequential constraints and thus loses the flexibility of gait recognition. In this paper we present a novel perspective, where a gait is regarded as a set consisting of independent frames. We propose a new network named GaitSet to learn identity information from the set. Based on the set perspective, our method is immune to permutation of frames, and can naturally integrate frames from different videos which have been filmed under different scenarios, such as diverse viewing angles, different clothes/carrying conditions. Experiments show that under normal walking conditions, our single-model method achieves an average rank-1 accuracy of 95.0% on the CASIA-B gait dataset and an 87.1% accuracy on the OU-MVLP gait dataset. These results represent new state-of-the-art recognition accuracy. On various complex scenarios, our model exhibits a significant level of robustness. It achieves accuracies of 87.2% and 70.4% on CASIA-B under bag-carrying and coat-wearing walking conditions, respectively. These outperform the existing best methods by a large margin. The method presented can also achieve a satisfactory accuracy with a small number of frames in a test sample, e.g., 82.5% on CASIA-B with only 7 frames. The source code has been released at https://github.com/AbnerHqC/GaitSet.

On Learning Disentangled Representations for Gait Recognition

Abstract: Gait, the walking pattern of individuals, is one of the important biometrics modalities. Most of the existing gait recognition methods take silhouettes or articulated body models as gait features. These methods suffer from degraded recognition performance when handling confounding variables, such as clothing, carrying and viewing angle. To remedy this issue, we propose a novel AutoEncoder framework, GaitNet, to explicitly disentangle appearance, canonical and pose features from RGB imagery. The LSTM integrates pose features over time as a dynamic gait feature while canonical features are averaged as a static gait feature. Both of them are utilized as classification features. In addition, we collect a Frontal-View Gait (FVG) dataset to focus on gait recognition from frontal-view walking, which is a challenging problem since it contains minimal gait cues compared to other views. FVG also includes other important variations, e.g., walking speed, carrying, and clothing. With extensive experiments on CASIA-B, USF, and FVG datasets, our method demonstrates superior performance to the SOTA quantitatively, the ability of feature disentanglement qualitatively, and promising computational efficiency. We further compare our GaitNet with state-of-the-art face recognition to demonstrate the advantages of gait biometrics identification under certain scenarios, e.g., long distance/lower resolutions, cross viewing angles.

Overview of state-of-the-art deep gait recognition methods.

Gait recognition solutions can be classified into model-based {{cite:37a8a37}} and appearance-based {{cite:4d5cd0f}} categories, respectively dealing with the structure and appearance of the body. Nowadays, deep learning networks provide state-of-the-art results on both model-based and appearance-based gait recognition {{cite:e053559}}. Table {{table:653bdef5-55f1-4a09-87e9-50beca2cf438}}  presents an overview of the main characteristics of deep learning methods for gait recognition, sorted according to their release date, highlighting their network architectures and the templates used to aggregate temporal information along with the video. These temporal templates can be obtained either in the initial layer of the deep network in the form of Gait Energy Image (GEI) {{cite:4d5cd0f}}, {{cite:812e06c}}, {{cite:285b6c0}}, Period Energy Image (PEI) {{cite:91c4b86}}, Chrono Gait Image (CGI) {{cite:dbc64c9}}, and others, or in an intermediate layer of the network, for instance in the form of Convolutional Gait Energy Map (Conv. GEM).

The state-of-the-art deep gait recognition methods presented in Table {{table:653bdef5-55f1-4a09-87e9-50beca2cf438}}  are used for benchmarking. All these methods have reported their results using the same datasets and test protocols that we also strictly followed. For the CASIA-B dataset {{cite:b6abc76}}, we form the training and testing sets, respectively with the data from the first 74 subjects and the remaining 50 subjects. Then, the first four normal walking sequences are considered as the reference set, and the remaining 2 normal walking, 2 walking with coat, and 2 walking with bag sequences, form the NM, CL, BG probe sets respectively. The OU-MVLP dataset {{cite:786497a}} is divided into two different recording sessions, one for training (5153 subjects) and one for testing (5154 subjects). During the test stage, one sequence (index #00) is used as probe and the other sequence (index #01) is held in the gallery. The results for both datasets have been reported for all combinations of angles, except the identical ones.


Iris recognition using scattering transform and textural features

Abstract: Iris recognition has drawn a lot of attention since the mid-twentieth century. Among all biometric features, iris is known to possess a rich set of features. Different features have been used to perform iris recognition in the past. In this paper, two powerful sets of features are introduced to be used for iris recognition: scattering transform-based features and textural features. PCA is also applied on the extracted features to reduce the dimensionality of the feature vector while preserving most of the information of its initial value. Minimum distance classifier is used to perform template matching for each new test sample. The proposed scheme is tested on a well-known iris database, and showed promising results with the best accuracy rate of 99.2%.

An experimental study of deep convolutional features for iris recognition

Abstract: Iris is one of the popular biometrics that is widely used for identity authentication. Different features have been used to perform iris recognition in the past. Most of them are based on hand-crafted features designed by biometrics experts. Due to tremendous success of deep learning in computer vision problems, there has been a lot of interest in applying features learned by convolutional neural networks on general image recognition to other tasks such as segmentation, face recognition, and object detection. In this paper, we have investigated the application of deep features extracted from VGG-Net for iris recognition. The proposed scheme has been tested on two well-known iris databases, and has shown promising results with the best accuracy rate of 99.4%, which outperforms the previous best result.

The performance of iris recognition models on some of the most popular datasets.

Popular Metrics For Evaluating Biometrics Recognition Systems

Performance of Models for Iris Recognition:
Many of the recent iris recognition works have reported their accuracy rates on different iris databases, making it hard to compare all of them on a single benchmark.
The performance of deep learning-based iris recognition algorithms, and their comparison with some of the promising classical iris recognition models are provided in Table {{table:ee3853bb-d135-46cd-b885-3daa877a20c2}} .
As we can see models based on deep learning algorithms achieve superior performance over classical techniques.
Some of these numbers are taken from {{cite:426073c}} and {{cite:adb3790}}.


Adaptive two-regime method: application to front propagation

Abstract: The Adaptive Two-Regime Method (ATRM) is developed for hybrid (multiscale) stochastic simulation of reaction-diffusion problems. It efficiently couples detailed Brownian dynamics simulations with coarser lattice-based models. The ATRM is a generalization of the previously developed Two-Regime Method [Flegg et al, Journal of the Royal Society Interface, 2012] to multiscale problems which require a dynamic selection of regions where detailed Brownian dynamics simulation is used. Typical applications include a front propagation or spatio-temporal oscillations. In this paper, the ATRM is used for an in-depth study of front propagation in a stochastic reaction-diffusion system which has its mean-field model given in terms of the Fisher equation [Fisher, Annals of Eugenics, 1937]. It exhibits a travelling reaction front which is sensitive to stochastic fluctuations at the leading edge of the wavefront. Previous studies into stochastic effects on the Fisher wave propagation speed have focused on lattice-based models, but there has been limited progress using off-lattice (Brownian dynamics) models, which suffer due to their high computational cost, particularly at the high molecular numbers that are necessary to approach the Fisher mean-field model. By modelling only the wavefront itself with the off-lattice model, it is shown that the ATRM leads to the same Fisher wave results as purely off-lattice models, but at a fraction of the computational cost. The error analysis of the ATRM is also presented for a morphogen gradient model.

Analysis of the two-regime method on square meshes

Abstract: The two-regime method (TRM) has been recently developed for optimizing stochastic reaction-diffusion simulations. It is a multiscale (hybrid) algorithm which uses stochastic reaction-diffusion models with different levels of detail in different parts of the computational domain. The coupling condition on the interface between different modelling regimes of the TRM was previously derived for one-dimensional models. In this paper, the TRM is generalized to higher dimensional reaction-diffusion systems. Coupling Brownian dynamics models with compartment-based models on regular (square) two-dimensional lattices is studied in detail. In this case, the interface between different modelling regimes contain either flat parts or right-angled corners. Both cases are studied in the paper. For flat interfaces, it is shown that the one-dimensional theory can be used along the line perpendicular to the TRM interface. In the direction tangential to the interface, two choices of the TRM parameters are presented. Their applicability depends on the compartment size and the time step used in the molecular-based regime. The two-dimensional generalization of the TRM is also discussed in the case of corners.

Hybrid spatial Gillespie and particle tracking simulation

Abstract: Motivation: Cellular signal transduction involves spatial–temporal dynamics and often stochastic effects due to the low particle abundance of some molecular species. Others can, however, be of high abundances. Such a system can be simulated either with the spatial Gillespie/Stochastic Simulation Algorithm (SSA) or Brownian/Smoluchowski dynamics if space and stochasticity are important. To combine the accuracy of particle-based methods with the superior performance of the SSA, we suggest a hybrid simulation. Results: The proposed simulation allows an interactive or automated switching for regions or species of interest in the cell. Especially we see an application if for instance receptor clustering at the membrane is modeled in detail and the transport through the cytoplasm is included as well. The results show the increase in performance of the overall simulation, and the limits of the approach if crowding is included. Future work will include the development of a GUI to improve control of the simulation. Availability of Implementation: www.bison.ethz.ch/research/spatial_simulations. Contact: mklann@ee.ethz.ch or koeppl@ethz.ch Supplementary/Information: Supplementary data are available at Bioinformatics online.

A summary of the meso-micro hybrid papers that will be covered in this section. The methods in all the meso-micro hybrid papers summarised here are designed for modelling reaction-diffusion systems. Descriptors are as in Table .

In this section we will begin by introducing, in broad terms, models which couple microscopic dynamics to mesoscopic dynamics, which we will refer to as “meso-micro” hybrid models. After summarising the key properties of the meso-micro hybrid methods covered in this section, in Table {{table:b6cedffd-8eee-4e80-8ad0-1c8b89019974}} , we go on to describe them in more detail. We begin by giving a detailed description of an illustrative example of a meso-micro hybrid model, the ghost cell method {{cite:912d590}} and present pseudocode for its implementation. We then summarise other existing meso-micro hybrid methods.


SoulMate: Short-text author linking through Multi-aspect temporal-textual embedding

Abstract: Linking authors of short-text contents has important usages in many applications, including Named Entity Recognition (NER) and human community detection. However, certain challenges lie ahead. Firstly, the input short-text contents are noisy, ambiguous, and do not follow the grammatical rules. Secondly, traditional text mining methods fail to effectively extract concepts through words and phrases. Thirdly, the textual contents are temporally skewed, which can affect the semantic understanding by multiple time facets. Finally, using the complementary knowledge-bases makes the results biased to the content of the external database and deviates the understanding and interpretation away from the real nature of the given short text corpus. To overcome these challenges, we devise a neural network-based temporal-textual framework that generates the tightly connected author subgraphs from microblog short-text contents. Our approach, on the one hand, computes the relevance score (edge weight) between the authors through considering a portmanteau of contents and concepts, and on the other hand, employs a stack-wise graph cutting algorithm to extract the communities of the related authors. Experimental results show that compared to other knowledge-centered competitors, our multi-aspect vector space model can achieve a higher performance in linking short-text authors. Additionally, given the author linking task, the more comprehensive the dataset is, the higher the significance of the extracted concepts will be.

Efficient Estimation of Word Representations in Vector Space

Abstract: We propose two novel model architectures for computing continuous vector representations of words from very large data sets. The quality of these representations is measured in a word similarity task, and the results are compared to the previously best performing techniques based on different types of neural networks. We observe large improvements in accuracy at much lower computational cost, i.e. it takes less than a day to learn high quality word vectors from a 1.6 billion words data set. Furthermore, we show that these vectors provide state-of-the-art performance on our test set for measuring syntactic and semantic word similarities.

Literature

As briefed in Table {{table:f511d457-40d2-4eb9-8ae4-43b8180bb3c0}} , the related work comprises contextual semantics, inferring professions, and cognitive perspective.


Fully convolutional networks for semantic segmentation

Abstract: Convolutional networks are powerful visual models that yield hierarchies of features. We show that convolutional networks by themselves, trained end-to-end, pixels-to-pixels, exceed the state-of-the-art in semantic segmentation. Our key insight is to build “fully convolutional” networks that take input of arbitrary size and produce correspondingly-sized output with efficient inference and learning. We define and detail the space of fully convolutional networks, explain their application to spatially dense prediction tasks, and draw connections to prior models. We adapt contemporary classification networks (AlexNet [20], the VGG net [31], and GoogLeNet [32]) into fully convolutional networks and transfer their learned representations by fine-tuning [3] to the segmentation task. We then define a skip architecture that combines semantic information from a deep, coarse layer with appearance information from a shallow, fine layer to produce accurate and detailed segmentations. Our fully convolutional network achieves state-of-the-art segmentation of PASCAL VOC (20% relative improvement to 62.2% mean IU on 2012), NYUDv2, and SIFT Flow, while inference takes less than one fifth of a second for a typical image.

Attention to Scale: Scale-Aware Semantic Image Segmentation

Abstract: Incorporating multi-scale features in fully convolutional neural networks (FCNs) has been a key element to achieving state-of-the-art performance on semantic image segmentation. One common way to extract multi-scale features is to feed multiple resized input images to a shared deep network and then merge the resulting features for pixelwise classification. In this work, we propose an attention mechanism that learns to softly weight the multi-scale features at each pixel location. We adapt a state-of-the-art semantic image segmentation model, which we jointly train with multi-scale input images and the attention model. The proposed attention model not only outperforms averageand max-pooling, but allows us to diagnostically visualize the importance of features at different positions and scales. Moreover, we show that adding extra supervision to the output at each scale is essential to achieving excellent performance when merging multi-scale features. We demonstrate the effectiveness of our model with extensive experiments on three challenging datasets, including PASCAL-Person-Part, PASCAL VOC 2012 and a subset of MS-COCO 2014.

Look into Person: Self-supervised Structure-sensitive Learning and A New Benchmark for Human Parsing

Abstract: Human parsing has recently attracted a lot of research interests due to its huge application potentials. However existing datasets have limited number of images and annotations, and lack the variety of human appearances and the coverage of challenging cases in unconstrained environment. In this paper, we introduce a new benchmark "Look into Person (LIP)" that makes a significant advance in terms of scalability, diversity and difficulty, a contribution that we feel is crucial for future developments in human-centric analysis. This comprehensive dataset contains over 50,000 elaborately annotated images with 19 semantic part labels, which are captured from a wider range of viewpoints, occlusions and background complexity. Given these rich annotations we perform detailed analyses of the leading human parsing approaches, gaining insights into the success and failures of these methods. Furthermore, in contrast to the existing efforts on improving the feature discriminative capability, we solve human parsing by exploring a novel self-supervised structure-sensitive learning approach, which imposes human pose structures into parsing results without resorting to extra supervision (i.e., no need for specifically labeling human joints in model training). Our self-supervised learning framework can be injected into any advanced neural networks to help incorporate rich high-level knowledge regarding human joints from a global perspective and improve the parsing results. Extensive evaluations on our LIP and the public PASCAL-Person-Part dataset demonstrate the superiority of our method.

Look into Person: Joint Body Parsing & Pose Estimation Network and a New Benchmark

Abstract: Human parsing and pose estimation have recently received considerable interest due to their substantial application potentials. However, the existing datasets have limited numbers of images and annotations and lack a variety of human appearances and coverage of challenging cases in unconstrained environments. In this paper, we introduce a new benchmark named “Look into Person (LIP)” that provides a significant advancement in terms of scalability, diversity, and difficulty, which are crucial for future developments in human-centric analysis. This comprehensive dataset contains over 50,000 elaborately annotated images with 19 semantic part labels and 16 body joints, which are captured from a broad range of viewpoints, occlusions, and background complexities. Using these rich annotations, we perform detailed analyses of the leading human parsing and pose estimation approaches, thereby obtaining insights into the successes and failures of these methods. To further explore and take advantage of the semantic correlation of these two tasks, we propose a novel joint human parsing and pose estimation network to explore efficient context modeling, which can simultaneously predict parsing and pose with extremely high quality. Furthermore, we simplify the network to solve human parsing by exploring a novel self-supervised structure-sensitive learning approach, which imposes human pose structures into the parsing results without resorting to extra supervision. The datasets, code and models are available at http://www.sysu-hcp.net/lip/.

Devil in the Details: Towards Accurate Single and Multiple Human Parsing

Abstract: Human parsing has received considerable interest due to its wide application potentials. Nevertheless, it is still unclear how to develop an accurate human parsing system in an efficient and elegant way. In this paper, we identify several useful properties, including feature resolution, global context information and edge details, and perform rigorous analyses to reveal how to leverage them to benefit the human parsing task. The advantages of these useful properties finally result in a simple yet effective Context Embedding with Edge Perceiving (CE2P) framework for single human parsing. Our CE2P is end-to-end trainable and can be easily adopted for conducting multiple human parsing. Benefiting the superiority of CE2P, we achieved the 1st places on all three human parsing benchmarks. Without any bells and whistles, we achieved 56.50\% (mIoU), 45.31\% (mean $AP^r$) and 33.34\% ($AP^p_{0.5}$) in LIP, CIHP and MHP v2.0, which outperform the state-of-the-arts more than 2.06\%, 3.81\% and 1.87\%, respectively. We hope our CE2P will serve as a solid baseline and help ease future research in single/multiple human parsing. Code has been made available at \url{https://github.com/liutinglt/CE2P}.

Correlating Edge, Pose With Parsing

Abstract: According to existing studies, human body edge and pose are two beneficial factors to human parsing. The effectiveness of each of the high-level features (edge and pose) is confirmed through the concatenation of their features with the parsing features. Driven by the insights, this paper studies how human semantic boundaries and keypoint locations can jointly improve human parsing. Compared with the existing practice of feature concatenation, we find that uncovering the correlation among the three factors is a superior way of leveraging the pivotal contextual cues provided by edges and poses. To capture such correlations, we propose a Correlation Parsing Machine (CorrPM) employing a heterogeneous non-local block to discover the spatial affinity among feature maps from the edge, pose and parsing. The proposed CorrPM allows us to report new state-of-the-art accuracy on three human parsing datasets. Importantly, comparative studies confirm the advantages of feature correlation over the concatenation.

Self-Learning With Rectification Strategy for Human Parsing

Abstract: In this paper, we solve the sample shortage problem in the human parsing task. We begin with the self-learning strategy, which generates pseudo-labels for unlabeled data to retrain the model. However, directly using noisy pseudo-labels will cause error amplification and accumulation. Considering the topology structure of human body, we propose a trainable graph reasoning method that establishes internal structural connections between graph nodes to correct two typical errors in the pseudo-labels, i.e., the global structural error and the local consistency error. For the global error, we first transform category-wise features into a high-level graph model with coarse-grained structural information, and then decouple the high-level graph to reconstruct the category features. The reconstructed features have a stronger ability to represent the topology structure of the human body. Enlarging the receptive field of features can effectively reducing the local error. We first project feature pixels into a local graph model to capture pixel-wise relations in a hierarchical graph manner, then reverse the relation information back to the pixels. With the global structural and local consistency modules, these errors are rectified and confident pseudo-labels are generated for retraining. Extensive experiments on the LIP and the ATR datasets demonstrate the effectiveness of our global and local rectification modules. Our method outperforms other state-of-the-art methods in supervised human parsing tasks.

Learning Compositional Neural Information Fusion for Human Parsing

Abstract: This work proposes to combine neural networks with the compositional hierarchy of human bodies for efficient and complete human parsing. We formulate the approach as a neural information fusion framework. Our model assembles the information from three inference processes over the hierarchy: direct inference (directly predicting each part of a human body using image information), bottom-up inference (assembling knowledge from constituent parts), and top-down inference (leveraging context from parent nodes). The bottom-up and top-down inferences explicitly model the compositional and decompositional relations in human bodies, respectively. In addition, the fusion of multi-source information is conditioned on the inputs, i.e., by estimating and considering the confidence of the sources. The whole model is end-to-end differentiable, explicitly modeling information flows and structures. Our approach is extensively evaluated on four popular datasets, outperforming the state-of-the-arts in all cases, with a fast processing speed of 23fps. Our code and results have been released to help ease future research in this direction.

CDGNet: Class Distribution Guided Network for Human Parsing

Abstract: The objective of human parsing is to partition a human in an image into constituent parts. This task involves labeling each pixel of the human image according to the classes. Since the human body comprises hierarchically structured parts, each body part of an image can have its sole position distribution characteristic. Probably, a human head is less likely to be under the feet, and arms are more likely to be near the torso. Inspired by this observation, we make instance class distributions by accumulating the original human parsing label in the horizontal and vertical directions, which can be utilized as supervision signals. Using these horizontal and vertical class distribution labels, the network is guided to exploit the intrinsic position distribution of each class. We combine two guided features to form a spatial guidance map, which is then superimposed onto the baseline network by multiplication and concatenation to distinguish the human parts precisely. We conducted extensive experiments to demonstrate the effectiveness and superiority of our method on three well-known benchmarks: LIP, ATR, and CIHP databases.††Our code is available at https://github.com/tjpulkl/CDGNet.

Quantitative results on LIP val in terms of mean pixel Intersection-over-Union (mIoU) (%)

In this section, we outline the performance of several of the previous methods on popular segmentation benchmarks. Since some models are not reported their performance on standard datasets, it is hard to make a full comparison. The following tables summarize the performances of several of the prominent models on different datasets. Table {{table:f7de7341-a0e6-4f62-938b-a63c45ab7878}}  focuses on the PASCAL-Person-Part test set. Since the introduction of FCN-based models, the performance has been greatly improved. Table {{table:631a904f-0766-4e4b-82b1-4c78fb58d104}}  focuses on the LIP validation. On this dataset, the latest model improved by 32.01% in terms of mIoU over the original model. Table {{table:d936b018-501f-4a00-aeff-680c54e314e1}}  focuses on the CIHP test set. This dataset is more challenging than the PASCAL-Person-Part dataset, both two datasets are multi-person. Table {{table:ab1df25e-0982-4318-abfe-1fb26582083c}}  focuses on the ATR test set. The latest model achieves 87.16 in terms of F-1. Finally, Table {{table:af4d1e0a-dcc5-4a1d-9ad2-5141d7748284}}  summarizes the performance of several models for the Fashion Clothing dataset.


Focused on feed forward neural networks,seq2seq models, temporal CNNs and RNNs in electric load forecasting

To forecast the annual load of distribution feeders

Deep Learning for Time Series Forecasting: The Electric Load Case

Abstract: Management and efficient operations in critical infrastructure such as Smart Grids take huge advantage of accurate power load forecasting which, due to its nonlinear nature, remains a challenging task. Recently, deep learning has emerged in the machine learning field achieving impressive performance in a vast range of tasks, from image classification to machine translation. Applications of deep learning models to the electric load forecasting problem are gaining interest among researchers as well as the industry, but a comprehensive and sound comparison among different architectures is not yet available in the literature. This work aims at filling the gap by reviewing and experimentally evaluating on two real-world datasets the most recent trends in electric load forecasting, by contrasting deep learning architectures on short term forecast (one day ahead prediction). Specifically, we focus on feedforward and recurrent neural networks, sequence to sequence models and temporal convolutional neural networks along with architectural variants, which are known in the signal processing community but are novel to the load forecasting one.

A Hybrid Distribution Feeder Long-Term Load Forecasting Method Based on Sequence Prediction

Abstract: Distribution feeder long-term load forecast (LTLF) is a critical task many electric utility companies perform on an annual basis. The goal of this task is to forecast the annual load of distribution feeders. The previous top-down and bottom-up LTLF methods are unable to incorporate different levels of information. This paper proposes a hybrid modeling method using sequence prediction for this classic and important task. The proposed method can seamlessly integrate top-down, bottom-up, and sequential information hidden in multi-year data. Two advanced sequence prediction models long short-term memory (LSTM) and gated recurrent unit (GRU) networks are investigated in this paper. They successfully solve the vanishing and exploding gradient problems a standard recurrent neural network has. This paper first explains the theories of LSTM and GRU networks and then discusses the steps of feature selection, feature engineering and model implementation in detail. In the end, a real-world application example for a large urban grid in West Canada is provided. LSTM and GRU networks under different sequential configurations and traditional models including bottom-up, ARIMA, and feed-forward neural network are all implemented and compared in detail. The proposed method demonstrates superior performance and great practicality.

Summary on Electric Load Forecasting.

Motepe et al. {{cite:a6efc77}} has proposed a hybrid DL and AI model for distribution network load forecasting in South Africa. The state-of-the-art hybrid DL technique and a AI technique investigated are LSTM and optimally pruned extreme learning machines, respectively. These are compared with adaptive neuro-fuzzy inference system (ANFIS) and LSTM was seen to achieve higher accuracy. To forecast the PV power output and residential power load, a deep RNN with LSTM (DRNN-LSTM) model is proposed by {{cite:5327ea8}}. To optimize the load dispatch of grid-connected community microgrid, particle swarm optimization (PSO) algorithm is used.
Results on two real-world data sets prove that the DRNN-LSTM model outperforms the support vector machine (SVM) and multilayer perceptron network. It is also observed that load dispatch optimization enables EVs, and energy storage systems (ESS) to shift the peak load and reduce 8.97% of the daily costs.
We summarize the existing works on electric load forecasting in Table {{table:f902c992-c99d-4c7b-96b7-f8bc88fd8b93}} .


DeepXDE: A Deep Learning Library for Solving Differential Equations

Abstract: Deep learning has achieved remarkable success in diverse applications; however, its use in solving partial differential equations (PDEs) has emerged only recently. Here, we present an overview of physics-informed neural networks (PINNs), which embed a PDE into the loss of the neural network using automatic differentiation. The PINN algorithm is simple, and it can be applied to different types of PDEs, including integro-differential equations, fractional PDEs, and stochastic PDEs. Moreover, from the implementation point of view, PINNs solve inverse problems as easily as forward problems. We propose a new residual-based adaptive refinement (RAR) method to improve the training efficiency of PINNs. For pedagogical reasons, we compare the PINN algorithm to a standard finite element method. We also present a Python library for PINNs, DeepXDE, which is designed to serve both as an education tool to be used in the classroom as well as a research tool for solving problems in computational science and engineering. Specifically, DeepXDE can solve forward problems given initial and boundary conditions, as well as inverse problems given some extra measurements. DeepXDE supports complex-geometry domains based on the technique of constructive solid geometry, and enables the user code to be compact, resembling closely the mathematical formulation. We introduce the usage of DeepXDE and its customizability, and we also demonstrate the capability of PINNs and the user-friendliness of DeepXDE for five different examples. More broadly, DeepXDE contributes to the more rapid development of the emerging Scientific Machine Learning field.

PyDEns: a Python Framework for Solving Differential Equations with Neural Networks

Abstract: Recently, a lot of papers proposed to use neural networks to approximately solve partial differential equations (PDEs). Yet, there has been a lack of flexible framework for convenient experimentation. In an attempt to fill the gap, we introduce a PyDEns-module open-sourced on GitHub. Coupled with capabilities of BatchFlow, open-source framework for convenient and reproducible deep learning, PyDEns-module allows to 1) solve partial differential equations from a large family, including heat equation and wave equation 2) easily search for the best neural-network architecture among the zoo, that includes ResNet and DenseNet 3) fully control the process of model-training by testing different point-sampling schemes. With that in mind, our main contribution goes as follows: implementation of a ready-to-use and open-source numerical solver of PDEs of a novel format, based on neural networks.

NVIDIA SimNet^{TM}: an AI-accelerated multi-physics simulation framework

Abstract: We present SimNet, an AI-driven multi-physics simulation framework, to accelerate simulations across a wide range of disciplines in science and engineering. Compared to traditional numerical solvers, SimNet addresses a wide range of use cases - coupled forward simulations without any training data, inverse and data assimilation problems. SimNet offers fast turnaround time by enabling parameterized system representation that solves for multiple configurations simultaneously, as opposed to the traditional solvers that solve for one configuration at a time. SimNet is integrated with parameterized constructive solid geometry as well as STL modules to generate point clouds. Furthermore, it is customizable with APIs that enable user extensions to geometry, physics and network architecture. It has advanced network architectures that are optimized for high-performance GPU computing, and offers scalable performance for multi-GPU and multi-Node implementation with accelerated linear algebra as well as FP32, FP64 and TF32 computations. In this paper we review the neural network solver methodology, the SimNet architecture, and the various features that are needed for effective solution of the PDEs. We present real-world use cases that range from challenging forward multi-physics simulations with turbulence and complex 3D geometries, to industrial design optimization and inverse problems that are not addressed efficiently by the traditional solvers. Extensive comparisons of SimNet results with open source and commercial solvers show good correlation.

NeuroDiffEq: A Python package for solving differential equations with neural networks

Abstract: Differential equations emerge in various scientific and engineering domains for modeling physical phenomena. Most differential equations of practical interest are analytically intractable. Traditionally, differential equations are solved by numerical methods. Sophisticated algorithms exist to integrate differential equations in time and space. Time integration techniques continue to be an active area of research and include backward difference formulas and Runge-Kutta methods (Conde, Gottlieb, Grant, & Shadid, 2017). Common spatial discretization approaches include the finite difference method (FDM), finite volume method (FVM), and finite element method (FEM) as well as spectral methods such as the Fourier-spectral method. These classical methods have been studied in detail and much is known about their convergence properties. Moreover, highly optimized codes exist for solving differential equations of practical interest with these techniques (Seefeldt et al., 2017; Smith & Abeysinghe, 2017). While these methods are efficient and well-studied, their expressibility is limited by their function representation.

ADCME: Learning Spatially-varying Physical Fields using Deep Neural Networks

Abstract: ADCME is a novel computational framework to solve inverse problems involving physical simulations and deep neural networks (DNNs). This paper benchmarks its capability to learn spatially-varying physical fields using DNNs. We demonstrate that our approach has superior accuracy compared to the discretization approach on a variety of problems, linear or nonlinear, static or dynamic. Technically, we formulate our inverse problem as a PDE-constrained optimization problem. We express both the numerical simulations and DNNs using computational graphs and therefore, we can calculate the gradients using reverse-mode automatic differentiation. We apply a physics constrained learning algorithm (PCL) to efficiently back-propagate gradients through iterative solvers for nonlinear equations. The open-source software which accompanies the present paper can be found at this https URL.

TensorDiffEq: Scalable Multi-GPU Forward and Inverse Solvers for Physics Informed Neural Networks

Abstract: Physics-Informed Neural Networks promise to revolutionize science and engineering practice, by introducing domain-aware deep machine learning models into scientific computation. Several software suites have emerged to make the implementation and usage of these architectures available to the research and industry communities. Here we introduce\linebreak TensorDiffEq, built on Tensorflow 2.x, which presents an intuitive Keras-like interface for problem domain definition, model definition, and solution of forward and inverse problems using physics-aware deep learning methods. TensorDiffEq takes full advantage of Tensorflow 2.x infrastructure for deployment on multiple GPUs, allowing the implementation of large high-dimensional and complex models. Simultaneously, TensorDiffEq supports the Keras API for custom neural network architecture definitions. In the case of smaller or simpler models, the package allows for rapid deployment on smaller-scale CPU platforms with negligible changes to the implementation scripts. We demonstrate the basic usage and capabilities of TensorDiffEq in solving forward, inverse, and data assimilation problems of varying sizes and levels of complexity. The source code is available at https://github.com/tensordiffeq.

IDRLnet: A Physics-Informed Neural Network Library

Abstract: Physics Informed Neural Network (PINN) is a scientific computing framework used to solve both forward and inverse problems modeled by Partial Differential Equations (PDEs). This paper introduces IDRLnet, a Python toolbox for modeling and solving problems through PINN systematically. IDRLnet constructs the framework for a wide range of PINN algorithms and applications. It provides a structured way to incorporate geometric objects, data sources, artificial neural networks, loss metrics, and optimizers within Python. Furthermore, it provides functionality to solve noisy inverse problems, variational minimization, and integral differential equations. New PINN variants can be integrated into the framework easily. Source code, tutorials, and documentation are available at \url{https://github.com/idrl-lab/idrlnet}.

Elvet - a neural network-based differential equation and variational problem solver

Abstract: We present Elvet, a Python package for solving differential equations and variational problems using machine learning methods. Elvet can deal with any system of coupled ordinary or partial differential equations with arbitrary initial and boundary conditions. It can also minimize any functional that depends on a collection of functions of several variables while imposing constraints on them. The solution to any of these problems is represented as a neural network trained to produce the desired function.

GPyTorch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU Acceleration

Abstract: Despite advances in scalable models, the inference tools used for Gaussian processes (GPs) have yet to fully capitalize on developments in computing hardware. We present an efficient and general approach to GP inference based on Blackbox Matrix-Matrix multiplication (BBMM). BBMM inference uses a modified batched version of the conjugate gradients algorithm to derive all terms for training and inference in a single call. BBMM reduces the asymptotic complexity of exact GP inference from $O(n^3)$ to $O(n^2)$. Adapting this algorithm to scalable approximations and complex GP models simply requires a routine for efficient matrix-matrix multiplication with the kernel and its derivative. In addition, BBMM uses a specialized preconditioner to substantially speed up convergence. In experiments we show that BBMM effectively uses GPU hardware to dramatically accelerate both exact GP inference and scalable approximations. Additionally, we provide GPyTorch, a software platform for scalable GP inference via BBMM, built on PyTorch.

Neural Tangents: Fast and Easy Infinite Neural Networks in Python

Abstract: Neural Tangents is a library for working with infinite-width neural networks. It provides a high-level API for specifying complex and hierarchical neural network architectures. These networks can then be trained and evaluated either at finite-width as usual, or in their infinite-width limit. For the infinite-width networks, Neural Tangents performs exact inference either via Bayes' rule or gradient descent, and generates the corresponding Neural Network Gaussian Process and Neural Tangent kernels. Additionally, Neural Tangents provides tools to study gradient descent training dynamics of wide but finite networks. The entire library runs out-of-the-box on CPU, GPU, or TPU. All computations can be automatically distributed over multiple accelerators with near-linear scaling in the number of devices. In addition to the repository below, we provide an accompanying interactive Colab notebook at https://colab.sandbox.google.com/github/neural-tangents/neural-tangents/blob/master/notebooks/neural_tangents_cookbook.ipynb

Major software libraries specifically designed for physics-informed machine learning

More libraries have been built in recent years, and others are being updated on a continuous basis, making this a dynamic field of research and development.
In this subsection, we will examine each library while also presenting a comprehensive synthesis in
Table {{table:0a7325d1-e340-442e-bb4d-b67ec44d9135}} .


Deep Learning Face Attributes in the Wild

Abstract: Predicting face attributes in the wild is challenging due to complex face variations. We propose a novel deep learning framework for attribute prediction in the wild. It cascades two CNNs, LNet and ANet, which are fine-tuned jointly with attribute tags, but pre-trained differently. LNet is pre-trained by massive general object categories for face localization, while ANet is pre-trained by massive face identities for attribute prediction. This framework not only outperforms the state-of-the-art with a large margin, but also reveals valuable facts on learning face representation. (1) It shows how the performances of face localization (LNet) and attribute prediction (ANet) can be improved by different pre-training strategies. (2) It reveals that although the filters of LNet are fine-tuned only with image-level attribute tags, their response maps over entire images have strong indication of face locations. This fact enables training LNet for face localization with only image-level annotations, but without face bounding boxes or landmarks, which are required by all attribute recognition works. (3) It also demonstrates that the high-level hidden neurons of ANet automatically discover semantic concepts after pre-training with massive face identities, and such concepts are significantly enriched after fine-tuning with attribute tags. Each attribute can be well explained with a sparse linear combination of these concepts.

Multi-attention Recurrent Network for Human Communication Comprehension

Abstract: Human face-to-face communication is a complex multimodal signal. We use words (language modality), gestures (vision modality) and changes in tone (acoustic modality) to convey our intentions. Humans easily process and understand face-to-face communication, however, comprehending this form of communication remains a significant challenge for Artificial Intelligence (AI). AI must understand each modality and the interactions between them that shape the communication. In this paper, we present a novel neural architecture for understanding human communication called the Multi-attention Recurrent Network (MARN). The main strength of our model comes from discovering interactions between modalities through time using a neural component called the Multi-attention Block (MAB) and storing them in the hybrid memory of a recurrent component called the Long-short Term Hybrid Memory (LSTHM). We perform extensive comparisons on six publicly available datasets for multimodal sentiment analysis, speaker trait recognition and emotion recognition. MARN shows state-of-the-art results performance in all the datasets.

VoxCeleb2: Deep Speaker Recognition

Abstract:
The objective of this paper is speaker recognition under noisy and unconstrained conditions.
We make two key contributions. First, we introduce a very large-scale audio-visual speaker recognition dataset collected from open-source media. Using a fully automated pipeline, we curate VoxCeleb2 which contains over a million utterances from over 6,000 speakers. This is several times larger than any publicly available speaker recognition dataset.
Second, we develop and compare Convolutional Neural Network (CNN) models and training strategies that can effectively recognise identities from voice under various conditions. The models trained on the VoxCeleb2 dataset surpass the performance of previous works on a benchmark dataset by a significant margin.

CelebV-HQ: A Large-Scale Video Facial Attributes Dataset

Abstract: Large-scale datasets have played indispensable roles in the recent success of face generation/editing and significantly facilitated the advances of emerging research fields. However, the academic community still lacks a video dataset with diverse facial attribute annotations, which is crucial for the research on face-related videos. In this work, we propose a large-scale, high-quality, and diverse video dataset with rich facial attribute annotations, named the High-Quality Celebrity Video Dataset (CelebV-HQ). CelebV-HQ contains 35,666 video clips with the resolution of 512x512 at least, involving 15,653 identities. All clips are labeled manually with 83 facial attributes, covering appearance, action, and emotion. We conduct a comprehensive analysis in terms of age, ethnicity, brightness stability, motion smoothness, head pose diversity, and data quality to demonstrate the diversity and temporal coherence of CelebV-HQ. Besides, its versatility and potential are validated on two representative tasks, i.e., unconditional video generation and video facial attribute editing. Furthermore, we envision the future potential of CelebV-HQ, as well as the new opportunities and challenges it would bring to related research directions. Data, code, and models are publicly available. Project page: https://celebv-hq.github.io.

Facial Analysis Tasks. Overview of different face related tasks and relevant datasets down the lane.

Facial Representation Learning.
Till date, most of the existing facial analysis approaches are conducted in a task-specific way with fully supervised manner {{cite:6233270}}, {{cite:31ca9d5}}, {{cite:6848c04}} on manually annotated data to enhance performance. Any state-of-the-art model's performance on benchmarked datasets is impacted by the quality and quantity of annotated data used during training. Table {{table:a2cf2fff-845e-4d78-9ff0-4b976af301a0}}  shows an overview of the task-specific large-scale facial image or video datasets that have been curated over the past decade {{cite:8dd4708}} to facilitate research in Face Verification (LFW {{cite:6c93f3f}}, MS-celeb-1M {{cite:468ecb2}}, VGG-FACE {{cite:31ca9d5}}, VGGFace2 {{cite:edd0cd1}}), Facial Attribute Recognition(CelebA {{cite:889afef}}, CelebV-HQ {{cite:9981bd8}}), Facial Emotion Recognition (CMU-MOSEI {{cite:78d1932}}), DeepFake Detection (FF++ {{cite:5429a2b}}) and Lip Synchronization (LRS2 {{cite:61657f4}}). However, data curation encounters several challenges such as requirements of specialized hardware (e.g. for FER and action unit data), the discrepancy in data distribution that prevent merging of multiple datasets {{cite:e3935d2}}, and most importantly time consuming and resource expensive annotation process. To eliminate these drawbacks, some of the existing approaches {{cite:6dfc402}}, {{cite:4d2a22e}}, {{cite:6066499}} adopt data augmentation strategy via image or video synthesis as the surge in face generation technology fueled by Generative Adversarial Network (GAN) {{cite:6dfc402}}, {{cite:4d2a22e}}, {{cite:6066499}}, {{cite:f786eb1}} and other generation techniques {{cite:10d6cd7}}, {{cite:333fba5}} aids realistic face generation even with the control over facial attributes. These generation techniques add variation in training set quantitatively, but in some cases it still lags in qualitative aspects due to domain specific inconsistency and more importantly high network complexity.


Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks

Abstract: State-of-the-art object detection networks depend on region proposal algorithms to hypothesize object locations. Advances like SPPnet and Fast R-CNN have reduced the running time of these detection networks, exposing region proposal computation as a bottleneck. In this work, we introduce a Region Proposal Network (RPN) that shares full-image convolutional features with the detection network, thus enabling nearly cost-free region proposals. An RPN is a fully convolutional network that simultaneously predicts object bounds and objectness scores at each position. The RPN is trained end-to-end to generate high-quality region proposals, which are used by Fast R-CNN for detection. We further merge RPN and Fast R-CNN into a single network by sharing their convolutional features---using the recently popular terminology of neural networks with 'attention' mechanisms, the RPN component tells the unified network where to look. For the very deep VGG-16 model, our detection system has a frame rate of 5fps (including all steps) on a GPU, while achieving state-of-the-art object detection accuracy on PASCAL VOC 2007, 2012, and MS COCO datasets with only 300 proposals per image. In ILSVRC and COCO 2015 competitions, Faster R-CNN and RPN are the foundations of the 1st-place winning entries in several tracks. Code has been made publicly available

Deformable Convolutional Networks

Abstract: Convolutional neural networks (CNNs) are inherently limited to model geometric transformations due to the fixed geometric structures in their building modules. In this work, we introduce two new modules to enhance the transformation modeling capability of CNNs, namely, deformable convolution and deformable RoI pooling. Both are based on the idea of augmenting the spatial sampling locations in the modules with additional offsets and learning the offsets from the target tasks, without additional supervision. The new modules can readily replace their plain counterparts in existing CNNs and can be easily trained end-to-end by standard back-propagation, giving rise to deformable convolutional networks. Extensive experiments validate the performance of our approach. For the first time, we show that learning dense spatial transformation in deep CNNs is effective for sophisticated vision tasks such as object detection and semantic segmentation. The code is released at https://github.com/msracver/Deformable-ConvNets.

Cascade R-CNN: Delving Into High Quality Object Detection

Abstract: In object detection, an intersection over union (IoU) threshold is required to define positives and negatives. An object detector, trained with low IoU threshold, e.g. 0.5, usually produces noisy detections. However, detection performance tends to degrade with increasing the IoU thresholds. Two main factors are responsible for this: 1) overfitting during training, due to exponentially vanishing positive samples, and 2) inference-time mismatch between the IoUs for which the detector is optimal and those of the input hypotheses. A multi-stage object detection architecture, the Cascade R-CNN, is proposed to address these problems. It consists of a sequence of detectors trained with increasing IoU thresholds, to be sequentially more selective against close false positives. The detectors are trained stage by stage, leveraging the observation that the output of a detector is a good distribution for training the next higher quality detector. The resampling of progressively improved hypotheses guarantees that all detectors have a positive set of examples of equivalent size, reducing the overfitting problem. The same cascade procedure is applied at inference, enabling a closer match between the hypotheses and the detector quality of each stage. A simple implementation of the Cascade R-CNN is shown to surpass all single-model object detectors on the challenging COCO dataset. Experiments also show that the Cascade R-CNN is widely applicable across detector architectures, achieving consistent gains independently of the baseline detector strength. The code is available at https://github.com/zhaoweicai/cascade-rcnn.

Object detection results on the MS-COCO validation set. Both Faster-RCNN and Cascade-RCNN are significantly improved by our ResNeSt backbone.

We report our detection result on MS-COCO{{cite:25a9cce}} in Table {{table:4a6f28e3-6817-4749-a227-9fb424ee003c}} . All models are trained on COCO-2017 training set with 118k images, and evaluated on COCO-2017 validation set with 5k images (aka. minival) using the standard COCO AP metric of single scale.
We train all models with FPN{{cite:00029dd}}, synchronized batch normalization {{cite:2c4ced5}} and image scale augmentation (short size of a image is picked randomly from 640 to 800). 1x learning rate schedule is used. We conduct Faster-RCNNs and Cascade-RCNNs experiments using Detectron2{{cite:265e0c8}}.
For comparison, we simply replaced the vanilla ResNet backbones with our ResNeSt, while using the default settings for the hyper-parameters and detection heads {{cite:7536cf8}}, {{cite:265e0c8}}.

Compared to the baselines using standard ResNet, Our backbone is able to boost mean average precision by around 3% on both Faster-RCNNs and Cascade-RCNNs.
The result demonstrates our backbone has good generalization ability and can be easily transferred to the downstream task. Notably, our ResNeSt50 outperforms ResNet101 on both Faster-RCNN and Cascade-RCNN detection models, using significantly fewer parameters.
Detailed results in Table {{table:4a6f28e3-6817-4749-a227-9fb424ee003c}} .
We evaluate our Cascade-RCNN with ResNeSt101 deformable, that is trained using 1x learning rate schedule on COCO test-dev set as well. It yields a box mAP of 49.2 using single scale inference.

Object Detection and Instance Segmentation

For object detection, we add deformable convolution to our Cascade-RCNN model with ResNeSt-101 backbone and train the model on the MS-COCO training set for 1x schedule.
The resulting model achieves 49.2% mAP on COCO test-dev set, which surpass all previous methods including these employing multi-scale evaluation.
Detailed results are shown in Table {{table:4a6f28e3-6817-4749-a227-9fb424ee003c}} .


Distillation-Based Semi-Supervised Federated Learning for Communication-Efficient Collaborative Training with Non-IID Private Data

Abstract: This study develops a federated learning (FL) framework overcoming largely incremental communication costs due to model sizes in typical frameworks without compromising model performance. To this end, based on the idea of leveraging an unlabeled open dataset, we propose a distillation-based semi-supervised FL (DS-FL) algorithm that exchanges the outputs of local models among mobile devices, instead of model parameter exchange employed by the typical frameworks. In DS-FL, the communication cost depends only on the output dimensions of the models and does not scale up according to the model size. The exchanged model outputs are used to label each sample of the open dataset, which creates an additionally labeled dataset. Based on the new dataset, local models are further trained, and model performance is enhanced owing to the data augmentation effect. We further highlight that in DS-FL, the heterogeneity of the devices' dataset leads to ambiguous of each data sample and lowing of the training convergence. To prevent this, we propose entropy reduction averaging, where the aggregated model outputs are intentionally sharpened. Moreover, extensive experiments show that DS-FL reduces communication costs up to 99% relative to those of the FL benchmark while achieving similar or higher classification accuracy.

Communication-Efficient On-Device Machine Learning: Federated Distillation and Augmentation under Non-IID Private Data

Abstract: On-device machine learning (ML) enables the training process to exploit a massive amount of user-generated private data samples. To enjoy this benefit, inter-device communication overhead should be minimized. With this end, we propose federated distillation (FD), a distributed model training algorithm whose communication payload size is much smaller than a benchmark scheme, federated learning (FL), particularly when the model size is large. Moreover, user-generated data samples are likely to become non-IID across devices, which commonly degrades the performance compared to the case with an IID dataset. To cope with this, we propose federated augmentation (FAug), where each device collectively trains a generative model, and thereby augments its local data towards yielding an IID dataset. Empirical studies demonstrate that FD with FAug yields around 26x less communication overhead while achieving 95-98% test accuracy compared to FL.

FedGEMS: Federated Learning of Larger Server Models via Selective Knowledge Fusion

Abstract: Today data is often scattered among billions of resource-constrained edge devices with security and privacy constraints. Federated Learning (FL) has emerged as a viable solution to learn a global model while keeping data private, but the model complexity of FL is impeded by the computation resources of edge nodes. In this work, we investigate a novel paradigm to take advantage of a powerful server model to break through model capacity in FL. By selectively learning from multiple teacher clients and itself, a server model develops in-depth knowledge and transfers its knowledge back to clients in return to boost their respective performance. Our proposed framework achieves superior performance on both server and client models and provides several advantages in a unified framework, including flexibility for heterogeneous client architectures, robustness to poisoning attacks, and communication efficiency between clients and server on various image classification tasks.

Heterogeneous Ensemble Knowledge Transfer for Training Large Models in Federated Learning

Abstract: Federated learning (FL) enables edge-devices to collaboratively learn a model without disclosing their private data to a central aggregating server. Most existing FL algorithms require models of identical architecture to be deployed across the clients and server, making it infeasible to train large models due to clients' limited system resources. In this work, we propose a novel ensemble knowledge transfer method named Fed-ET in which small models (different in architecture) are trained on clients, and used to train a larger model at the server. Unlike in conventional ensemble learning, in FL the ensemble can be trained on clients' highly heterogeneous data. Cognizant of this property, Fed-ET uses a weighted consensus distillation scheme with diversity regularization that efficiently extracts reliable consensus from the ensemble while improving generalization by exploiting the diversity within the ensemble. We show the generalization bound for the ensemble of weighted models trained on heterogeneous datasets that supports the intuition of Fed-ET. Our experiments on image and language tasks show that Fed-ET significantly outperforms other state-of-the-art FL algorithms with fewer communicated parameters, and is also robust against high data-heterogeneity.

Mix2FLD: Downlink Federated Learning After Uplink Federated Distillation With Two-Way Mixup

Abstract: This letter proposes a novel communication-efficient and privacy-preserving distributed machine learning framework, coined Mix2FLD. To address uplink-downlink capacity asymmetry, local model outputs are uploaded to a server in the uplink as in federated distillation (FD), whereas global model parameters are downloaded in the downlink as in federated learning (FL). This requires a model output-to-parameter conversion at the server, after collecting additional data samples from devices. To preserve privacy while not compromising accuracy, linearly mixed-up local samples are uploaded, and inversely mixed up across different devices at the server. Numerical evaluations show that Mix2FLD achieves up to 16.7% higher test accuracy while reducing convergence time by up to 18.8% under asymmetric uplink-downlink channels compared to FL.

Wireless Federated Distillation for Distributed Edge Learning with Heterogeneous Data

Abstract: Cooperative training methods for distributed machine learning typically assume noiseless and ideal communication channels. This work studies some of the opportunities and challenges arising from the presence of wireless communication links. We specifically consider wireless implementations of Federated Learning (FL) and Federated Distillation (FD), as well as of a novel Hybrid Federated Distillation (HFD) scheme. Both digital implementations based on separate source-channel coding and over-the-air computing implementations based on joint source-channel coding are proposed and evaluated over Gaussian multiple-access channels.

Decentralized and Model-Free Federated Learning: Consensus-Based Distillation in Function Space

Abstract: This paper proposes a fully decentralized federated learning (FL) scheme for Internet of Everything (IoE) devices that are connected via multi-hop networks. Because FL algorithms hardly converge the parameters of machine learning (ML) models, this paper focuses on the convergence of ML models in function spaces. Considering that the representative loss functions of ML tasks e.g, mean squared error (MSE) and Kullback-Leibler (KL) divergence, are convex functionals, algorithms that directly update functions in function spaces could converge to the optimal solution. The key concept of this paper is to tailor a consensus-based optimization algorithm to work in the function space and achieve the global optimum in a distributed manner. This paper first analyzes the convergence of the proposed algorithm in a function space, which is referred to as a meta-algorithm, and shows that the spectral graph theory can be applied to the function space in a manner similar to that of numerical vectors. Then, consensus-based multi-hop federated distillation (CMFD) is developed for a neural network (NN) to implement the meta-algorithm. CMFD leverages knowledge distillation to realize function aggregation among adjacent devices without parameter averaging. An advantage of CMFD is that it works even with different NN models among the distributed learners. Although CMFD does not perfectly reflect the behavior of the meta-algorithm, the discussion of the meta-algorithm's convergence property promotes an intuitive understanding of CMFD, and simulation evaluations show that NN models converge using CMFD for several tasks. The simulation results also show that CMFD achieves higher accuracy than parameter aggregation for weakly connected networks, and CMFD is more stable than parameter aggregation methods.

FedZKT: Zero-Shot Knowledge Transfer towards Resource-Constrained Federated Learning with Heterogeneous On-Device Models

Abstract: Federated learning enables multiple distributed devices to collaboratively learn a shared prediction model without centralizing their on-device data. Most of the current algorithms require comparable individual efforts for local training with the same structure and size of on-device models, which, however, impedes participation from resource-constrained devices. Given the widespread yet heterogeneous devices nowadays, in this paper, we propose an innovative federated learning framework with heterogeneous on-device models through Zero-shot Knowledge Transfer, named by FedZKT. Specifically, FedZKT allows devices to independently determine the on-device models upon their local resources. To achieve knowledge transfer across these heterogeneous on-device models, a zero-shot distillation approach is designed without any prerequisites for private on-device data, which is contrary to certain prior research based on a public dataset or a pre-trained data generator. Moreover, this compute-intensive distillation task is assigned to the server to allow the participation of resource-constrained devices, where a generator is adversarially learned with the ensemble of collected on-device models. The distilled central knowledge is then sent back in the form of the corresponding on-device model parameters, which can be easily absorbed on the device side. Extensive experimental studies demonstrate the effectiveness and robustness of FedZKT towards on-device knowledge agnostic, on-device model heterogeneity, and other challenging federated learning scenarios, such as heterogeneous on-device data and straggler effects.

Federated Learning via Decentralized Dataset Distillation in Resource-Constrained Edge Environments

Abstract: In federated learning, all networked clients contribute to the model training cooperatively. However, with model sizes increasing, even sharing the trained partial models often leads to severe communication bottlenecks in underlying networks, especially when communicated iteratively. In this paper, we introduce a federated learning framework FedD3 requiring only one-shot communication by integrating dataset distillation instances. Instead of sharing model updates in other federated learning approaches, FedD3 allows the connected clients to distill the local datasets independently, and then aggregates those decentralized distilled datasets (e.g. a few unrecognizable images) from networks for model training. Our experimental results show that FedD3 significantly outperforms other federated learning frameworks in terms of needed communication volumes, while it provides the additional benefit to be able to balance the trade-off between accuracy and communication cost, depending on usage scenario or target dataset. For instance, for training an AlexNet model on CIFAR-10 with 10 clients under non-independent and identically distributed (Non-IID) setting, FedD3 can either increase the accuracy by over 71% with a similar communication volume, or save 98% of communication volume, while reaching the same accuracy, compared to other one-shot federated learning approaches.

Comparison of representative works that apply KD in FEL regarding knowledge types and requirements for public datasets.

Why Concern Knowledge Distillation in Federated Edge Learning

KD has achieved success in FL, such as realizing efficient communication {{cite:c37498a}}, {{cite:ec36766}}, enabling model heterogeneity {{cite:8a90b7e}}, {{cite:f97ebcf}} and personalization {{cite:b76d905}}, {{cite:a7764f1}} among devices, etc.
Recent trends suggest that KD has great potential to apply to various learning processes in FEL as an important tool for knowledge transfer or model collaborative training in diversity-constrained mobile edge networks.
Specifically, the technical characteristics of KD meet the core demands of FEL, and the roles it can play include but not limited to compressing large-scale edge models for on-device deployment {{cite:84f904c}}, transferring local adaptive knowledge to on-device models for personalization {{cite:ed969ca}}, {{cite:13f955e}}, {{cite:df0dc50}}, and helping establish novel FL frameworks for enabling heterogeneous device supports {{cite:48d41e4}}, {{cite:80c6f52}}, {{cite:a39b85d}}.
Representative works that apply KD in FEL are summarized in Table {{table:f48b8b59-3909-4e69-83d2-424f55e47f1c}} , {{table:60a01bd4-ca36-4970-8d9b-9f77c28b40b0}} .

As shown in Table {{table:f48b8b59-3909-4e69-83d2-424f55e47f1c}} , prior works are classified into four types according to the role of KD in the FEL: 1) Knowledge transfer, where KD is utilized to transfer knowledge from one ML model to another, occurring only at the end-side or edge-side.
2) Model representation exchange protocol, where on-device and edge models achieve collaborative optimization based on exchanged logits or features. 3) Component of backbone algorithm, where KD-based FEL algorithms are improved as the base algorithm.
4) Dataset distillation, where small datasets derived from on-device data are synthesized via KD.
In addition, the deployment modes of KD in FEL are classified into edge, end and edge-end collaboration, depending on the application location of KD and whether end-side and edge-side collaboration are required.
As shown in Table {{table:60a01bd4-ca36-4970-8d9b-9f77c28b40b0}} , some FEL approaches require public datasets, in which edge models and on-device models leverage the same shared public data during distillation to synchronize the knowledge they generate.


Practices and A Strong Baseline for Traffic Anomaly Detection

Abstract: The detection of traffic anomalies is a critical component of the intelligent city transportation management sys-tem. Previous works have proposed a variety of notable in-sights and taken a step forward in this field, however, dealing with the complex traffic environment remains a challenge. Moreover, the lack of high-quality data and the complexity of the traffic scene, motivate us to study this problem from a hand-crafted perspective. In this paper, we propose a straightforward and efficient framework that includes pre-processing, a dynamic track module, and post-processing. With video stabilization, background modeling, and vehicle detection, the pro-processing phase aims to generate candidate anomalies. The dynamic tracking module seeks and locates the start time of anomalies by utilizing vehicle motion patterns and spatiotemporal status. Finally, we use the post-processing to fine-tune the temporal boundary of anomalies. Not surprisingly, our proposed framework was ranked 1st in the NVIDIA AI CITY 2021 leaderboard for traffic anomaly detection. Codes will be available.

An Efficient Approach for Anomaly Detection in Traffic Videos

Abstract: Due to its relevance in intelligent transportation systems, anomaly detection in traffic videos has recently received much interest. It remains a difficult problem due to a variety of factors influencing the video quality of a real-time traffic feed, such as temperature, perspective, lighting conditions, and so on. Even though state-of-the-art methods perform well on the available benchmark datasets, they need a large amount of external training data as well as substantial computational resources. In this paper, we propose an efficient approach for a video anomaly detection system which is capable of running at the edge devices, e.g., on a roadside camera. The proposed approach comprises a preprocessing module that detects changes in the scene and removes the corrupted frames, a two-stage background modelling module and a two-stage object detector. Finally, a backtracking anomaly detection algorithm computes a similarity statistic and decides on the onset time of the anomaly. We also propose a sequential change detection algorithm that can quickly adapt to a new scene and detect changes in the similarity statistic. Experimental results on the Track 4 test set of the 2021 AI City Challenge show the efficacy of the proposed framework as we achieve an F1-score of 0. 9157 along with 8.4027 root mean square error (RMSE) and are ranked fourth in the competition.

A Vision-based System for Traffic Anomaly Detection using Deep Learning and Decision Trees

Abstract: Any intelligent traffic monitoring system must be able to detect anomalies such as traffic accidents in real time. In this paper, we propose a Decision-Tree enabled approach powered by deep learning for extracting anomalies from traffic cameras while accurately estimating the start and end times of the anomalous event. Our approach included creating a detection model, followed by anomaly detection and analysis. YOLOv5 served as the foundation for our detection model. The anomaly detection and analysis step entail traffic scene background estimation, road mask extraction, and adaptive thresholding. Candidate anomalies were passed through a decision tree to detect and analyze final anomalies. The proposed approach yielded an F1 score of 0.8571, and an S4 score of 0.5686, per the experimental validation.

Summary of the Track 4 leader board.

Tables {{table:72a4c2b2-d7ba-430e-9ed7-c4e4b4cc8c58}} , {{table:93f18a7d-eff9-471e-91a2-5efec0f42d1c}} , {{table:aafabcc2-7150-4619-900f-87c018a0a250}} , {{table:af295c5a-dfba-4437-bc89-0456a89732b1}}  and {{table:6a4e96a1-b681-4994-9003-1387dd3bd14f}}  summarize the leader boards for Track 1 (turn-counts for signal timing planning), Track 2 (vehicle ReID), Track 3 (city-scale MTMC vehicle tracking), Track 4 (traffic anomaly detection), and Track 5 (NL based vehicle retrieval) challenges, respectively.