Learning Continuous Control Policies by Stochastic Value Gradients

Abstract: We present a unified framework for learning continuous control policies using backpropagation. It supports stochastic control by treating stochasticity in the Bellman equation as a deterministic function of exogenous noise. The product is a spectrum of general policy gradient algorithms that range from model-free methods with value functions to model-based methods without value functions. We use learned models but only require observations from the environment in- stead of observations from model-predicted trajectories, minimizing the impact of compounded model errors. We apply these algorithms first to a toy stochastic control problem and then to several physics-based control problems in simulation. One of these variants, SVG(1), shows the effectiveness of learning models, value functions, and policies simultaneously in continuous domains.

Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models

Abstract: Model-based reinforcement learning (RL) algorithms can attain excellent sample efficiency, but often lag behind the best model-free algorithms in terms of asymptotic performance. This is especially true with high-capacity parametric function approximators, such as deep networks. In this paper, we study how to bridge this gap, by employing uncertainty-aware dynamics models. We propose a new algorithm called probabilistic ensembles with trajectory sampling (PETS) that combines uncertainty-aware deep network dynamics models with sampling-based uncertainty propagation. Our comparison to state-of-the-art model-based and model-free deep RL algorithms shows that our approach matches the asymptotic performance of model-free algorithms on several challenging benchmark tasks, while requiring significantly fewer samples (e.g., 8 and 125 times fewer samples than Soft Actor Critic and Proximal Policy Optimization respectively on the half-cheetah task).

Deep Visual Foresight for Planning Robot Motion

Abstract: A key challenge in scaling up robot learning to many skills and environments is removing the need for human supervision, so that robots can collect their own data and improve their own performance without being limited by the cost of requesting human feedback. Model-based reinforcement learning holds the promise of enabling an agent to learn to predict the effects of its actions, which could provide flexible predictive models for a wide range of tasks and environments, without detailed human supervision. We develop a method for combining deep action-conditioned video prediction models with model-predictive control that uses entirely unlabeled training data. Our approach does not require a calibrated camera, an instrumented training set-up, nor precise sensing and actuation. Our results show that our method enables a real robot to perform nonprehensile manipulation -- pushing objects -- and can handle novel objects not seen during training.

Overview of Explicit Planning/Learned Transitions Methods

Explicit Planning on Learned Transitions

Algorithm REF  shows the steps of using explicit planning
and transition learning by
backpropagation. Table {{table:c3d2d292-da9e-466f-aa30-3dd29e17a1c1}}  summarizes the approaches of
this subsection, showing both the learning and the
planning approach. Two variants of this approach are also discussed in this
subsection: hybrid imagination and latent models, see
Table {{table:264a2883-7326-4077-86c8-dc36f84fb645}}  and Table {{table:dd92a140-6771-4a81-9977-d9d0b83ba19f}} .


MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographs

Abstract: Chest radiography is an extremely powerful imaging modality, allowing for a detailed inspection of a patient's thorax, but requiring specialized training for proper interpretation. With the advent of high performance general purpose computer vision algorithms, the accurate automated analysis of chest radiographs is becoming increasingly of interest to researchers. However, a key challenge in the development of these techniques is the lack of sufficient data. Here we describe MIMIC-CXR-JPG v2.0.0, a large dataset of 377,110 chest x-rays associated with 227,827 imaging studies sourced from the Beth Israel Deaconess Medical Center between 2011 - 2016. Images are provided with 14 labels derived from two natural language processing tools applied to the corresponding free-text radiology reports. MIMIC-CXR-JPG is derived entirely from the MIMIC-CXR database, and aims to provide a convenient processed version of MIMIC-CXR, as well as to provide a standard reference for data splits and image labels. All images have been de-identified to protect patient privacy. The dataset is made freely available to facilitate and encourage a wide range of research in medical computer vision.

CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison

Abstract: Large, labeled datasets have driven deep learning methods to achieve expert-level performance on a variety of medical imaging tasks. We present CheXpert, a large dataset that contains 224,316 chest radiographs of 65,240 patients. We design a labeler to automatically detect the presence of 14 observations in radiology reports, capturing uncertainties inherent in radiograph interpretation. We investigate different approaches to using the uncertainty labels for training convolutional neural networks that output the probability of these observations given the available frontal and lateral radiographs. On a validation set of 200 chest radiographic studies which were manually annotated by 3 board-certified radiologists, we find that different uncertainty approaches are useful for different pathologies. We then evaluate our best model on a test set composed of 500 chest radiographic studies annotated by a consensus of 5 board-certified radiologists, and compare the performance of our model to that of 3 additional radiologists in the detection of 5 selected pathologies. On Cardiomegaly, Edema, and Pleural Effusion, the model ROC and PR curves lie above all 3 radiologist operating points. We release the dataset to the public as a standard benchmark to evaluate performance of chest radiograph interpretation models. The dataset is freely available at https://stanfordmlgroup.github.io/competitions/chexpert .

A summary of medical imaging datasets used in this study.

In this study, we conducted our knowledge transfer experiments on four different publicly available medical imaging datasets listed in Table {{table:8a40daf4-00ce-43b5-9544-82bf3dc09e77}} . CheXpert {{cite:7f9d2ac}}, ChestX-ray8 {{cite:af388bc}}, and MIMIC-CXR {{cite:3a032d8}} are chest X-ray images annotated for a number of diseases and Diabetic Retinopathy (Retina)https://www.kaggle.com/c/diabetic-retinopathy-detection is Retina images annotated for the diabetic scale of retinopathy. Fig. {{figure:2133d827-e8fb-475b-88c0-d623afecb804}}  shows some sample images included in these datasets.


Guide source separation with audio frequency bases learned with a framework that maps to visual objects.

Separate audio sources into components that can be localised in the video frames.

Perform independent image co-segmentation and sound source separation for not synchronised data.

Use predicted binaural audio to aid sound source separation.

Use of a multiple instance learning paradigm for separation and localisation of weakly-labeled data.

Incorporate temporal motion information and employ a curriculum learning scheme for training.

Do not separate the sounds independently to avoid that acoustic components from the original mixture get lost.

Learning to Separate Object Sounds by Watching Unlabeled Video

Abstract: Perceiving a scene most fully requires all the senses. Yet modeling how objects look and sound is challenging: most natural scenes and events contain multiple objects, and the audio track mixes all the sound sources together. We propose to learn audio-visual object models from unlabeled video, then exploit the visual context to perform audio source separation in novel videos. Our approach relies on a deep multi-instance multi-label learning framework to disentangle the audio frequency bases that map to individual visual objects, even without observing/hearing those objects in isolation. We show how the recovered disentangled bases can be used to guide audio source separation to obtain better-separated, object-level sounds. Our work is the first to learn audio source separation from large-scale "in the wild" videos containing multiple audio sources per video. We obtain state-of-the-art results on visually-aided audio source separation and audio denoising. Our video results: http://vision.cs.utexas.edu/projects/separating_object_sounds/

The Sound of Pixels

Abstract: We introduce PixelPlayer, a system that, by leveraging large amounts of unlabeled videos, learns to locate image regions which produce sounds and separate the input sounds into a set of components that represents the sound from each pixel. Our approach capitalizes on the natural synchronization of the visual and audio modalities to learn models that jointly parse sounds and images, without requiring additional manual supervision. Experimental results on a newly collected MUSIC dataset show that our proposed Mix-and-Separate framework outperforms several baselines on source separation. Qualitative results suggest our model learns to ground sounds in vision, enabling applications such as independently adjusting the volume of sound sources.

Self-supervised Audio-visual Co-segmentation

Abstract: Segmenting objects in images and separating sound sources in audio are challenging tasks, in part because traditional approaches require large amounts of labeled data. In this paper we develop a neural network model for visual object segmentation and sound source separation that learns from natural videos through self-supervision. The model is an extension of recently proposed work that maps image pixels to sounds [1]. Here, we introduce a learning approach to disentangle concepts in the neural networks, and assign semantic categories to network feature channels to enable independent image segmentation and sound source separation after audio-visual training on videos. Our evaluations show that the disentangled model outperforms several baselines in semantic segmentation and sound source separation.

2.5D Visual Sound

Abstract: Binaural audio provides a listener with 3D sound sensation, allowing a rich perceptual experience of the scene. However, binaural recordings are scarcely available and require nontrivial expertise and equipment to obtain. We propose to convert common monaural audio into binaural audio by leveraging video. The key idea is that visual frames reveal significant spatial cues that, while explicitly lacking in the accompanying single-channel audio, are strongly linked to it. Our multi-modal approach recovers this link from unlabeled video. We devise a deep convolutional neural network that learns to decode the monaural (single-channel) soundtrack into its binaural counterpart by injecting visual information about object and scene configurations. We call the resulting output 2.5D visual sound---the visual stream helps "lift" the flat single channel audio into spatialized sound. In addition to sound generation, we show the self-supervised representation learned by our network benefits audio-visual source separation. Our video results: http://vision.cs.utexas.edu/projects/2.5D_visual_sound/

Identify, Locate and Separate: Audio-Visual Object Extraction in Large Video Collections Using Weak Supervision

Abstract: We tackle the problem of audio-visual scene analysis for weakly-labeled data. To this end, we build upon our previous audio-visual representation learning framework to perform object classification in noisy acoustic environments and integrate audio source enhancement capability. This is made possible by a novel use of non-negative matrix factorization for the audio modality. Our approach is founded on the multiple instance learning paradigm. Its effectiveness is established through experiments over a challenging dataset of music instrument performance videos. We also show encouraging visual object localization results.

The Sound of Motions

Abstract: Sounds originate from object motions and vibrations of surrounding air. Inspired by the fact that humans is capable of interpreting sound sources from how objects move visually, we propose a novel system that explicitly captures such motion cues for the task of sound localization and separation. Our system is composed of an end-to-end learnable model called Deep Dense Trajectory (DDT), and a curriculum learning scheme. It exploits the inherent coherence of audio-visual signals from a large quantities of unlabeled videos. Quantitative and qualitative evaluations show that comparing to previous models that rely on visual appearance cues, our motion based system improves performance in separating musical instrument sounds. Furthermore, it separates sound components from duets of the same category of instruments, a challenging problem that has not been addressed before.

Recursive Visual Sound Separation Using Minus-Plus Net

Abstract: Sounds provide rich semantics, complementary to visual data, for many tasks. However, in practice, sounds from multiple sources are often mixed together. In this paper we propose a novel framework, referred to as MinusPlus Network (MP-Net), for the task of visual sound separation. MP-Net separates sounds recursively in the order of average energy, removing the separated sound from the mixture at the end of each prediction, until the mixture becomes empty or contains only noise. In this way, MP-Net could be applied to sound mixtures with arbitrary numbers and types of sounds. Moreover, while MP-Net keeps removing sounds with large energy from the mixture, sounds with small energy could emerge and become clearer, so that the separation is more accurate. Compared to previous methods, MP-Net obtains state-of-the-art results on two large scale datasets, across mixtures with different types and numbers of sounds.

Co-Separating Sounds of Visual Objects

Abstract: Learning how objects sound from video is challenging, since they often heavily overlap in a single audio channel. Current methods for visually-guided audio source separation sidestep the issue by training with artificially mixed video clips, but this puts unwieldy restrictions on training data collection and may even prevent learning the properties of "true" mixed sounds. We introduce a co-separation training paradigm that permits learning object-level sounds from unlabeled multi-source videos. Our novel training objective requires that the deep neural network's separated audio for similar-looking objects be consistently identifiable, while simultaneously reproducing accurate video-level audio tracks for each source training pair. Our approach disentangles sounds in realistic test videos, even in cases where an object was not observed individually during training. We obtain state-of-the-art results on visually-guided audio source separation and audio denoising for the MUSIC, AudioSet, and AV-Bench datasets.

Conditioned Source Separation for Musical Instrument Performances

Abstract: In music source separation, the number of sources may vary for each piece and some of the sources may belong to the same family of instruments, thus sharing timbral characteristics and making the sources more correlated. This leads to additional challenges in the source separation problem. This paper proposes a source separation method for multiple musical instruments sounding simultaneously and explores how much additional information apart from the audio stream can lift the quality of source separation. We explore conditioning techniques at different levels of a primary source separation network and utilize two extra modalities of data, namely presence or absence of instruments in the mixture, and the corresponding video stream data.

Music Gesture for Visual Sound Separation

Abstract: Recent deep learning approaches have achieved impressive performance on visual sound separation tasks. However, these approaches are mostly built on appearance and optical flow like motion feature representations, which exhibit limited abilities to find the correlations between audio signals and visual points, especially when separating multiple instruments of the same types, such as multiple violins in a scene. To address this, we propose "Music Gesture," a keypoint-based structured representation to explicitly model the body and finger movements of musicians when they perform music. We first adopt a context-aware graph network to integrate visual semantic context with body dynamics, and then apply an audio-visual fusion model to associate body movements with the corresponding audio signals. Experimental results on three music performance datasets show: 1) strong improvements upon benchmark metrics for hetero-musical separation tasks (i.e. different instruments); 2) new ability for effective homo-musical separation for piano, flute, and trumpet duets, which to our best knowledge has never been achieved with alternative methods. Project page: http://music-gesture.csail.mit.edu.

Visually Guided Sound Source Separation using Cascaded Opponent Filter Network

Abstract: The objective of this paper is to recover the original component signals from a mixture audio with the aid of visual cues of the sound sources. Such task is usually referred as visually guided sound source separation. The proposed Cascaded Opponent Filter (COF) framework consists of multiple stages, which recursively refine the source separation. A key element in COF is a novel opponent filter module that identifies and relocates residual components between sources. The system is guided by the appearance and motion of the source, and, for this purpose, we study different representations based on video frames, optical flows, dynamic images, and their combinations. Finally, we propose a Sound Source Location Masking (SSLM) technique, which, together with COF, produces a pixel level mask of the source location. The entire system is trained end-to-end using a large set of unlabelled videos. We compare COF with recent baselines and obtain the state-of-the-art performance in three challenging datasets (MUSIC, A-MUSIC, and A-NATURAL). Project page: https://ly-zhu.github.io/cof-net.

Leveraging Category Information for Single-Frame Visual Sound Source Separation

Abstract: Visual sound source separation aims at identifying sound components from a given sound mixture with the presence of visual cues. Prior works have demonstrated impressive results, but with the expense of large multi-stage architectures and complex data representations (e.g. optical flow trajectories). In contrast, we study simple yet efficient models for visual sound separation using only a single video frame. Furthermore, our models are able to exploit the information of the sound source category in the separation process. To this end, we propose two models where we assume that i) the category labels are available at the training time, or ii) we know if the training sample pairs are from the same or different category. The experiments with the MUSIC dataset show that our model obtains comparable or better performance compared to several recent baseline methods. The code is available at https://github.com/ly-zhu/Leveraging-Category-Information-for-Single-Frame-Visual-Sound-Source-Separation

Materials Property Prediction with Uncertainty Quantification: A Benchmark Study

Abstract: Uncertainty quantification (UQ) has increasing importance in building robust high-performance and generalizable materials property prediction models. It can also be used in active learning to train better models by focusing on getting new training data from uncertain regions. There are several categories of UQ methods each considering different types of uncertainty sources. Here we conduct a comprehensive evaluation on the UQ methods for graph neural network based materials property prediction and evaluate how they truly reflect the uncertainty that we want in error bound estimation or active learning. Our experimental results over four crystal materials datasets (including formation energy, adsorption energy, total energy, and band gap properties) show that the popular ensemble methods for uncertainty estimation is NOT the best choice for UQ in materials property prediction. For the convenience of the community, all the source code and data sets can be accessed freely at \url{https://github.com/usccolumbia/materialsUQ}.

Deep-learning-based approaches for audio-visual source separation for non-speech signals. L: Localisation.

Audio-Visual Sound Source Separation for Non-Speech Signals

Several works addressed AV source separation for non-speech signals. Similarly to other fields, classical methods {{cite:2bb2e7d}}, {{cite:a2d5aa9}}, {{cite:7b685e7}}, {{cite:0b23dca}}, {{cite:6778e7a}} were recently replaced by deep-learning-based approaches (cf. Table {{table:4ecc5f32-bf43-42a4-8443-21b8286cbf8a}} ). The first two works that concurrently proposed deep processing stages for the task under analysis were {{cite:a447e3f}} and {{cite:fbf3045}}.


IEEE Communication Surveys and Tutorials

Reviews the use of deep learning as a tool for wireless physical layer design. This is the reverse of what we cover in this paper where wireless protocol design is used as a tool to help ML.

Reviews the use of ML as a tool to address resource management for IoT networks. This is the reverse of what we cover in this paper where radio resource management design is used as a tool to help ML.

Classifies the main network architectures of Edge AI and discusses enabling technologies such as FL and other ML algorithmic.

Presents a research road-map to guide future initiatives in Edge AI. Contains some discussion on wireless for ML.

A broad survey of technologies for Edge AI including: (1) potential applications for deep learning on the edge, (2) deep learning inference, (3) edge computing hardware overview, and (4) federated learning.

Comprehensive survey of technologies for communication-efficient DML. Discusses ML training algorithms, coding techniques, model partitioning, and briefly mentions over-the-air computation.

Deep Learning for Wireless Physical Layer: Opportunities and Challenges

Abstract: Machine learning (ML) has been widely applied to the upper layers of wireless communication systems for various purposes, such as deployment of cognitive radio and communication network. However, its application to the physical layer is hampered by sophisticated channel environments and limited learning ability of conventional ML algorithms. Deep learning (DL) has been recently applied for many fields, such as computer vision and natural language processing, given its expressive capacity and convenient optimization capability. The potential application of DL to the physical layer has also been increasingly recognized because of the new features for future communications, such as complex scenarios with unknown channel models, high speed and accurate processing requirements; these features challenge conventional communication theories. This paper presents a comprehensive overview of the emerging studies on DL-based physical layer processing, including leveraging DL to redesign a module of the conventional communication system (for modulation recognition, channel decoding, and detection) and replace the communication system with a radically new architecture based on an autoencoder. These DL-based methods show promising performance improvements but have certain limitations, such as lack of solid analytical tools and use of architectures that are specifically designed for communication and implementation research, thereby motivating future research in this field.

Machine Learning for Resource Management in Cellular and IoT Networks: Potentials, Current Solutions, and Open Challenges

Abstract: Internet-of-Things (IoT) refers to a massively heterogeneous network formed through smart devices connected to the Internet. In the wake of disruptive IoT with a huge amount and variety of data, Machine Learning (ML) and Deep Learning (DL) mechanisms will play a pivotal role to bring intelligence to the IoT networks. Among other aspects, ML and DL can play an essential role in addressing the challenges of resource management in large-scale IoT networks. In this article, we conduct a systematic and in-depth survey of the ML- and DL-based resource management mechanisms in cellular wireless and IoT networks. We start with the challenges of resource management in cellular IoT and low-power IoT networks, review the traditional resource management mechanisms for IoT networks, and motivate the use of ML and DL techniques for resource management in these networks. Then, we provide a comprehensive survey of the existing ML- and DL-based resource allocation techniques in wireless IoT networks and also techniques specifically designed for HetNets, MIMO and D2D communications, and NOMA networks. To this end, we also identify the future research directions in using ML and DL for resource allocation and management in IoT networks.

Edge Intelligence: Paving the Last Mile of Artificial Intelligence With Edge Computing

Abstract: With the breakthroughs in deep learning, the recent years have witnessed a booming of artificial intelligence (AI) applications and services, spanning from personal assistant to recommendation systems to video/audio surveillance. More recently, with the proliferation of mobile computing and Internet of Things (IoT), billions of mobile and IoT devices are connected to the Internet, generating zillions bytes of data at the network edge. Driving by this trend, there is an urgent need to push the AI frontiers to the network edge so as to fully unleash the potential of the edge big data. To meet this demand, edge computing, an emerging paradigm that pushes computing tasks and services from the network core to the network edge, has been widely recognized as a promising solution. The resulted new interdiscipline, edge AI or edge intelligence (EI), is beginning to receive a tremendous amount of interest. However, research on EI is still in its infancy stage, and a dedicated venue for exchanging the recent advances of EI is highly desired by both the computer system and AI communities. To this end, we conduct a comprehensive survey of the recent research efforts on EI. Specifically, we first review the background and motivation for AI running at the network edge. We then provide an overview of the overarching architectures, frameworks, and emerging key technologies for deep learning model toward training/inference at the network edge. Finally, we discuss future research opportunities on EI. We believe that this survey will elicit escalating attentions, stimulate fruitful discussions, and inspire further research ideas on EI.

Edge Intelligence: The Confluence of Edge Computing and Artificial Intelligence

Abstract: Along with the rapid developments in communication technologies and the surge in the use of mobile devices, a brand-new computation paradigm, Edge Computing, is surging in popularity. Meanwhile, Artificial Intelligence (AI) applications are thriving with the breakthroughs in deep learning and the many improvements in hardware architectures. Billions of data bytes, generated at the network edge, put massive demands on data processing and structural optimization. Thus, there exists a strong demand to integrate Edge Computing and AI, which gives birth to Edge Intelligence. In this paper, we divide Edge Intelligence into AI for edge (Intelligence-enabled Edge Computing) and AI on edge (Artificial Intelligence on Edge). The former focuses on providing more optimal solutions to key problems in Edge Computing with the help of popular and effective AI technologies while the latter studies how to carry out the entire process of building AI models, i.e., model training and inference, on the edge. This paper provides insights into this new inter-disciplinary field from a broader perspective. It discusses the core concepts and the research road-map, which should provide the necessary background for potential future research initiatives in Edge Intelligence.

Convergence of Edge Computing and Deep Learning: A Comprehensive Survey

Abstract: Ubiquitous sensors and smart devices from factories and communities are generating massive amounts of data, and ever-increasing computing power is driving the core of computation and services from the cloud to the edge of the network. As an important enabler broadly changing people's lives, from face recognition to ambitious smart factories and cities, developments of artificial intelligence (especially deep learning, DL) based applications and services are thriving. However, due to efficiency and latency issues, the current cloud computing service architecture hinders the vision of "providing artificial intelligence for every person and every organization at everywhere". Thus, unleashing DL services using resources at the network edge near the data sources has emerged as a desirable solution. Therefore, edge intelligence, aiming to facilitate the deployment of DL services by edge computing, has received significant attention. In addition, DL, as the representative technique of artificial intelligence, can be integrated into edge computing frameworks to build intelligent edge for dynamic, adaptive edge maintenance and management. With regard to mutually beneficial edge intelligence and intelligent edge, this paper introduces and discusses: 1) the application scenarios of both; 2) the practical implementation methods and enabling technologies, namely DL training and inference in the customized edge computing framework; 3) challenges and future trends of more pervasive and fine-grained intelligence. We believe that by consolidating information scattered across the communication, networking, and DL areas, this survey can help readers to understand the connections between enabling technologies while promoting further discussions on the fusion of edge intelligence and intelligent edge, i.e., Edge DL.

Communication-Efficient Edge AI: Algorithms and Systems

Abstract: Artificial intelligence (AI) has achieved remarkable breakthroughs in a wide range of fields, ranging from speech processing, image classification to drug discovery. This is driven by the explosive growth of data, advances in machine learning (especially deep learning), and easy access to vastly powerful computing resources. Particularly, the wide scale deployment of edge devices (e.g., IoT devices) generates an unprecedented scale of data, which provides the opportunity to derive accurate models and develop various intelligent applications at the network edge. However, such enormous data cannot all be sent from end devices to the cloud for processing, due to the varying channel quality, traffic congestion and/or privacy concerns. By pushing inference and training processes of AI models to edge nodes, edge AI has emerged as a promising alternative. AI at the edge requires close cooperation among edge devices, such as smart phones and smart vehicles, and edge servers at the wireless access points and base stations, which however result in heavy communication overheads. In this paper, we present a comprehensive survey of the recent developments in various techniques for overcoming these communication challenges. Specifically, we first identify key communication challenges in edge AI systems. We then introduce communication-efficient techniques, from both algorithmic and system perspectives for training and inference tasks at the network edge. Potential future research directions are also highlighted.

Summary of related survey and overview papers written on the intersection of wireless networks and ML. These papers cover several important fields within such intersections but despite significant progress in the field, there is no survey on wireless protocol design for DML. Our survey is arguably the first attempt to address this gap.

Although the general intersection of ML and wireless communications is currently at the center of high interest and has already generated multiple surveys, there is no survey on wireless for ML yet. These surveys can roughly be classified into two categories: ML for wireless communications and communication-efficient DML, see Table {{table:586ac79e-d0f1-421d-bb79-5925b86a9c55}} . These categories are classified in Fig. {{figure:154a4940-b972-4613-9e06-64047176c920}} .


Localizing Moments in Video with Natural Language

Abstract: We consider retrieving a specific temporal segment, or moment, from a video given a natural language text description. Methods designed to retrieve whole video clips with natural language determine what occurs in a video but not when. To address this issue, we propose the Moment Context Network (MCN) which effectively localizes natural language queries in videos by integrating local and global video features over time. A key obstacle to training our MCN model is that current video datasets do not include pairs of localized video segments and referring expressions, or text descriptions which uniquely identify a corresponding moment. Therefore, we collect the Distinct Describable Moments (DiDeMo) dataset which consists of over 10,000 unedited, personal videos in diverse visual settings with pairs of localized video segments and referring expressions. We demonstrate that MCN outperforms several baseline methods and believe that our initial results together with the release of DiDeMo will inspire further research on localizing video moments with natural language.

Dense-Captioning Events in Videos

Abstract: Most natural videos contain numerous events. For example, in a video of a “man playing a piano”, the video might also contain “another man dancing” or “a crowd clapping”. We introduce the task of dense-captioning events, which involves both detecting and describing events in a video. We propose a new model that is able to identify all events in a single pass of the video while simultaneously describing the detected events with natural language. Our model introduces a variant of an existing proposal module that is designed to capture both short as well as long events that span minutes. To capture the dependencies between the events in a video, our model introduces a new captioning module that uses contextual information from past and future events to jointly describe all events. We also introduce ActivityNet Captions, a large-scale benchmark for dense-captioning events. ActivityNet Captions contains 20k videos amounting to 849 video hours with 100k total descriptions, each with its unique start and end time. Finally, we report performances of our model for dense-captioning events, video retrieval and localization.

Towards Automatic Learning of Procedures From Web Instructional Videos

Abstract: The potential for agents, whether embodied or software, to learn by observing other agents performing procedures involving objects and actions is rich. Current research on automatic procedure learning heavily relies on action labels or video subtitles, even during the evaluation phase, which makes them infeasible in real-world scenarios. This leads to our question: can the human-consensus structure of a procedure be learned from a large set of long, unconstrained videos (e.g., instructional videos from YouTube) with only visual evidence? To answer this question, we introduce the problem of procedure segmentation---to segment a video procedure into category-independent procedure segments. Given that no large-scale dataset is available for this problem, we collect a large-scale procedure segmentation dataset with procedure segments temporally localized and described; we use cooking videos and name the dataset YouCook2. We propose a segment-level recurrent network for generating procedure segments by modeling the dependencies across segments. The generated segments can be used as pre-processing for other tasks, such as dense video captioning and event parsing. We show in our experiments that the proposed model outperforms competitive baselines in procedure segmentation.

Statistics of the five datasets and four partitions used in the Video Pentathlon Challenge. Paired data for the train and val splits were made available for model development. Paired data for the public_server_val and public_server_test partitions was withheld and stored on an evaluation server. The former was provided to enable participants to sanity check their models, while the latter was used to produce the final ranking of the challenge (the challenge phases corresponding to these splits are described in Sec. ).

The statistics of the five datasets are provided in Table {{table:6633a386-65ed-4ef0-b186-b38c26d44e8a}} , together with information about the train/test partitions.


Exploit the rateless property to generate unlimited number of encoded symbol from a finite set of source symbols

Reduce the length of dot-products computed at the processing nodes by introducing sparsity to the encoded matrices

Exploit the diagonal structure of the matrices to achieve both optimal recovery threshold and optimal computation load

- Design the algebraic structure of the encoded matrices such that the MDS structure is found in both the encoded matrices and the intermediate computations. - Reconstruct the final results by solving the polynomial interpolation problem

Exploit to sparsity in both input and output matrices to reduce computation load, while achieving near optimal threshold

The entries to the columns of the encoding matrix are cyclic shifts of the entries to the first column

Use balanced mask matrix and chooses appropriate codewords from the RS codes to construct the encoding matrix

Rateless Codes for Near-Perfect Load Balancing in Distributed Matrix-Vector Multiplication

Abstract: Large-scale machine learning and data mining applications require computer systems to perform massive matrix-vector and matrix-matrix multiplication operations that need to be parallelized across multiple nodes. The presence of straggling nodes -- computing nodes that unpredictably slowdown or fail -- is a major bottleneck in such distributed computations. Ideal load balancing strategies that dynamically allocate more tasks to faster nodes require knowledge or monitoring of node speeds as well as the ability to quickly move data. Recently proposed fixed-rate erasure coding strategies can handle unpredictable node slowdown, but they ignore partial work done by straggling nodes thus resulting in a lot of redundant computation. We propose a \emph{rateless fountain coding} strategy that achieves the best of both worlds -- we prove that its latency is asymptotically equal to ideal load balancing, and it performs asymptotically zero redundant computations. Our idea is to create linear combinations of the $m$ rows of the matrix and assign these encoded rows to different worker nodes. The original matrix-vector product can be decoded as soon as slightly more than $m$ row-vector products are collectively finished by the nodes. We conduct experiments in three computing environments: local parallel computing, Amazon EC2, and Amazon Lambda, which show that rateless coding gives as much as $3\times$ speed-up over uncoded schemes.

"Short-Dot": Computing Large Linear Transforms Distributedly Using Coded Short Dot Products

Abstract: Faced with saturation of Moore's law and increasing dimension of data, system designers have increasingly resorted to parallel and distributed computing. However, distributed computing is often bottle necked by a small fraction of slow processors called "stragglers" that reduce the speed of computation because the fusion node has to wait for all processors to finish. To combat the effect of stragglers, recent literature introduces redundancy in computations across processors, e.g.,~using repetition-based strategies or erasure codes. The fusion node can exploit this redundancy by completing the computation using outputs from only a subset of the processors, ignoring the stragglers. In this paper, we propose a novel technique -- that we call "Short-Dot" -- to introduce redundant computations in a coding theory inspired fashion, for computing linear transforms of long vectors. Instead of computing long dot products as required in the original linear transform, we construct a larger number of redundant and short dot products that can be computed faster and more efficiently at individual processors. In reference to comparable schemes that introduce redundancy to tackle stragglers, Short-Dot reduces the cost of computation, storage and communication since shorter portions are stored and computed at each processor, and also shorter portions of the input is communicated to each processor. We demonstrate through probabilistic analysis as well as experiments that Short-Dot offers significant speed-up compared to existing techniques. We also derive trade-offs between the length of the dot-products and the resilience to stragglers (number of processors to wait for), for any such strategy and compare it to that achieved by our strategy.

Fundamental Limits of Coded Linear Transform

Abstract: In large scale distributed linear transform problems, coded computation plays an important role to effectively deal with "stragglers" (distributed computations that may get delayed due to few slow or faulty processors). We propose a coded computation strategy, referred to as diagonal code, that achieves the optimum recovery threshold and the optimum computation load. This is the first code that simultaneously achieves two-fold optimality in coded distributed linear transforms. Furthermore, by leveraging the idea from random proposal graph theory, we design two random codes that can guarantee optimum recovery threshold with high probability but with much less computation load. These codes provide order-wise improvement over the state-of-the-art. Moreover, the experimental results show significant improvement compared to both uncoded and existing coding schemes.

Polynomial Codes: an Optimal Design for High-Dimensional Coded Matrix Multiplication

Abstract: We consider a large-scale matrix multiplication problem where the computation is carried out using a distributed system with a master node and multiple worker nodes, where each worker can store parts of the input matrices. We propose a computation strategy that leverages ideas from coding theory to design intermediate computations at the worker nodes, in order to efficiently deal with straggling workers. The proposed strategy, named as \emph{polynomial codes}, achieves the optimum recovery threshold, defined as the minimum number of workers that the master needs to wait for in order to compute the output. Furthermore, by leveraging the algebraic structure of polynomial codes, we can map the reconstruction problem of the final output to a polynomial interpolation problem, which can be solved efficiently. Polynomial codes provide order-wise improvement over the state of the art in terms of recovery threshold, and are also optimal in terms of several other metrics. Furthermore, we extend this code to distributed convolution and show its order-wise optimality.

Coded Sparse Matrix Multiplication

Abstract: In a large-scale and distributed matrix multiplication problem $C=A^{\intercal}B$, where $C\in\mathbb{R}^{r\times t}$, the coded computation plays an important role to effectively deal with "stragglers" (distributed computations that may get delayed due to few slow or faulty processors). However, existing coded schemes could destroy the significant sparsity that exists in large-scale machine learning problems, and could result in much higher computation overhead, i.e., $O(rt)$ decoding time. In this paper, we develop a new coded computation strategy, we call \emph{sparse code}, which achieves near \emph{optimal recovery threshold}, \emph{low computation overhead}, and \emph{linear decoding time} $O(nnz(C))$. We implement our scheme and demonstrate the advantage of the approach over both uncoded and current fastest coded strategies.

Gradient Coding from Cyclic MDS Codes and Expander Graphs

Abstract: Gradient coding is a technique for straggler mitigation in distributed learning. In this paper we design novel gradient codes using tools from classical coding theory, namely, cyclic MDS codes, which compare favorably with existing solutions, both in the applicable range of parameters and in the complexity of the involved algorithms. Second, we introduce an approximate variant of the gradient coding problem, in which we settle for approximate gradient computation instead of the exact one. This approach enables graceful degradation, i.e., the $\ell_2$ error of the approximate gradient is a decreasing function of the number of stragglers. Our main result is that normalized adjacency matrices of expander graphs yield excellent approximate gradient codes, which enable significantly less computation compared to exact gradient coding, and guarantee faster convergence than trivial solutions under standard assumptions. We experimentally test our approach on Amazon EC2, and show that the generalization error of approximate gradient coding is very close to the full gradient while requiring significantly less computation from the workers.

Improving Distributed Gradient Descent Using Reed-Solomon Codes

Abstract: Today's massively-sized datasets have made it necessary to often perform computations on them in a distributed manner. In principle, a computational task is divided into subtasks which are distributed over a cluster operated by a taskmaster. One issue faced in practice is the delay incurred due to the presence of slow machines, known as stragglers. Several schemes, including those based on replication, have been proposed in the literature to mitigate the effects of stragglers and more recently, those inspired by coding theory have begun to gain traction. In this work, we consider a distributed gradient descent setting suitable for a wide class of machine learning problems. We adopt the framework of Tandon et al. [1] and present a deterministic scheme that, for a prescribed per-machine computational effort, recovers the gradient from the least number of machines $f$ theoretically permissible, via an $O(f^{2})$ decoding algorithm. The idea is based on a suitably designed Reed-Solomon code that has a sparsest and balanced generator matrix. We also provide a theoretical delay model which can be used to minimize the expected waiting time per computation by optimally choosing the parameters of the scheme. Finally, we supplement our theoretical findings with numerical results that demonstrate the efficacy of the method and its advantages over competing schemes.

Near-Optimal Straggler Mitigation for Distributed Gradient Methods

Abstract: Modern learning algorithms use gradient descent updates to train inferential models that best explain data. Scaling these approaches to massive data sizes requires proper distributed gradient descent schemes where distributed worker nodes compute partial gradients based on their partial and local data sets, and send the results to a master node where all the computations are aggregated into a full gradient and the learning model is updated. However, a major performance bottleneck that arises is that some of the worker nodes may run slow. These nodes a.k.a. stragglers can significantly slow down computation as the slowest node may dictate the overall computational time. We propose a distributed computing scheme, called Batched Coupon's Collector (BCC) to alleviate the effect of stragglers in gradient methods. We prove that our BCC scheme is robust to a near optimal number of random stragglers. We also empirically demonstrate that our proposed BCC scheme reduces the run-time by up to 85.4% over Amazon EC2 clusters when compared with other straggler mitigation strategies. We also generalize the proposed BCC scheme to minimize the completion time when implementing gradient descent-based algorithms over heterogeneous worker nodes.

Coded convolution for parallel and distributed computing within a deadline

Abstract: We consider the problem of computing the convolution of two long vectors using parallel processing units in the presence of "stragglers". Stragglers refer to the small fraction of faulty or slow processors that delays the entire computation in time-critical distributed systems. We first show that splitting the vectors into smaller pieces and using a linear code to encode these pieces provides better resilience against stragglers than replication-based schemes under a simple, worst-case straggler analysis. We then demonstrate that under commonly used models of computation time, coding can dramatically improve the probability of finishing the computation within a target "deadline" time. As opposed to the more commonly used technique of expected computation time analysis, we quantify the exponents of the probability of failure in the limit of large deadlines. Our exponent metric captures the probability of failing to finish before a specified deadline time, i.e. , the behavior of the "tail". Moreover, our technique also allows for simple closed form expressions for more general models of computation time, e.g. shifted Weibull models instead of only shifted exponentials. Thus, through this problem of coded convolution, we establish the utility of a novel asymptotic failure exponent analysis for distributed systems.

Coding techniques to mitigate the straggler effects.

Although the MDS codes proposed in {{cite:f0a0ebc}} is able to mitigate the straggler effects, it cannot be generalized to all types of computation tasks. In order to mitigate the straggler effects of different distributed computation tasks, the coding techniques can be designed by exploiting the algebraic structures of the specific operations. An important performance metric that is introduced in the proposed CDC schemes is the recovery threshold, which refers to the worst-case required number of workers the master needs to wait to recover the final result for job completion {{cite:4c8f608}}. The smaller the recovery threshold, the shorter the computation latency. The objective is to reduce the recovery threshold so that the final result can be recovered by waiting for a smaller number of workers, thus contributing to a reduction in computation latency. Here, we discuss the coding techniques for various types of computation tasks, namely (i) matrix-vector multiplications, (ii) matrix-matrix multiplications, (iii) gradient descent, (iv) convolution and Fourier transform. Table {{table:afa40bd5-bf10-432c-9033-8f11e736d8b5}}  summarizes the coding techniques designed for different distributed computation tasks.


A sequence of approximated problems are designed such that the time required to solve these problems is shorter than solving the original problem directly

Divide the sketched matrices into blocks for computations

Use Bernoulli random variables as entries of the function assignment matrix

Distribute data based on pair-wise balanced scheme and provide a rigorous convergence analysis of the proposed coding scheme

Encode both the labels and data such that the redundancy is introduced in the formulation of the optimization problems

Design neural network architectures to learn and train the encoding and decoding functions to approximate unavailable outputs

A sequential approximation framework for coded distributed optimization

Abstract: Building on the previous work of Lee et al. [2] and Ferdinand et al. [3] on coded computation, we propose a sequential approximation framework for solving optimization problems in a distributed manner. In a distributed computation system, latency caused by individual processors (“stragglers”) usually causes a significant delay in the overall process. The proposed method is powered by a sequential computation scheme, which is designed specifically for systems with stragglers. This scheme has the desirable property that the user is guaranteed to receive useful (approximate) computation results whenever a processor finishes its subtask, even in the presence of uncertain latency. In this paper, we give a coding theorem for sequentially computing matrix-vector multiplications, and the optimality of this coding scheme is also established. As an application of the results, we demonstrate solving optimization problems using a sequential approximation approach, which accelerates the algorithm in a distributed system with stragglers.

OverSketch: Approximate Matrix Multiplication for the Cloud

Abstract: We propose OverSketch, an approximate algorithm for distributed matrix multiplication in serverless computing. OverSketch leverages ideas from matrix sketching and high-performance computing to enable cost-efficient multiplication that is resilient to faults and straggling nodes pervasive in low-cost serverless architectures. We establish statistical guarantees on the accuracy of OverSketch and empirically validate our results by solving a large-scale linear program using interior-point methods and demonstrate a 34% reduction in compute time on AWS Lambda.

Approximate Gradient Coding via Sparse Random Graphs

Abstract: Distributed algorithms are often beset by the straggler effect, where the slowest compute nodes in the system dictate the overall running time. Coding-theoretic techniques have been recently proposed to mitigate stragglers via algorithmic redundancy. Prior work in coded computation and gradient coding has mainly focused on exact recovery of the desired output. However, slightly inexact solutions can be acceptable in applications that are robust to noise, such as model training via gradient-based algorithms. In this work, we present computationally simple gradient codes based on sparse graphs that guarantee fast and approximately accurate distributed computation. We demonstrate that sacrificing a small amount of accuracy can significantly increase algorithmic robustness to stragglers.

Gradient Coding via the Stochastic Block Model

Abstract: Gradient descent and its many variants, including mini-batch stochastic gradient descent, form the algorithmic foundation of modern large-scale machine learning. Due to the size and scale of modern data, gradient computations are often distributed across multiple compute nodes. Unfortunately, such distributed implementations can face significant delays caused by straggler nodes, i.e., nodes that are much slower than average. Gradient coding is a new technique for mitigating the effect of stragglers via algorithmic redundancy. While effective, previously proposed gradient codes can be computationally expensive to construct, inaccurate, or susceptible to adversarial stragglers. In this work, we present the stochastic block code (SBC), a gradient code based on the stochastic block model. We show that SBCs are efficient, accurate, and that under certain settings, adversarial straggler selection becomes as hard as detecting a community structure in the multiple community, block stochastic graph model.

Stochastic Gradient Coding for Straggler Mitigation in Distributed Learning

Abstract: We consider distributed gradient descent in the presence of stragglers. Recent work on \em gradient coding \em and \em approximate gradient coding \em have shown how to add redundancy in distributed gradient descent to guarantee convergence even if some workers are \em stragglers\em---that is, slow or non-responsive. In this work we propose an approximate gradient coding scheme called \em Stochastic Gradient Coding \em (SGC), which works when the stragglers are random. SGC distributes data points redundantly to workers according to a pair-wise balanced design, and then simply ignores the stragglers. We prove that the convergence rate of SGC mirrors that of batched Stochastic Gradient Descent (SGD) for the $\ell_2$ loss function, and show how the convergence rate can improve with the redundancy. We also provide bounds for more general convex loss functions. We show empirically that SGC requires a small amount of redundancy to handle a large number of stragglers and that it can outperform existing approximate gradient codes when the number of stragglers is large.

Redundancy Techniques for Straggler Mitigation in Distributed Optimization and Learning

Abstract: Performance of distributed optimization and learning systems is bottlenecked by "straggler" nodes and slow communication links, which significantly delay computation. We propose a distributed optimization framework where the dataset is "encoded" to have an over-complete representation with built-in redundancy, and the straggling nodes in the system are dynamically left out of the computation at every iteration, whose loss is compensated by the embedded redundancy. We show that oblivious application of several popular optimization algorithms on encoded data, including gradient descent, L-BFGS, proximal gradient under data parallelism, and coordinate descent under model parallelism, converge to either approximate or exact solutions of the original problem when stragglers are treated as erasures. These convergence results are deterministic, i.e., they establish sample path convergence for arbitrary sequences of delay patterns or distributions on the nodes, and are independent of the tail behavior of the delay distribution. We demonstrate that equiangular tight frames have desirable properties as encoding matrices, and propose efficient mechanisms for encoding large-scale data. We implement the proposed technique on Amazon EC2 clusters, and demonstrate its performance over several learning problems, including matrix factorization, LASSO, ridge regression and logistic regression, and compare the proposed method with uncoded, asynchronous, and data replication strategies.

Learning a Code: Machine Learning for Approximate Non-Linear Coded Computation

Abstract: Machine learning algorithms are typically run on large scale, distributed compute infrastructure that routinely face a number of unavailabilities such as failures and temporary slowdowns. Adding redundant computations using coding-theoretic tools called "codes" is an emerging technique to alleviate the adverse effects of such unavailabilities. A code consists of an encoding function that proactively introduces redundant computation and a decoding function that reconstructs unavailable outputs using the available ones. Past work focuses on using codes to provide resilience for linear computations and specific iterative optimization algorithms. However, computations performed for a variety of applications including inference on state-of-the-art machine learning algorithms, such as neural networks, typically fall outside this realm. In this paper, we propose taking a learning-based approach to designing codes that can handle non-linear computations. We present carefully designed neural network architectures and a training methodology for learning encoding and decoding functions that produce approximate reconstructions of unavailable computation results. We present extensive experimental results demonstrating the effectiveness of the proposed approach: we show that the our learned codes can accurately reconstruct $64 - 98\%$ of the unavailable predictions from neural-network based image classifiers on the MNIST, Fashion-MNIST, and CIFAR-10 datasets. To the best of our knowledge, this work proposes the first learning-based approach for designing codes, and also presents the first coding-theoretic solution that can provide resilience for any non-linear (differentiable) computation. Our results show that learning can be an effective technique for designing codes, and that learned codes are a highly promising approach for bringing the benefits of coding to non-linear computations.

Approximate CDC schemes to mitigate the straggler effects.

For some applications, it is not necessary to obtain exact solutions. In this subsection, we review the coding approaches that are used to derive approximate solutions for the distributed computation tasks: (i) matrix multiplication, (ii) gradient descent, (iii) non-linear computations. Table {{table:5c736f15-6c08-47da-8926-0b7add3dceb5}}  summarizes the approximate coding schemes for various distributed computation tasks.


Xception: Deep Learning with Depthwise Separable Convolutions

Abstract: We present an interpretation of Inception modules in convolutional neural networks as being an intermediate step in-between regular convolution and the depthwise separable convolution operation (a depthwise convolution followed by a pointwise convolution). In this light, a depthwise separable convolution can be understood as an Inception module with a maximally large number of towers. This observation leads us to propose a novel deep convolutional neural network architecture inspired by Inception, where Inception modules have been replaced with depthwise separable convolutions. We show that this architecture, dubbed Xception, slightly outperforms Inception V3 on the ImageNet dataset (which Inception V3 was designed for), and significantly outperforms Inception V3 on a larger image classification dataset comprising 350 million images and 17,000 classes. Since the Xception architecture has the same number of parameters as Inception V3, the performance gains are not due to increased capacity but rather to a more efficient use of model parameters.

Very Deep Convolutional Networks for Large-Scale Image Recognition

Abstract: In this work we investigate the effect of the convolutional network depth on its accuracy in the large-scale image recognition setting. Our main contribution is a thorough evaluation of networks of increasing depth using an architecture with very small (3x3) convolution filters, which shows that a significant improvement on the prior-art configurations can be achieved by pushing the depth to 16-19 weight layers. These findings were the basis of our ImageNet Challenge 2014 submission, where our team secured the first and the second places in the localisation and classification tracks respectively. We also show that our representations generalise well to other datasets, where they achieve state-of-the-art results. We have made our two best-performing ConvNet models publicly available to facilitate further research on the use of deep visual representations in computer vision.

Rethinking the Inception Architecture for Computer Vision

Abstract: Convolutional networks are at the core of most state of-the-art computer vision solutions for a wide variety of tasks. Since 2014 very deep convolutional networks started to become mainstream, yielding substantial gains in various benchmarks. Although increased model size and computational cost tend to translate to immediate quality gains for most tasks (as long as enough labeled data is provided for training), computational efficiency and low parameter count are still enabling factors for various use cases such as mobile vision and big-data scenarios. Here we are exploring ways to scale up networks in ways that aim at utilizing the added computation as efficiently as possible by suitably factorized convolutions and aggressive regularization. We benchmark our methods on the ILSVRC 2012 classification challenge validation set demonstrate substantial gains over the state of the art: 21:2% top-1 and 5:6% top-5 error for single frame evaluation using a network with a computational cost of 5 billion multiply-adds per inference and with using less than 25 million parameters. With an ensemble of 4 models and multi-crop evaluation, we report 3:5% top-5 error and 17:3% top-1 error on the validation set and 3:6% top-5 error on the official test set.

Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning

Abstract: Very deep convolutional networks have been central to the largest advances in image recognition performance in recent years. One example is the Inception architecture that has been shown to achieve very good performance at relatively low computational cost. Recently, the introduction of residual connections in conjunction with a more traditional architecture has yielded state-of-the-art performance in the 2015 ILSVRC challenge; its performance was similar to the latest generation Inception-v3 network. This raises the question: Are there any benefits to combining Inception architectures with residual connections? Here we give clear empirical evidence that training with residual connections accelerates the training of Inception networks significantly. There is also some evidence of residual Inception networks outperforming similarly expensive Inception networks without residual connections by a thin margin. We also present several new streamlined architectures for both residual and non-residual Inception networks. These variations improve the single-frame recognition performance on the ILSVRC 2012 classification task significantly. We further demonstrate how proper activation scaling stabilizes the training of very wide residual Inception networks. With an ensemble of three residual and one Inception-v4 networks, we achieve 3.08% top-5 error on the test set of the ImageNet classification (CLS) challenge.

MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications

Abstract: We present a class of efficient models called MobileNets for mobile and embedded vision applications. MobileNets are based on a streamlined architecture that uses depth-wise separable convolutions to build light weight deep neural networks. We introduce two simple global hyper-parameters that efficiently trade off between latency and accuracy. These hyper-parameters allow the model builder to choose the right sized model for their application based on the constraints of the problem. We present extensive experiments on resource and accuracy tradeoffs and show strong performance compared to other popular models on ImageNet classification. We then demonstrate the effectiveness of MobileNets across a wide range of applications and use cases including object detection, finegrain classification, face attributes and large scale geo-localization.

MobileNetV2: Inverted Residuals and Linear Bottlenecks

Abstract: In this paper we describe a new mobile architecture, MobileNetV2, that improves the state of the art performance of mobile models on multiple tasks and benchmarks as well as across a spectrum of different model sizes. We also describe efficient ways of applying these mobile models to object detection in a novel framework we call SSDLite. Additionally, we demonstrate how to build mobile semantic segmentation models through a reduced form of DeepLabv3 which we call Mobile DeepLabv3. is based on an inverted residual structure where the shortcut connections are between the thin bottleneck layers. The intermediate expansion layer uses lightweight depthwise convolutions to filter features as a source of non-linearity. Additionally, we find that it is important to remove non-linearities in the narrow layers in order to maintain representational power. We demonstrate that this improves performance and provide an intuition that led to this design. Finally, our approach allows decoupling of the input/output domains from the expressiveness of the transformation, which provides a convenient framework for further analysis. We measure our performance on ImageNet [1] classification, COCO object detection [2], VOC image segmentation [3]. We evaluate the trade-offs between accuracy, and number of operations measured by multiply-adds (MAdd), as well as actual latency, and the number of parameters.

Densely Connected Convolutional Networks

Abstract: Recent work has shown that convolutional networks can be substantially deeper, more accurate, and efficient to train if they contain shorter connections between layers close to the input and those close to the output. In this paper, we embrace this observation and introduce the Dense Convolutional Network (DenseNet), which connects each layer to every other layer in a feed-forward fashion. Whereas traditional convolutional networks with L layers have L connections - one between each layer and its subsequent layer - our network has L(L+1)/2 direct connections. For each layer, the feature-maps of all preceding layers are used as inputs, and its own feature-maps are used as inputs into all subsequent layers. DenseNets have several compelling advantages: they alleviate the vanishing-gradient problem, strengthen feature propagation, encourage feature reuse, and substantially reduce the number of parameters. We evaluate our proposed architecture on four highly competitive object recognition benchmark tasks (CIFAR-10, CIFAR-100, SVHN, and ImageNet). DenseNets obtain significant improvements over the state-of-the-art on most of them, whilst requiring less computation to achieve high performance. Code and pre-trained models are available at https://github.com/liuzhuang13/DenseNet .

Learning Transferable Architectures for Scalable Image Recognition

Abstract: Developing neural network image classification models often requires significant architecture engineering. In this paper, we study a method to learn the model architectures directly on the dataset of interest. As this approach is expensive when the dataset is large, we propose to search for an architectural building block on a small dataset and then transfer the block to a larger dataset. The key contribution of this work is the design of a new search space (the "NASNet search space") which enables transferability. In our experiments, we search for the best convolutional layer (or "cell") on the CIFAR-10 dataset and then apply this cell to the ImageNet dataset by stacking together more copies of this cell, each with their own parameters to design a convolutional architecture, named "NASNet architecture". We also introduce a new regularization technique called ScheduledDropPath that significantly improves generalization in the NASNet models. On CIFAR-10 itself, NASNet achieves 2.4% error rate, which is state-of-the-art. On ImageNet, NASNet achieves, among the published works, state-of-the-art accuracy of 82.7% top-1 and 96.2% top-5 on ImageNet. Our model is 1.2% better in top-1 accuracy than the best human-invented architectures while having 9 billion fewer FLOPS - a reduction of 28% in computational demand from the previous state-of-the-art model. When evaluated at different levels of computational cost, accuracies of NASNets exceed those of the state-of-the-art human-designed models. For instance, a small version of NASNet also achieves 74% top-1 accuracy, which is 3.1% better than equivalently-sized, state-of-the-art models for mobile platforms. Finally, the learned features by NASNet used with the Faster-RCNN framework surpass state-of-the-art by 4.0% achieving 43.1% mAP on the COCO dataset.

Pre-trained DNN models from Keras used in this paper; Type: L - linear, B - branching

In this paper, 18 DNNs as shown in Table {{table:9fee77b7-e6e9-4b89-a31c-2a46d9021e02}}  are considered. The table presents the size of a trained model and its corresponding weights, the total number of layers in the DNN (including input and output layers), the number of valid points for partitioning, and the type of the DNN. These models are explored in the context of Kerashttps://keras.io, an open source neural network library that runs on TensorFlowhttps://www.tensorflow.org/. These models are trained on the ImageNet database {{cite:9926f1c}}.

On the other hand, in a branching DNN, a layer may be connected to more than two layers which results in parallel paths between the first and last layers.
Partitioning a model in a parallel region can lead to synchronization issues and may add additional communication overhead as multiple metadata outputs will need to be transferred from one resource to another {{cite:c0bba54}}. Therefore, layers within a branch are grouped together as a block of layers and treated as a single entity. This reduces the number of partitioning points (for example, the ResNet50 DNN has 177 layers, but only 23 valid partition points as shown in Table {{table:9fee77b7-e6e9-4b89-a31c-2a46d9021e02}} ). Figure {{figure:f9a7bac6-0601-4d66-a42a-05270096eefe}}  shows the execution time and the output data size of the 25 different entities (layers are identified by name, blocks are identified by the layer numbers) of ResNet50, an example branching model. Again the variable execution time and output data size of layers/blocks are noted.
Figure {{figure:1d1f50dd-63ea-49dd-ad85-33290d99576d}}  shows an example of a branching DNN that is distributed across the device, edge and cloud. Layers 2-5 are considered as a single block.

Step 3: Benchmark each layer/block on target hardware resources. In this step, given a set of target hardware resources, such as the device, edge, or cloud, each layer/block is benchmarked five times. The average execution time along with the size of the output data is recorded. In this article, the 18 DNNs shown in Table {{table:9fee77b7-e6e9-4b89-a31c-2a46d9021e02}}  are considered.

Table {{table:7652ecb7-72e5-4913-bda6-ae1cc9a6eb1b}}  shows the execution time of five sample DNNs (from Table {{table:9fee77b7-e6e9-4b89-a31c-2a46d9021e02}} ) when natively executed on the device, edge and cloud for a single input image of size 150KB. All experiments in this paper use a 150KB size input image unless otherwise stated. DNNs executing on the device have a higher execution time than on the edge and cloud. MobileNetV2 is the DNN that executes fastest on the device. Of both the edge resources available, it is determined via benchmarking that Edge (1) produces marginally better execution times. The cloud resource with the GPU executes the fastest.


Gradient quantization by quantizing each element of gradient vectors to a finite-bit low precision value

Gradient sparsification by transmitting only some values of the gradient vectors

First train a base network (teacher network) on a base dataset and task and then transfer the learned features to a second target network (student network) to be trained on a target dataset and task

The transition from generality to specificity

Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training

Abstract: Large-scale distributed training requires significant communication bandwidth for gradient exchange that limits the scalability of multi-node training, and requires expensive high-bandwidth network infrastructure. The situation gets even worse with distributed training on mobile devices (federated learning), which suffers from higher latency, lower throughput, and intermittent poor connections. In this paper, we find 99.9% of the gradient exchange in distributed SGD is redundant, and propose Deep Gradient Compression (DGC) to greatly reduce the communication bandwidth. To preserve accuracy during compression, DGC employs four methods: momentum correction, local gradient clipping, momentum factor masking, and warm-up training. We have applied Deep Gradient Compression to image classification, speech recognition, and language modeling with multiple datasets including Cifar10, ImageNet, Penn Treebank, and Librispeech Corpus. On these scenarios, Deep Gradient Compression achieves a gradient compression ratio from 270x to 600x without losing accuracy, cutting the gradient size of ResNet-50 from 97MB to 0.35MB, and for DeepSpeech from 488MB to 0.74MB. Deep gradient compression enables large-scale distributed training on inexpensive commodity 1Gbps Ethernet and facilitates distributed training on mobile.

Not Just Privacy: Improving Performance of Private Deep Learning in Mobile Cloud

Abstract: The increasing demand for on-device deep learning services calls for a highly efficient manner to deploy deep neural networks (DNNs) on mobile devices with limited capacity. The cloud-based solution is a promising approach to enabling deep learning applications on mobile devices where the large portions of a DNN are offloaded to the cloud. However, revealing data to the cloud leads to potential privacy risk. To benefit from the cloud data center without the privacy risk, we design, evaluate, and implement a cloud-based framework ARDEN which partitions the DNN across mobile devices and cloud data centers. A simple data transformation is performed on the mobile device, while the resource-hungry training and the complex inference rely on the cloud data center. To protect the sensitive information, a lightweight privacy-preserving mechanism consisting of arbitrary data nullification and random noise addition is introduced, which provides strong privacy guarantee. A rigorous privacy budget analysis is given. Nonetheless, the private perturbation to the original data inevitably has a negative impact on the performance of further inference on the cloud side. To mitigate this influence, we propose a noisy training method to enhance the cloud-side network robustness to perturbed data. Through the sophisticated design, ARDEN can not only preserve privacy but also improve the inference performance. To validate the proposed ARDEN, a series of experiments based on three image datasets and a real mobile application are conducted. The experimental results demonstrate the effectiveness of ARDEN. Finally, we implement ARDEN on a demo system to verify its practicality.

Technologies for distributed DNN training at the edge

In this subsection, we review the enabling technologies for improving one or more of the aforementioned key performance indicators when training edge intelligence model. Table {{table:756f5c5d-fabf-43da-86be-d63dc4a780ee}}  summarizes the highlights of each enabling technology.


Mapping instances from two domains into a new data space with better similarity.

Use adversarial technology to find transferable features that both suitable for two domains.

Deep Domain Confusion: Maximizing for Domain Invariance

Abstract: Recent reports suggest that a generic supervised deep CNN model trained on a large-scale dataset reduces, but does not remove, dataset bias on a standard benchmark. Fine-tuning deep models in a new domain can require a significant amount of data, which for many applications is simply not available. We propose a new CNN architecture which introduces an adaptation layer and an additional domain confusion loss, to learn a representation that is both semantically meaningful and domain invariant. We additionally show that a domain confusion metric can be used for model selection to determine the dimension of an adaptation layer and the best position for the layer in the CNN architecture. Our proposed adaptation method offers empirical performance which exceeds previously published results on a standard benchmark visual domain adaptation task.

Domain-Adversarial Neural Networks

Abstract: We introduce a new representation learning algorithm suited to the context of domain adaptation, in which data at training and test time come from similar but different distributions. Our algorithm is directly inspired by theory on domain adaptation suggesting that, for effective domain transfer to be achieved, predictions must be made based on a data representation that cannot discriminate between the training (source) and test (target) domains. We propose a training objective that implements this idea in the context of a neural network, whose hidden layer is trained to be predictive of the classification task, but uninformative as to the domain of the input. Our experiments on a sentiment analysis classification benchmark, where the target domain data available at training time is unlabeled, show that our neural network for domain adaption algorithm has better performance than either a standard neural network or an SVM, even if trained on input features extracted with the state-of-the-art marginalized stacked denoising autoencoders of Chen et al. (2012).

Categorizing of deep transfer learning.

Deep transfer learning studies how to utilize knowledge from other fields by deep neural networks. Since deep neural networks have become popular in various fields, a considerable amount of deep transfer learning methods have been proposed that it is very important to classify and summarize them.
Based on the techniques used in deep transfer learning, this paper classifies deep transfer learning into four categories: instances-based deep transfer learning, mapping-based deep transfer learning, network-based deep transfer learning, and adversarial-based deep transfer learning, which are shown in Table {{table:372f944e-a452-4982-bd0b-2e226e07309d}} .


Creating channel differences artificially

A Minorization-Maximization Method for Optimizing Sum Rate in the Downlink of Non-Orthogonal Multiple Access Systems

Abstract: Non-orthogonal multiple access (NOMA) systems have the potential to deliver higher system throughput, compared with contemporary orthogonal multiple access techniques. For a linearly precoded multiple-input single-output (MISO) system, we study the downlink sum rate maximization problem, when the NOMA principle is applied. Being a non-convex and intractable optimization problem, we resort to approximate it with a minorization-maximization algorithm (MMA), which is a widely used tool in statistics. In each step of the MMA, we solve a second-order cone program, such that the feasibility set in each step contains that of the previous one, and is always guaranteed to be a subset of the feasibility set of the original problem. It should be noted that the algorithm takes a few iterations to converge. Furthermore, we study the conditions under which the achievable rates maximization can be further simplified to a low complexity design problem, and we compute the probability of occurrence of this event. Numerical examples are conducted to show a comparison of the proposed approach against conventional multiple access systems.

The Application of MIMO to Non-Orthogonal Multiple Access

Abstract: This paper considers the application of multiple-input multiple-output (MIMO) techniques to nonorthogonal multiple access (NOMA) systems. A new design of precoding and detection matrices for MIMO-NOMA is proposed and its performance is analyzed for the case with a fixed set of power allocation coefficients. To further improve the performance gap between MIMO-NOMA and conventional orthogonal multiple access schemes, user pairing is applied to NOMA and its impact on the system performance is characterized. More sophisticated choices of power allocation coefficients are also proposed to meet various quality-of-service requirements. Finally, computer simulation results are provided to facilitate the performance evaluation of MIMO-NOMA and also demonstrate the accuracy of the developed analytical results.

A General MIMO Framework for NOMA Downlink and Uplink Transmission Based on Signal Alignment

Abstract: The application of multiple-input multiple-output (MIMO) techniques to non-orthogonal multiple access (NOMA) systems is important to enhance the performance gains of NOMA. In this paper, a novel MIMO-NOMA framework for downlink and uplink transmission is proposed by applying the concept of signal alignment. By using stochastic geometry, closed-form analytical results are developed to facilitate the performance evaluation of the proposed framework for randomly deployed users and interferers. The impact of different power allocation strategies, such as fixed power allocation and cognitive radio inspired power allocation, on the performance of MIMO-NOMA is also investigated. Computer simulation results are provided to demonstrate the performance of the proposed framework and the accuracy of the developed analytical results.

MIMO-NOMA Design for Small Packet Transmission in the Internet of Things

Abstract: A feature of the Internet of Things (IoT) is that some users in the system need to be served quickly for small packet transmission. To address this requirement, a new multiple-input multiple-output non-orthogonal multiple access (MIMO-NOMA) scheme is designed in this paper, where one user is served with its quality of service (QoS) requirement strictly met, and the other user is served opportunistically by using the NOMA concept. The novelty of this new scheme is that it confronts the challenge that most existing MIMO-NOMA schemes rely on the assumption that users' channel conditions are different, a strong assumption which may not be valid in practice. The developed precoding and detection strategies can effectively create a significant difference between the users' effective channel gains, and therefore the potential of NOMA can be realized even if the users' original channel conditions are similar. Analytical and numerical results are provided to demonstrate the performance of the proposed MIMO-NOMA scheme.

Design of Massive-MIMO-NOMA with Limited Feedback

Abstract: In this letter, a low-feedback non-orthogonal multiple access (NOMA) scheme using massive multiple-input multiple-output (MIMO) transmission is proposed. In particular, the proposed scheme can decompose a massive-MIMO-NOMA system into multiple separated single-input single-output NOMA channels, and analytical results are developed to evaluate the performance of the proposed scheme for two scenarios, with perfect user ordering and with one-bit feedback, respectively.

Enhancing the Physical Layer Security of Non-Orthogonal Multiple Access in Large-Scale Networks

Abstract: This paper investigates the physical layer security of non-orthogonal multiple access (NOMA) in large-scale networks with invoking stochastic geometry. Both single-antenna and multiple-antenna aided transmission scenarios are considered, where the base station (BS) communicates with randomly distributed NOMA users. In the single-antenna scenario, we adopt a protected zone around the BS to establish an eavesdropper-exclusion area with the aid of careful channel ordering of the NOMA users. In the multiple-antenna scenario, artificial noise is generated at the BS for further improving the security of a beamforming-aided system. In order to characterize the secrecy performance, we derive new exact expressions of the security outage probability for both single-antenna and multiple-antenna aided scenarios. For the single-antenna scenario, we perform secrecy diversity order analysis of the selected user pair. The analytical results derived demonstrate that the secrecy diversity order is determined by the specific user having the worse channel condition among the selected user pair. For the multiple-antenna scenario, we derive the asymptotic secrecy outage probability, when the number of transmit antennas tends to infinity. Monte Carlo simulations are provided for verifying the analytical results derived and to show that: 1) the security performance of the NOMA networks can be improved by invoking the protected zone and by generating artificial noise at the BS and 2) the asymptotic secrecy outage probability is close to the exact secrecy outage probability.

Important contributions on multi-antenna aided NOMA. “DL" and “UL" represent downlink and uplink, respectively. “BF" and “OP" represent beamforming and outage probability, respectively.“SU" and “MU" represent two-user and multi-user cases. The “sum-rate gain" implies that the gain brought by invoking NOMA technique over conventional OMA technique in terms of sum-rate.

NOMA Combined with Multiple Antennas Techniques

Multiple antenna techniques are of significant importance, since they offer the extra dimension of the spatial domain, for further performance improvements. The application of multiple antenna techniques in NOMA has attached substantial interest both from academia {{cite:70126e1}}, {{cite:30e83d7}}, {{cite:f85ddb2}}, {{cite:c76daaf}}, {{cite:6416ec3}}, {{cite:ab06230}}, {{cite:23b7684}}, {{cite:0ba5431}}, {{cite:09c78eb}}, {{cite:2e9a223}}, {{cite:52d95c4}}, {{cite:833861e}}, {{cite:6dbc550}}, {{cite:43b3973}}, {{cite:d2d361f}} and from industry {{cite:bd90820}}, {{cite:70126e1}}, {{cite:1a750c5}}, {{cite:ce18f5a}}. The distinct NOMA features such as channel ordering and PA inevitably require special attention in the context of multiple antennas. More specifically, in contrast to the SISO-NOMA scenarios whose channels
are all scalars, the channels of MIMO-NOMA scenarios are represented in form of matrices, which makes the power-based ordering of users rather challenging. As a consequence,
conceiving an appropriate beamforming/precoding design is essential for multi-antenna aided NOMA systems. NOMA relying on beamforming (BF) constitutes an efficient technique of improving the bandwidth efficiency by exploiting both the power domain and the angular domain. There are two popular MIMO-NOMA designs, namely the 1) Cluster-based (CB)
MIMO-NOMA design; and the 2) Beamformer-based (BB) MIMO-NOMA design, which will be introduced in the following.
Table {{table:5516ab32-ae40-48ef-b998-868e8382c897}}  summarizes some of the existing contributions on NOMA applying multiple antennas and illustrates their comparison.


Domain Generalization for Object Recognition with Multi-task Autoencoders

Abstract: The problem of domain generalization is to take knowledge acquired from a number of related domains, where training data is available, and to then successfully apply it to previously unseen domains. We propose a new feature learning algorithm, Multi-Task Autoencoder (MTAE), that provides good generalization performance for cross-domain object recognition. The algorithm extends the standard denoising autoencoder framework by substituting artificially induced corruption with naturally occurring inter-domain variability in the appearance of objects. Instead of reconstructing images from noisy versions, MTAE learns to transform the original image into analogs in multiple related domains. It thereby learns features that are robust to variations across domains. The learnt features are then used as inputs to a classifier. We evaluated the performance of the algorithm on benchmark image recognition datasets, where the task is to learn features from multiple datasets and to then predict the image label from unseen datasets. We found that (denoising) MTAE outperforms alternative autoencoder-based models as well as the current state-of-the-art algorithms for domain generalization.

Deeper, Broader and Artier Domain Generalization

Abstract: The problem of domain generalization is to learn from multiple training domains, and extract a domain-agnostic model that can then be applied to an unseen domain. Domain generalization (DG) has a clear motivation in contexts where there are target domains with distinct characteristics, yet sparse data for training. For example recognition in sketch images, which are distinctly more abstract and rarer than photos. Nevertheless, DG methods have primarily been evaluated on photo-only benchmarks focusing on alleviating the dataset bias where both problems of domain distinctiveness and data sparsity can be minimal. We argue that these benchmarks are overly straightforward, and show that simple deep learning baselines perform surprisingly well on them. In this paper, we make two main contributions: Firstly, we build upon the favorable domain shift-robust properties of deep learning methods, and develop a low-rank parameterized CNN model for end-to-end DG learning. Secondly, we develop a DG benchmark dataset covering photo, sketch, cartoon and painting domains. This is both more practically relevant, and harder (bigger domain shift) than existing benchmarks. The results show that our method outperforms existing DG alternatives, and our dataset provides a more significant DG challenge to drive future research.

Deep Hashing Network for Unsupervised Domain Adaptation

Abstract: In recent years, deep neural networks have emerged as a dominant machine learning tool for a wide variety of application domains. However, training a deep neural network requires a large amount of labeled data, which is an expensive process in terms of time, labor and human expertise. Domain adaptation or transfer learning algorithms address this challenge by leveraging labeled data in a different, but related source domain, to develop a model for the target domain. Further, the explosive growth of digital data has posed a fundamental challenge concerning its storage and retrieval. Due to its storage and retrieval efficiency, recent years have witnessed a wide application of hashing in a variety of computer vision applications. In this paper, we first introduce a new dataset, Office-Home, to evaluate domain adaptation algorithms. The dataset contains images of a variety of everyday objects from multiple domains. We then propose a novel deep learning framework that can exploit labeled source data and unlabeled target data to learn informative hash codes, to accurately classify unseen target data. To the best of our knowledge, this is the first research effort to exploit the feature learning capabilities of deep neural networks to learn representative hash codes to address the domain adaptation problem. Our extensive empirical studies on multiple transfer tasks corroborate the usefulness of the framework in learning efficient hash codes which outperform existing competitive baselines for unsupervised domain adaptation.

The table presents the statistics of datasets we use for evaluation.

Our work focuses on the DG in deep neural networks, and for pre-deep learning efforts, we refer the reader to the review by Moreno et al. {{cite:245cb6c}}. Furthermore, we limit our discussion to DG in image classification. TDG formulation involves learning from multiple domains and testing on an unseen domain. TDG on image classification is commonly evaluated on six datasets: {{cite:9f91971}}, {{cite:7e32869}}, {{cite:a4237d3}}, {{cite:c9b122b}}, {{cite:5f464cb}}, {{cite:732266e}}. The details of these datasets are illustrated in Table {{table:ce4c35fb-eeb5-41e9-9d73-8ed08cde6575}} . We perform experiments on all these datasets.

Exploiting IID generalization methods for TDG

We compare the optimized ERM baseline against the top-performing methods on DomainBed, on six different datasets (Table {{table:ce4c35fb-eeb5-41e9-9d73-8ed08cde6575}} ). Outside of DomainBed, we also use the multi-branch reverse-gradient (GRL) {{cite:52f1441}} model on the Inception-ResNet backbone. We present the results of other methods in their original form. We also train all the methods on top of the best performing ERM baseline (Inception-Resnet backbone) and present the results in the supplementary material.


Wiki-CS: A Wikipedia-Based Benchmark for Graph Neural Networks

Abstract: We present Wiki-CS, a novel dataset derived from Wikipedia for benchmarking Graph Neural Networks. The dataset consists of nodes corresponding to Computer Science articles, with edges based on hyperlinks and 10 classes representing different branches of the field. We use the dataset to evaluate semi-supervised node classification and single-relation link prediction models. Our experiments show that these methods perform well on a new domain, with structural properties different from earlier benchmarks. The dataset is publicly available, along with the implementation of the data pipeline and the benchmark experiments, at this https URL .

Image-based Recommendations on Styles and Substitutes

Abstract: Humans inevitably develop a sense of the relationships between objects, some of which are based on their appearance. Some pairs of objects might be seen as being alternatives to each other (such as two pairs of jeans), while others may be seen as being complementary (such as a pair of jeans and a matching shirt). This information guides many of the choices that people make, from buying clothes to their interactions with each other. We seek here to model this human sense of the relationships between objects based on their appearance. Our approach is not based on fine-grained modeling of user annotations but rather on capturing the largest dataset possible and developing a scalable method for uncovering human notions of the visual relationships within. We cast this as a network inference problem defined on graphs of related images, and provide a large-scale dataset for the training and evaluation of the same. The system we develop is capable of recommending which clothes and accessories will go well together (and which will not), amongst a host of other applications.

Datasets used for evaluation.

This section describes the datasets and the experiments conducted to evaluate the efficacy of our approach.
We choose eight benchmark datasets including Wikipedia articles (WikiCS) {{cite:111defc}}, Amazon co-purchase data networks (AmazonPhoto and AmazonComputers {{cite:c6c48cd}}), extracts from Microsoft Academic Graph (CoauthorCS and CoauthorPhysics) {{cite:49234fd}}, and citation networks (Cora, CiteSeer and PubMed) {{cite:8438748}}.
The basic characteristics of these datasets are briefed in table {{table:621f2b3f-2234-4915-8c1b-adb20d6403f3}} .
We first report the results for link prediction.
The network embedding learned by BGAE and BVGAE is unsupervised as no node labels are used during training.
Hence, to measure the quality of the network embedding, we analyze our approach for two downstream tasks: clustering and transductive node classification.
For the interested readers, the supplementary material contains a detailed analysis of BGAE and BVGAE in different settings.
We use the AWS EC2 instance type g4dn.4xlarge with 16GB GPU for training.
For reproducibility, the implementation details of all the experiments along with the code are provided in {{cite:9a54546}}.
For all the experiments, we report publicly available results from our competitors.