N, S, H, A, Sr, F, D and 19 other classes

Context Based Emotion Recognition Using EMOTIC Dataset

Abstract: In our everyday lives and social interactions we often try to perceive the emotional states of people. There has been a lot of research in providing machines with a similar capacity of recognizing emotions. From a computer vision perspective, most of the previous efforts have been focusing in analyzing the facial expressions and, in some cases, also the body pose. Some of these methods work remarkably well in specific settings. However, their performance is limited in natural, unconstrained environments. Psychological studies show that the scene context, in addition to facial expression and body pose, provides important information to our perception of people's emotions. However, the processing of the context for automatic emotion recognition has not been explored in depth, partly due to the lack of proper data. In this paper we present EMOTIC, a dataset of images of people in a diverse set of natural situations, annotated with their apparent emotion. The EMOTIC dataset combines two different types of emotion representation: (1) a set of 26 discrete categories, and (2) the continuous dimensions Valence, Arousal, and Dominance. We also present a detailed statistical and algorithmic analysis of the dataset along with annotators’ agreement analysis. Using the EMOTIC dataset we train different CNN models for emotion recognition, combining the information of the bounding box containing the person with the contextual information extracted from the scene. Our results show how scene context provides important information to automatically recognize emotional states and motivate further research in this direction.

Context-Aware Emotion Recognition Networks

Abstract: Traditional techniques for emotion recognition have focused on the facial expression analysis only, thus providing limited ability to encode context that comprehensively represents the emotional responses. We present deep networks for context-aware emotion recognition, called CAER-Net, that exploit not only human facial expression but also context information in a joint and boosting manner. The key idea is to hide human faces in a visual scene and seek other contexts based on an attention mechanism. Our networks consist of two sub-networks, including two-stream encoding networks to separately extract the features of face and context regions, and adaptive fusion networks to fuse such features in an adaptive fashion. We also introduce a novel benchmark for context-aware emotion recognition, called CAER, that is appropriate than existing benchmarks both qualitatively and quantitatively. On several benchmarks, CAER-Net proves the effect of context for emotion recognition. Our dataset is available at http://caer-dataset.github.io.

EmotiCon: Context-Aware Multimodal Emotion Recognition Using Frege’s Principle

Abstract: We present EmotiCon, a learning-based algorithm for context-aware perceived human emotion recognition from videos and images. Motivated by Frege's Context Principle from psychology, our approach combines three interpretations of context for emotion recognition. Our first interpretation is based on using multiple modalities (e.g.faces and gaits) for emotion recognition. For the second interpretation, we gather semantic context from the input image and use a self-attention-based CNN to encode this information. Finally, we use depth maps to model the third interpretation related to socio-dynamic interactions and proximity among agents. We demonstrate the efficiency of our network through experiments on EMOTIC, a benchmark dataset. We report an Average Precision (AP) score of 35.48 across 26 classes, which is an improvement of 7-8 over prior methods. We also introduce a new dataset, GroupWalk, which is a collection of videos captured in multiple real-world settings of people walking. We report an AP of 65.83 across 4 categories on GroupWalk, which is also an improvement over prior methods.

Summary of the various emotion recognition datasets, withN representing Neutral, S representing Sadness, Sr representing Surprise, H representing Happiness, F representing Fear, A representing Anger, B representing Boredom, P representing Puzzlement, Ax representing Anxiety, C representing Contempt, and D representing Disgust.

In the present body of literature, a plethora of datasets have been proposed for the task of emotion recognition. Table {{table:39c1a316-02d5-4d37-ad89-1cb1bbcabb66}}  presents a comprehensive summary of the datasets employed in our work, highlighting their unique characteristics. For instance, FER-2013 {{cite:725080c}} solely consists of facial images, without any accompanying information on posture, gait, or situational context. The Emotic {{cite:f64e948}} dataset, on the other hand, contains images with the face, upper body, and background. The CAER and CAER-S {{cite:832bca9}} datasets encompass video clips and images from TV shows, primarily focusing on upper body and facial images with background. The FABO {{cite:5bf4a44}} dataset comprises acted emotion videos depicting the face and upper body against a green screen. The EWalk {{cite:2f59768}} dataset is composed of acted videos featuring walking individuals with gaits and annotated emotions. Lastly, the GroupWalk {{cite:295d618}} dataset contains 45 videos of real-world scenarios, encompassing face, posture, and gait information. This dataset also provides situational intelligence in the form of background and other people's emotions.

We leveraged the images from the datasets enlisted in Table {{table:39c1a316-02d5-4d37-ad89-1cb1bbcabb66}}  to train our facial expression recognition module. Subsequently, we evaluated its efficacy in recognizing emotions from facial expressions by benchmarking its accuracy against various state-of-the-art approaches proposed in the recent literature. The test accuracy of our method is computed as follows:
{{formula:e8d485df-12fa-41c8-8c5a-8ecedf19f2f2}}

We analyze the accuracy of each module of EMERSK in a stand-alone manner, and in various combinations with other modules. Tests are performed with the datasets listed in table {{table:39c1a316-02d5-4d37-ad89-1cb1bbcabb66}} . The resulting outputs from these experiments are shown in table {{table:6543898f-e4cd-4981-a7f0-f72a66aa3769}} . We can see the face is most the effective source of information for emotion recognition. Adding background context improves accuracy but its effect reduces when more modes are used as multiple modes already give a good picture of the subject's emotion. Maximum accuracy is achieved when we use all three modes and contexts. Therefore, to achieve maximum accuracy we should always strive for using all the available data modes.


Abstract: The problem of session-based recommendation aims to predict user actions based on anonymous sessions. Previous methods model a session as a sequence and estimate user representations besides item representations to make recommendations. Though achieved promising results, they are insufficient to obtain accurate user vectors in sessions and neglect complex transitions of items. To obtain accurate item embedding and take complex transitions of items into account, we propose a novel method, i.e. Session-based Recommendation with Graph Neural Networks, SR-GNN for brevity. In the proposed method, session sequences are modeled as graphstructured data. Based on the session graph, GNN can capture complex transitions of items, which are difficult to be revealed by previous conventional sequential methods. Each session is then represented as the composition of the global preference and the current interest of that session using an attention network. Extensive experiments conducted on two real datasets show that SR-GNN evidently outperforms the state-of-the-art session-based recommendation methods consistently.

TAGNN: Target Attentive Graph Neural Networks for Session-based Recommendation

Abstract: Session-based recommendation nowadays plays a vital role in many websites, which aims to predict users' actions based on anonymous sessions. There have emerged many studies that model a session as a sequence or a graph via investigating temporal transitions of items in a session. However, these methods compress a session into one fixed representation vector without considering the target items to be predicted. The fixed vector will restrict the representation ability of the recommender model, considering the diversity of target items and users' interests. In this paper, we propose a novel target attentive graph neural network (TAGNN) model for session-based recommendation. In TAGNN, target-aware attention adaptively activates different user interests with respect to varied target items. The learned interest representation vector varies with different target items, greatly improving the expressiveness of the model. Moreover, TAGNN harnesses the power of graph neural networks to capture rich item transitions in sessions. Comprehensive experiments conducted on real-world datasets demonstrate its superiority over state-of-the-art methods.

Rethinking the Item Order in Session-based Recommendation with Graph Neural Networks

Abstract: Predicting a user's preference in a short anonymous interaction session instead of long-term history is a challenging problem in the real-life session-based recommendation, e.g., e-commerce and media stream. Recent research of the session-based recommender system mainly focuses on sequential patterns by utilizing the attention mechanism, which is straightforward for the session's natural sequence sorted by time. However, the user's preference is much more complicated than a solely consecutive time pattern in the transition of item choices. In this paper, therefore, we study the item transition pattern by constructing a session graph and propose a novel model which collaboratively considers the sequence order and the latent order in the session graph for a session-based recommender system. We formulate the next item recommendation within the session as a graph classification problem. Specifically, we propose a weighted attention graph layer and a Readout function to learn embeddings of items and sessions for the next item recommendation. Extensive experiments have been conducted on two benchmark E-commerce datasets, Yoochoose and Diginetica, and the experimental results show that our model outperforms other state-of-the-art methods.

Beyond Clicks: Modeling Multi-Relational Item Graph for Session-Based Target Behavior Prediction

Abstract: Session-based target behavior prediction aims to predict the next item to be interacted with specific behavior types (e.g., clicking). Although existing methods for session-based behavior prediction leverage powerful representation learning approaches to encode items’ sequential relevance in a low-dimensional space, they suffer from several limitations. Firstly, they focus on only utilizing the same type of user behavior for prediction, but ignore the potential of taking other behavior data as auxiliary information. This is particularly crucial when the target behavior is sparse but important (e.g., buying or sharing an item). Secondly, item-to-item relations are modeled separately and locally in one behavior sequence, and they lack a principled way to globally encode these relations more effectively. To overcome these limitations, we propose a novel Multi-relational Graph Neural Network model for Session-based target behavior Prediction, namely MGNN-SPred for short. Specifically, we build a Multi-Relational Item Graph (MRIG) based on all behavior sequences from all sessions, involving target and auxiliary behavior types. Based on MRIG, MGNN-SPred learns global item-to-item relations and further obtains user preferences w.r.t. current target and auxiliary behavior sequences, respectively. In the end, MGNN-SPred leverages a gating mechanism to adaptively fuse user representations for predicting next item interacted with target behavior. The extensive experiments on two real-world datasets demonstrate the superiority of MGNN-SPred by comparing with state-of-the-art session-based prediction methods, validating the benefits of leveraging auxiliary behavior and learning item-to-item relations over MRIG.

GAG: Global Attributed Graph Neural Network for Streaming Session-based Recommendation

Abstract: Streaming session-based recommendation (SSR) is a challenging task that requires the recommender system to do the session-based recommendation (SR) in the streaming scenario. In the real-world applications of e-commerce and social media, a sequence of user-item interactions generated within a certain period are grouped as a session, and these sessions consecutively arrive in the form of streams. Most of the recent SR research has focused on the static setting where the training data is first acquired and then used to train a session-based recommender model. They need several epochs of training over the whole dataset, which is infeasible in the streaming setting. Besides, they can hardly well capture long-term user interests because of the neglect or the simple usage of the user information. Although some streaming recommendation strategies have been proposed recently, they are designed for streams of individual interactions rather than streams of sessions. In this paper, we propose a Global Attributed Graph (GAG) neural network model with a Wasserstein reservoir for the SSR problem. On one hand, when a new session arrives, a session graph with a global attribute is constructed based on the current session and its associate user. Thus, the GAG can take both the global attribute and the current session into consideration to learn more comprehensive representations of the session and the user, yielding a better performance in the recommendation. On the other hand, for the adaptation to the streaming session scenario, a Wasserstein reservoir is proposed to help preserve a representative sketch of the historical data. Extensive experiments on two real-world datasets have been conducted to verify the superiority of the GAG model compared with the state-of-the-art methods.

Global Context Enhanced Graph Neural Networks for Session-based Recommendation

Abstract: Session-based recommendation (SBR) is a challenging task, which aims at recommending items based on anonymous behavior sequences. Almost all the existing solutions for SBR model user preference only based on the current session without exploiting the other sessions, which may contain both relevant and irrelevant item-transitions to the current session. This paper proposes a novel approach, called Global Context Enhanced Graph Neural Networks (GCE-GNN) to exploit item transitions over all sessions in a more subtle manner for better inferring the user preference of the current session. Specifically, GCE-GNN learns two levels of item embeddings from session graph and global graph, respectively: (i) Session graph, which is to learn the session-level item embedding by modeling pairwise item-transitions within the current session; and (ii) Global graph, which is to learn the global-level item embedding by modeling pairwise item-transitions over all sessions. In GCE-GNN, we propose a novel global-level item representation learning layer, which employs a session-aware attention mechanism to recursively incorporate the neighbors' embeddings of each node on the global graph. We also design a session-level item representation learning layer, which employs a GNN on the session graph to learn session-level item embeddings within the current session. Moreover, GCE-GNN aggregates the learnt item representations in the two levels with a soft attention mechanism. Experiments on three benchmark datasets demonstrate that GCE-GNN outperforms the state-of-the-art methods consistently.

DGTN: Dual-channel Graph Transition Network for Session-based Recommendation

Abstract: The task of session-based recommendation is to predict user actions based on anonymous sessions. Recent research mainly models the target session as a sequence or a graph to capture item transitions within it, ignoring complex transitions between items in different sessions that have been generated by other users. These item transitions include potential collaborative information and reflect similar behavior patterns, which we assume may help with the recommendation for the target session. In this paper, we propose a novel method, namely Dual-channel Graph Transition Network (DGTN), to model item transitions within not only the target session but also the neighbor sessions. Specifically, we integrate the target session and its neighbor (similar) sessions into a single graph. Then the transition signals are explicitly injected into the embedding by channel-aware propagation. Experiments on real-world datasets demonstrate that DGTN outperforms other state-of-the-art methods. Further analysis verifies the rationality of dual-channel item transition modeling, suggesting a potential future direction for session-based recommendation.

Self-Supervised Hypergraph Convolutional Networks for Session-based Recommendation

Abstract: Session-based recommendation (SBR) focuses on next-item prediction at a certain time point. As user profiles are generally not available in this scenario, capturing the user intent lying in the item transitions plays a pivotal role. Recent graph neural networks (GNNs) based SBR methods regard the item transitions as pairwise relations, which neglect the complex high-order information among items. Hypergraph provides a natural way to capture beyond-pairwise relations, while its potential for SBR has remained unexplored. In this paper, we fill this gap by modeling session-based data as a hypergraph and then propose a dual channel hypergraph convolutional network -- DHCN to improve SBR. Moreover, to enhance hypergraph modeling, we innovatively integrate self-supervised learning into the training of our network by maximizing mutual information between the session representations learned via the two channels in DHCN, serving as an auxiliary task to improve the recommendation task. Extensive experiments on three benchmark datasets demonstrate the superiority of our model over the SOTA methods, and the ablation study validates the effectiveness and rationale of hypergraph modeling and self-supervised task. The implementation of our model is available via https://github.com/xiaxin1998/DHCN.

Session-based Recommendation with Hypergraph Attention Networks

Abstract: Session-based recommender systems aim to improve recommendations in short-term sessions that can be found across many platforms. A critical challenge is to accurately model user intent with only limited evidence in these short sessions. For example, is a flower bouquet being viewed meant as part of a wedding purchase or for home decoration? Such different perspectives greatly impact what should be recommended next. Hence, this paper proposes a novel session-based recommendation system empowered by hypergraph attention networks. Three unique properties of the proposed approach are: (i) it constructs a hypergraph for each session to model the item correlations defined by various contextual windows in the session simultaneously, to uncover item meanings; (ii) it is equipped with hypergraph attention layers to generate item embeddings by flexibly aggregating the contextual information from correlated items in the session; and (iii) it aggregates the dynamic item representations for each session to infer the general purpose and current need, which is decoded to infer the next interesting item in the session. Through experiments on three benchmark datasets, we find the proposed model is effective in generating informative dynamic item embeddings and providing more accurate recommendations compared to the state-of-the-art.

Incorporating User Micro-behaviors and Item Knowledge into Multi-task Learning for Session-based Recommendation

Abstract: Session-based recommendation (SR) has become an important and popular component of various e-commerce platforms, which aims to predict the next interacted item based on a given session. Most of existing SR models only focus on exploiting the consecutive items in a session interacted by a certain user, to capture the transition pattern among the items. Although some of them have been proven effective, the following two insights are often neglected. First, a user's micro-behaviors, such as the manner in which the user locates an item, the activities that the user commits on an item (e.g., reading comments, adding to cart), offer fine-grained and deep understanding of the user's preference. Second, the item attributes, also known as item knowledge, provide side information to model the transition pattern among interacted items and alleviate the data sparsity problem. These insights motivate us to propose a novel SR model MKM-SR in this paper, which incorporates user Micro-behaviors and item Knowledge into Multi-task learning for Session-based Recommendation. Specifically, a given session is modeled on micro-behavior level in MKM-SR, i.e., with a sequence of item-operation pairs rather than a sequence of items, to capture the transition pattern in the session sufficiently. Furthermore, we propose a multi-task learning paradigm to involve learning knowledge embeddings which plays a role as an auxiliary task to promote the major task of SR. It enables our model to obtain better session representations, resulting in more precise SR recommendation results. The extensive evaluations on two benchmark datasets demonstrate MKM-SR's superiority over the state-of-the-art SR models, justifying the strategy of incorporating knowledge learning.

Self-Supervised Graph Co-Training for Session-based Recommendation

Abstract: Session-based recommendation targets next-item prediction by exploiting user behaviors within a short time period. Compared with other recommendation paradigms, session-based recommendation suffers more from the problem of data sparsity due to the very limited short-term interactions. Self-supervised learning, which can discover ground-truth samples from the raw data, holds vast potentials to tackle this problem. However, existing self-supervised recommendation models mainly rely on item/segment dropout to augment data, which are not fit for session-based recommendation because the dropout leads to sparser data, creating unserviceable self-supervision signals. In this paper, for informative session-based data augmentation, we combine self-supervised learning with co-training, and then develop a framework to enhance session-based recommendation. Technically, we first exploit the session-based graph to augment two views that exhibit the internal and external connectivities of sessions, and then we build two distinct graph encoders over the two views, which recursively leverage the different connectivity information to generate ground-truth samples to supervise each other by contrastive learning. In contrast to the dropout strategy, the proposed self-supervised graph co-training preserves the complete session information and fulfills genuine data augmentation. Extensive experiments on multiple benchmark datasets show that, session-based recommendation can be remarkably enhanced under the regime of self-supervised graph co-training, achieving the state-of-the-art performance.

Details of GNN models for session-based recommendation.

Table {{table:a6af8fc8-fb6e-44b3-9f0b-b2ee30343725}}  shows the differences among the above approaches for session-based recommendation.
To sum up, the development of session-based recommendation (SBR) with GNN can be summarized in Fig. {{figure:1c18c064-a4f7-4e16-ab50-cdaf8083fb1f}} . Early efforts in SBR only model each session sequence with a directed graph, such as SR-GNN {{cite:b2fa4d1}}, GC-SAN {{cite:cdeb856}}, TA-GNN {{cite:7ac5b9a}}, and FGNN {{cite:580a996}}. Then, some methods attempt to enrich the relation and information in the session graph with other sessions or additional links. The methods in {{cite:fbfbd50}}, {{cite:ec2d9ce}}, {{cite:2031a2d}}, {{cite:512359f}}, {{cite:d789efe}}, {{cite:fd78f61}}, {{cite:c766076}}, {{cite:7d0a0d2}}, {{cite:6160161}}, {{cite:612d769}}, {{cite:dcb8e27}}, {{cite:ee8003f}} combine the information from other sessions to capture more information from similar sessions or all sessions.
Moreover, the methods with additional links, such as SGNN-HN {{cite:c6361f5}}, SHARE {{cite:4733d71}} and LESSR {{cite:cf303ae}}, attempt to introduce additional edges to the given session to capture the complex relations and information in session data.
Furthermore, some studies, such as DHCN {{cite:fd78f61}} and SHARE {{cite:4733d71}}, attempt to enhance the recommendation by modeling the high-order relations in session data more sufficiently.
As we can observe, the current works of GNN models for session-based recommendation have been developed from simple graphs to complex graphs, with even hyper-edges. Unlike sequential recommendation, session-based recommendation, in which users have repeated interactions with specific items, is more suitable for GNN-based models. This makes the existing works of session-based recommendation can build more powerful and complex graph structures. A possible improvement of this area is to explore larger-scale datasets and further evaluate GNN's ability in these datasets.


What Makes Good In-Context Examples for GPT-$3$?

Abstract: GPT-$3$ has attracted lots of attention due to its superior performance across a wide range of NLP tasks, especially with its powerful and versatile in-context few-shot learning ability. Despite its success, we found that the empirical results of GPT-$3$ depend heavily on the choice of in-context examples. In this work, we investigate whether there are more effective strategies for judiciously selecting in-context examples (relative to random sampling) that better leverage GPT-$3$'s few-shot capabilities. Inspired by the recent success of leveraging a retrieval module to augment large-scale neural network models, we propose to retrieve examples that are semantically-similar to a test sample to formulate its corresponding prompt. Intuitively, the in-context examples selected with such a strategy may serve as more informative inputs to unleash GPT-$3$'s extensive knowledge. We evaluate the proposed approach on several natural language understanding and generation benchmarks, where the retrieval-based prompt selection approach consistently outperforms the random baseline. Moreover, it is observed that the sentence encoders fine-tuned on task-related datasets yield even more helpful retrieval results. Notably, significant gains are observed on tasks such as table-to-text generation (41.9% on the ToTTo dataset) and open-domain question answering (45.5% on the NQ dataset). We hope our investigation could help understand the behaviors of GPT-$3$ and large-scale pre-trained LMs in general and enhance their few-shot capabilities.

Self-Generated In-Context Learning: Leveraging Auto-regressive Language Models as a Demonstration Generator

Abstract: Large-scale pre-trained language models (PLMs) are well-known for being capable of solving a task simply by conditioning a few input-label pairs dubbed demonstrations on a prompt without being explicitly tuned for the desired downstream task. Such a process (i.e., in-context learning), however, naturally leads to high reliance on the demonstrations which are usually selected from external datasets. In this paper, we propose self-generated in-context learning (SG-ICL), which generates demonstrations for in-context learning from PLM itself to minimize the reliance on the external demonstration. We conduct experiments on four different text classification tasks and show SG-ICL significantly outperforms zero-shot learning and is generally worth approximately 0.6 gold training samples. Moreover, our generated demonstrations show more consistent performance with low variance compared to randomly selected demonstrations from the training dataset.

Learning To Retrieve Prompts for In-Context Learning

Abstract: In-context learning is a recent paradigm in natural language understanding, where a large pre-trained language model (LM) observes a test instance and a few training examples as its input, and directly decodes the output without any update to its parameters. However, performance has been shown to strongly depend on the selected training examples (termed prompt). In this work, we propose an efficient method for retrieving prompts for in-context learning using annotated data and a LM. Given an input-output pair, we estimate the probability of the output given the input and a candidate training example as the prompt, and label training examples as positive or negative based on this probability. We then train an efficient dense retriever from this data, which is used to retrieve training examples as prompts at test time. We evaluate our approach on three sequence-to-sequence tasks where language utterances are mapped to meaning representations, and find that it substantially outperforms prior work and multiple baselines across the board.

Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity

Abstract: When primed with only a handful of training samples, very large, pretrained language models such as GPT-3 have shown competitive results when compared to fully-supervised, fine-tuned, large, pretrained language models. We demonstrate that the order in which the samples are provided can make the difference between near state-of-the-art and random guess performance: essentially some permutations are "fantastic" and some not. We analyse this phenomenon in detail, establishing that: it is present across model sizes (even for the largest current models), it is not related to a specific subset of samples, and that a given good permutation for one model is not transferable to another. While one could use a development set to determine which permutations are performant, this would deviate from the true few-shot setting as it requires additional annotated data. Instead, we use the generative nature of language models to construct an artificial development set and based on entropy statistics of the candidate permutations on this set, we identify performant prompts. Our method yields a 13% relative improvement for GPT-family models across eleven different established text classification tasks.

Self-Instruct: Aligning Language Models with Self-Generated Instructions

Abstract: Large "instruction-tuned" language models (i.e., finetuned to respond to instructions) have demonstrated a remarkable ability to generalize zero-shot to new tasks. Nevertheless, they depend heavily on human-written instruction data that is often limited in quantity, diversity, and creativity, therefore hindering the generality of the tuned model. We introduce Self-Instruct, a framework for improving the instruction-following capabilities of pretrained language models by bootstrapping off their own generations. Our pipeline generates instructions, input, and output samples from a language model, then filters invalid or similar ones before using them to finetune the original model. Applying our method to the vanilla GPT3, we demonstrate a 33% absolute improvement over the original model on Super-NaturalInstructions, on par with the performance of InstructGPT-001, which was trained with private user data and human annotations. For further evaluation, we curate a set of expert-written instructions for novel tasks, and show through human evaluation that tuning GPT3 with Self-Instruct outperforms using existing public instruction datasets by a large margin, leaving only a 5% absolute gap behind InstructGPT-001. Self-Instruct provides an almost annotation-free method for aligning pre-trained language models with instructions, and we release our large synthetic dataset to facilitate future studies on instruction tuning. Our code and data are available at https://github.com/yizhongw/self-instruct.

Chain-of-Thought Prompting Elicits Reasoning in Large Language Models

Abstract: We explore how generating a chain of thought -- a series of intermediate reasoning steps -- significantly improves the ability of large language models to perform complex reasoning. In particular, we show how such reasoning abilities emerge naturally in sufficiently large language models via a simple method called chain of thought prompting, where a few chain of thought demonstrations are provided as exemplars in prompting. Experiments on three large language models show that chain of thought prompting improves performance on a range of arithmetic, commonsense, and symbolic reasoning tasks. The empirical gains can be striking. For instance, prompting a 540B-parameter language model with just eight chain of thought exemplars achieves state of the art accuracy on the GSM8K benchmark of math word problems, surpassing even finetuned GPT-3 with a verifier.

Measuring and Narrowing the Compositionality Gap in Language Models

Abstract: We investigate the ability of language models to perform compositional reasoning tasks where the overall solution depends on correctly composing the answers to sub-problems. We measure how often models can correctly answer all sub-problems but not generate the overall solution, a ratio we call the compositionality gap. We evaluate this ratio by asking multi-hop questions with answers that require composing multiple facts unlikely to have been observed together during pretraining. In the GPT-3 family of models, as model size increases we show that the single-hop question answering performance improves faster than the multi-hop performance does, therefore the compositionality gap does not decrease. This surprising result suggests that while more powerful models memorize and recall more factual knowledge, they show no corresponding improvement in their ability to perform this kind of compositional reasoning. We then demonstrate how elicitive prompting (such as chain of thought) narrows the compositionality gap by reasoning explicitly. We present a new method, self-ask, that further improves on chain of thought. In our method, the model explicitly asks itself (and answers) follow-up questions before answering the initial question. We finally show that self-ask's structured prompting lets us easily plug in a search engine to answer the follow-up questions, which additionally improves accuracy.

Summary of representative demonstration designing methods.

Many studies have shown that the performance of ICL strongly relies on the demonstration surface, including prompt format, the order of demonstration examples, and so on {{cite:ebd846f}}, {{cite:d91745a}}. As demonstrations play a vital role in ICL, in this section, we survey prompt designing strategies and classify them into two groups: demonstration organization and demonstration formatting, as shown in Table {{table:c684d927-5b6b-4f60-8908-6686d3fb2074}} .


FedAdapter: Efficient Federated Learning for Modern NLP

Abstract: Transformer-based pre-trained models have revolutionized NLP for superior performance and generality. Fine-tuning pre-trained models for downstream tasks often requires private data, for which federated learning is the de-facto approach (i.e., FedNLP). However, our measurements show that FedNLP is prohibitively slow due to the large model sizes and the resultant high network/computation cost. Towards practical FedNLP, we identify as the key building blocks adapters, small bottleneck modules inserted at a variety of model layers. A key challenge is to properly configure the depth and width of adapters, to which the training speed and efficiency is highly sensitive. No silver-bullet configuration exists: the optimal choice varies across downstream NLP tasks, desired model accuracy, and mobile resources. To automate adapter configuration, we propose FedAdapter, a framework that enhances the existing FedNLP with two novel designs. First, FedAdapter progressively upgrades the adapter configuration throughout a training session; the principle is to quickly learn shallow knowledge by only training fewer and smaller adapters at the model's top layers, and incrementally learn deep knowledge by incorporating deeper and larger adapters. Second, FedAdapter continuously profiles future adapter configurations by allocating participant devices to trial groups. Extensive experiments show that FedAdapter can reduce FedNLP's model convergence delay to no more than several hours, which is up to 155.5$\times$ faster compared to vanilla FedNLP and 48$\times$ faster compared to strong baselines.

FedBalancer: data and pace control for efficient federated learning on heterogeneous clients

Abstract: Federated Learning (FL) trains a machine learning model on distributed clients without exposing individual data. Unlike centralized training that is usually based on carefully-organized data, FL deals with on-device data that are often unfiltered and imbalanced. As a result, conventional FL training protocol that treats all data equally leads to a waste of local computational resources and slows down the global learning process. To this end, we propose FedBalancer, a systematic FL framework that actively selects clients' training samples. Our sample selection strategy prioritizes more "informative" data while respecting privacy and computational capabilities of clients. To better utilize the sample selection to speed up global training, we further introduce an adaptive deadline control scheme that predicts the optimal deadline for each round with varying client training data. Compared with existing FL algorithms with deadline configuration methods, our evaluation on five datasets from three different domains shows that FedBalancer improves the time-to-accuracy performance by 1.20~4.48× while improving the model accuracy by 1.1~5.0%. We also show that FedBalancer is readily applicable to other FL approaches by demonstrating that FedBalancer improves the convergence speed and accuracy when operating jointly with three different FL algorithms.

FedNLP: Benchmarking Federated Learning Methods for Natural Language Processing Tasks

Abstract: Increasing concerns and regulations about data privacy and sparsity necessitate the study of privacy-preserving, decentralized learning methods for natural language processing (NLP) tasks. Federated learning (FL) provides promising approaches for a large number of clients (e.g., personal devices or organizations) to collaboratively learn a shared global model to benefit all clients while allowing users to keep their data locally. Despite interest in studying FL methods for NLP tasks, a systematic comparison and analysis is lacking in the literature. Herein, we present the FedNLP, a benchmarking framework for evaluating federated learning methods on four different task formulations: text classification, sequence tagging, question answering, and seq2seq. We propose a universal interface between Transformer-based language models (e.g., BERT, BART) and FL methods (e.g., FedAvg, FedOPT, etc.) under various non-IID partitioning strategies. Our extensive experiments with FedNLP provide empirical comparisons between FL methods and helps us better understand the inherent challenges of this direction. The comprehensive analysis points to intriguing and exciting future research aimed at developing FL methods for NLP tasks.

GradMA: A Gradient-Memory-based Accelerated Federated Learning with Alleviated Catastrophic Forgetting

Abstract: Federated Learning (FL) has emerged as a de facto machine learning area and received rapid increasing research interests from the community. However, catastrophic forgetting caused by data heterogeneity and partial participation poses distinctive challenges for FL, which are detrimental to the performance. To tackle the problems, we propose a new FL approach (namely GradMA), which takes inspiration from continual learning to simultaneously correct the server-side and worker-side update directions as well as take full advantage of server's rich computing and memory resources. Furthermore, we elaborate a memory reduction strategy to enable GradMA to accommodate FL with a large scale of workers. We then analyze convergence of GradMA theoretically under the smooth non-convex setting and show that its convergence rate achieves a linear speed up w.r.t the increasing number of sampled active workers. At last, our extensive experiments on various image classification tasks show that GradMA achieves significant performance gains in accuracy and communication efficiency compared to SOTA baselines. We provide our code here: https://github.com/lkyddd/GradMA.

FedScale: Benchmarking Model and System Performance of Federated Learning at Scale

Abstract: We present FedScale, a federated learning (FL) benchmarking suite with realistic datasets and a scalable runtime to enable reproducible FL research. FedScale datasets encompass a wide range of critical FL tasks, ranging from image classification and object detection to language modeling and speech recognition. Each dataset comes with a unified evaluation protocol using real-world data splits and evaluation metrics. To reproduce realistic FL behavior, FedScale contains a scalable and extensible runtime. It provides high-level APIs to implement FL algorithms, deploy them at scale across diverse hardware and software backends, and evaluate them at scale, all with minimal developer efforts. We combine the two to perform systematic benchmarking experiments and highlight potential opportunities for heterogeneity-aware co-optimizations in FL. FedScale is open-source and actively maintained by contributors from different institutions at http://fedscale.ai. We welcome feedback and contributions from the community.

FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropout

Abstract: Federated Learning (FL) has been gaining significant traction across different ML tasks, ranging from vision to keyboard predictions. In large-scale deployments, client heterogeneity is a fact and constitutes a primary problem for fairness, training performance and accuracy. Although significant efforts have been made into tackling statistical data heterogeneity, the diversity in the processing capabilities and network bandwidth of clients, termed as system heterogeneity, has remained largely unexplored. Current solutions either disregard a large portion of available devices or set a uniform limit on the model's capacity, restricted by the least capable participants. In this work, we introduce Ordered Dropout, a mechanism that achieves an ordered, nested representation of knowledge in deep neural networks (DNNs) and enables the extraction of lower footprint submodels without the need of retraining. We further show that for linear maps our Ordered Dropout is equivalent to SVD. We employ this technique, along with a self-distillation methodology, in the realm of FL in a framework called FjORD. FjORD alleviates the problem of client system heterogeneity by tailoring the model width to the client's capabilities. Extensive evaluation on both CNNs and RNNs across diverse modalities shows that FjORD consistently leads to significant performance gains over state-of-the-art baselines, while maintaining its nested structure.

Prior FL literature (mobile/system/AI) use a small ratio of devices in experiments. Maximal numbers are selected if many datasets are used.

In real-world scenarios, there could be easily more than millions devices, such as smartphones and IoTs, that are capable of contributing local data and computing resources. After filtering out devices in non-optimal conditions, such as those with low battery or high utilization, the remaining training-available device number still easily exceeds hundreds. For instance, in Google's deployed FL system {{cite:56d251e}}, around 10,000 devices are simultaneously available for local training. The failure to scale to a larger number of idle devices seriously constrains the rate at which the model can converge.
As indicated in Table {{table:584bde9f-0a68-4b6b-82e0-3b8a7bdbe685}} , even when millions of devices are available, the existing literature on FL uniformly adopts the default experimental setting of using no more than 100 devices.
The root cause of this scalability issue can be attributed to backpropagation-based optimizers {{cite:2bac674}}, {{cite:429797d}}, {{cite:5e0d430}} and can hardly be addressed at systems aspect.


Dynamic Patch Generation for Null Pointer Exceptions using Metaprogramming

Abstract: Null pointer exceptions (NPE) are the number one cause of uncaught crashing exceptions in production. In this paper, we aim at exploring the search space of possible patches for null pointer exceptions with metaprogramming. Our idea is to transform the program under repair with automated code transformation, so as to obtain a metaprogram. This metaprogram contains automatically injected hooks, that can be activated to emulate a null pointer exception patch. This enables us to perform a fine-grain analysis of the runtime context of null pointer exceptions. We set up an experiment with 16 real null pointer exceptions that have happened in the field. We compare the effectiveness of our metaprogramming approach against simple templates for repairing null pointer exceptions.

Precise Condition Synthesis for Program Repair

Abstract: Due to the difficulty of repairing defect, many research efforts have been devoted into automatic defect repair. Given a buggy program that fails some test cases, a typical automatic repair technique tries to modify the program to make all tests pass. However, since the test suites in real world projects are usually insufficient, aiming at passing the test suites often leads to incorrect patches. In this paper we aim to produce precise patches, that is, any patch we produce has a relatively high probability to be correct. More concretely, we focus on condition synthesis, which was shown to be able to repair more than half of the defects in existing approaches. Our key insight is threefold. First, it is important to know what variables in a local context should be used in an "if" condition, and we propose a sorting method based on the dependency relations between variables. Second, we observe that the API document can be used to guide the repair process, and propose document analysis technique to further filter the variables. Third, it is important to know what predicates should be performed on the set of variables, and we propose to mine a set of frequently used predicates in similar contexts from existing projects. We develop a novel program repair system, ACS, that could generate precise conditions at faulty locations. Furthermore, given the generated conditions are very precise, we can perform a repair operation that is previously deemed to be too overfitting: directly returning the test oracle to repair the defect. Using our approach, we successfully repaired 17 defects on four projects of Defects4J, which is the largest number of fully automatically repaired defects reported on the dataset so far. More importantly, the precision of our approach in the evaluation is 73.9%, which is significantly higher than previous approaches, which are usually less than 40%.

Revisiting RTS strategies on APR systems

We first systematically revisit RTS strategies adopted in existing APR systems. In particular, we consider APR systems targeting at JAVA programs due to its popularity in the APR community, and use two sources of information to collect representative APR systems (i.e., program-repair.org and the living review of APR {{cite:259223a}}). For each APR system, we check which RTS strategy is used both in its paper and its implementation (if the source code is released). Table {{table:c17d8ccb-c9b7-496b-a819-8642eed9a128}}  summarizes the results of our literature review.

As suggested by the equation, the fault localization strategy and the patch generation strategy directly determine the number and the order of generated patches, while the patch validation strategy (i.e., which tests are executed and in what order they are executed) determines the time cost within the validation process for each patch.
As an importance strategy that is tightly correlated to the APR performance, we revisit the literature of existing automated program repair techniques to collect the patch validation strategies adopted in these work.
Table {{table:c17d8ccb-c9b7-496b-a819-8642eed9a128}}  presents the patch validation strategies of existing automated program repair systems. The columns “Tool”, “Category”, “Time” present the name, category, and publish time of each APR tool. The columns “paper” and “code” present whether the patch validation strategy is mentioned in the paper or the implementation (if its code is released).
For the APR systems whose source code is publicly released, we inspect the implementation to confirm the patch validation strategy. For the APR systems whose source code is not released and the patch validation strategy is neither mentioned in the paper, we regard them as adopting the default setting, i.e., executing the whole test suite without a specific order. The column “Test Selection Strategy” presents the RTS strategies adopted on the originally-failed tests and originally-pass tests, respectively.
Therefore four types of RTS strategies. (1) No RTS: no RTS strategy is adopted and all tests are executed; (2) CLASS RTS: {{formula:76ae8b08-64e5-42b3-b794-b8dfa57455e3}}  is adopted and only the tests covering the patched classes are executed; (3) METHOD RTS: {{formula:fb79cc43-5af8-4693-91da-d9feca376f5f}}  is adopted and only the tests covering the patched methods are executed; (4) ST RTS: {{formula:f0f319cf-a886-4612-a2c7-0a38704c14d8}}  is adopted and only the tests covering the patched statements are executed.
According to Table {{table:c17d8ccb-c9b7-496b-a819-8642eed9a128}} , although as an important strategy in APR, existing work pays little attention to RTS strategies in patch validation phase, e.g., most APR systems have not integrated any RTS strategies during patch validation. For those APR systems with RTS, they adopt different RTS strategies (e.g., PraPR uses {{formula:827452fd-6a0b-4165-ac5e-903d90e8edc5}}  for all the tests while CapGen adopts {{formula:32b1bcdc-e43b-418f-859e-1da166f56f35}}  for all the tests). Therefore, it remains unclean whether RTS is necessary for patch validation and how different RTS strategies impact the efficiency of patch validation in automated program repair.
To bridge the knowledge gap, in this work, we perform the first study to investigate the impact of RTS strategies on patch validation efficiency.

Since we have discussed the background about RTS and APR in Section , in this section, we mainly focus on the closely related work for improving APR efficiency.
APR is resource consuming due to the large number of generated patches and the non-trivial costs in validating each patch. In addition to the large body of APR approaches that aim to reduce the number of generated patches (as mentioned in Section REF ), researchers have proposed various approaches to reduce the test executions for each patch.
For example, Qi et al. {{cite:51eb39b}} proposed a patch-history-based test prioritization strategy, TrpAutoRepair, to schedule test execution order, so that the failure-triggering tests could be executed earlier.
Later, Mehne et al. {{cite:08dd957}} proposed to reduce the number of test executions for each patch based on statement coverage (i.e., statement-level RTS).
Similarly, as suggested by Table {{table:c17d8ccb-c9b7-496b-a819-8642eed9a128}} , several existing APR systems also leveraged RTS to accelerate patch validation, e.g., the ARJA family {{cite:c2a2c98}} adopted statement-level RTS and CapGen {{cite:6eb8865}} adopted class-level RTS.
However, there is limited study on the effectiveness of RTS techniques, and our work conducts the first study to extensively investigate the impact of representative RTS techniques on a wide range of state-of-the-art APR systems.


Solver for a linear equation of one qubit

Trotter steps for molecule LiH at equilibrium

Error correction with distance 3 and 5 qubits

Open Quantum Assembly Language

Abstract: This document describes a quantum assembly language (QASM) called OpenQASM that is used to implement experiments with low depth quantum circuits. OpenQASM represents universal physical circuits over the CNOT plus SU(2) basis with straight-line code that includes measurement, reset, fast feedback, and gate subroutines. The simple text language can be written by hand or by higher level tools and may be executed on the IBM Q Experience.

Open Quantum Assembly Language

Open Quantum Assembly Language

A Hybrid System for Learning Classical Data in Quantum States

Abstract: Deep neural network powered artificial intelligence has rapidly changed our daily life with various applications. However, as one of the essential steps of deep neural networks, training a heavily weighted network requires a tremendous amount of computing resources. Especially in the post-Moore's Law era, the limit of semiconductor fabrication technology has restricted the development of learning algorithms to cope with the increasing high-intensity training data. Meanwhile, quantum computing has demonstrated its significant potential in terms of speeding up the traditionally compute-intensive workloads. For example, Google illustrated quantum supremacy by completing a sampling calculation task in 200 seconds, which is otherwise impracticable on the world's largest supercomputers. To this end, quantum-based learning has become an area of interest, with the potential of a quantum speedup. In this paper, we propose GenQu, a hybrid and general-purpose quantum framework for learning classical data through quantum states. We evaluate GenQu with real datasets and conduct experiments on both simulations and real quantum computer IBM-Q. Our evaluation demonstrates that, compared with classical solutions, the proposed models running on GenQu framework achieve similar accuracy with a much smaller number of qubits, while significantly reducing the parameter size by up to 95.86% and converging speedup by 33.33% faster.

A quantum teleportation experiment for undergraduate students

Abstract: With the rapid progress of quantum information these recent years, it becomes more and more relevant to dedicate efforts in introducing this research topic to undergraduate students. However, as if in various fields of physics the theoretical learning is closely accompanied with experimental illustrations, such a teaching method can be expensive to organise for quantum information. We propose one way to circumvent these difficulties by using the recently made available 5-qubit quantum processor of IBM, in order to illustrate quantum information tasks, by actually implementing them on the superconducting chip through the web interface of the IBM Quantum Experience. We focus on the realization of a quantum teleportation protocol on this device, analyze the results and discuss the issues encountered, providing a complete analysis of this software as a pedagogical tool.

Security of two quantum cryptography protocols using the same four qubit states

Abstract: The first quantum cryptography protocol, proposed by Bennett and Brassard in 1984 (BB84), has been widely studied in recent years. This protocol uses four states (more precisely, two complementary bases) for the encoding of the classical bit. Recently, it has been noticed that by using the same four states, but a different encoding of information, one can define a protocol which is more robust in practical implementations, specifically when attenuated laser pulses are used instead of single-photon sources [V. Scarani et al., Phys. Rev. Lett. 92, 057901 (2004), referred to as the SARG04 protocol]. We present a detailed study of SARG04 in two different regimes. In the first part, we consider an implementation with a single-photon source: we derive bounds on the error rate $Q$ for security against all possible attacks by the eavesdropper. The lower and the upper bound obtained for SARG04 ($Q\ensuremath{\lesssim}10.95%$ and $Q\ensuremath{\gtrsim}14.9%$, respectively) are close to those obtained for BB84 ($Q\ensuremath{\lesssim}12.4%$ and $Q\ensuremath{\gtrsim}14.6%$, respectively). In the second part, we consider a realistic source consisting of an attenuated laser and improve on previous analysis by allowing Alice to optimize the mean number of photons as a function of the distance. The SARG04 protocol is found to perform better than BB84, both in secret-key rate and in maximal achievable distance, for a wide class of Eve's attacks.

Open Quantum Assembly Language

OpenFermion: The Electronic Structure Package for Quantum Computers

Abstract: Quantum simulation of chemistry and materials is predicted to be an important application for both near-term and fault-tolerant quantum devices. However, at present, developing and studying algorithms for these problems can be difficult due to the prohibitive amount of domain knowledge required in both the area of chemistry and quantum algorithms. To help bridge this gap and open the field to more researchers, we have developed the OpenFermion software package (www.openfermion.org). OpenFermion is an open-source software library written largely in Python under an Apache 2.0 license, aimed at enabling the simulation of fermionic models and quantum chemistry problems on quantum hardware. Beginning with an interface to common electronic structure packages, it simplifies the translation between a molecular specification and a quantum circuit for solving or studying the electronic structure problem on a quantum computer, minimizing the amount of domain expertise required to enter the field. The package is designed to be extensible and robust, maintaining high software standards in documentation and testing. This release paper outlines the key motivations behind design choices in OpenFermion and discusses some basic OpenFermion functionality which we believe will aid the community in the development of better quantum algorithms and tools for this exciting area of research.

Open Quantum Assembly Language

Open Quantum Assembly Language

Open Quantum Assembly Language

Open Quantum Assembly Language

Benchmarking gate-based quantum computers

Abstract: With the advent of public access to small gate-based quantum processors, it becomes necessary to develop a benchmarking methodology such that independent researchers can validate the operation of these processors. We explore the usefulness of a number of simple quantum circuits as benchmarks for gate-based quantum computing devices and show that circuits performing identity operations are very simple, scalable and sensitive to gate errors and are therefore very well suited for this task. We illustrate the procedure by presenting benchmark results for the IBM Quantum Experience, a cloud-based platform for gate-based quantum computing.

QASMBench Benchmark Suite version 1.4 {{cite:0606014}}.

In this paper, we used QASMBench Benchmark Suite version 1.4 {{cite:0606014}} for NISQ evaluation. Unless otherwise specified, ibm_lagos, a 7-qubit H-shape superconducting quantum computer (coupling map is shown in Figure {{figure:36b4a922-40a8-495e-a776-2c2503182171}} ) is used for transpilation and scheduling. Due to the limitation of the number of qubits of ibm_lagos, we chose all algorithms whose numbers of qubits are less or equal to 7. Detailed information about the benchmark can be found in Table {{table:54c6e4bb-0a22-4aeb-8665-6d2e053ae0f9}} .


Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification

Abstract: Federated Learning enables visual models to be trained in a privacy-preserving way using real-world data from mobile devices. Given their distributed nature, the statistics of the data across these devices is likely to differ significantly. In this work, we look at the effect such non-identical data distributions has on visual classification via Federated Learning. We propose a way to synthesize datasets with a continuous range of identicalness and provide performance measures for the Federated Averaging algorithm. We show that performance degrades as distributions differ more, and propose a mitigation strategy via server momentum. Experiments on CIFAR-10 demonstrate improved classification performance over a range of non-identicalness, with classification accuracy improved from 30.1% to 76.9% in the most skewed settings.

Distilling the Knowledge in a Neural Network

Abstract: A very simple way to improve the performance of almost any machine learning algorithm is to train many different models on the same data and then to average their predictions. Unfortunately, making predictions using a whole ensemble of models is cumbersome and may be too computationally expensive to allow deployment to a large number of users, especially if the individual models are large neural nets. Caruana and his collaborators have shown that it is possible to compress the knowledge in an ensemble into a single model which is much easier to deploy and we develop this approach further using a different compression technique. We achieve some surprising results on MNIST and we show that we can significantly improve the acoustic model of a heavily used commercial system by distilling the knowledge in an ensemble of models into a single model. We also introduce a new type of ensemble composed of one or more full models and many specialist models which learn to distinguish fine-grained classes that the full models confuse. Unlike a mixture of experts, these specialist models can be trained rapidly and in parallel.

Ensemble Distillation for Robust Model Fusion in Federated Learning

Abstract: Federated Learning (FL) is a machine learning setting where many devices collaboratively train a machine learning model while keeping the training data decentralized. In most of the current training schemes the central model is refined by averaging the parameters of the server model and the updated parameters from the client side. However, directly averaging model parameters is only possible if all models have the same structure and size, which could be a restrictive constraint in many scenarios. In this work we investigate more powerful and more flexible aggregation schemes for FL. Specifically, we propose ensemble distillation for model fusion, i.e. training the central classifier through unlabeled data on the outputs of the models from the clients. This knowledge distillation technique mitigates privacy risk and cost to the same extent as the baseline FL algorithms, but allows flexible aggregation over heterogeneous client models that can differ e.g. in size, numerical precision or structure. We show in extensive empirical experiments on various CV/NLP datasets (CIFAR-10/100, ImageNet, AG News, SST2) and settings (heterogeneous models/data) that the server model can be trained much faster, requiring fewer communication rounds than any existing FL technique so far.

FedMD: Heterogenous Federated Learning via Model Distillation

Abstract: Federated learning enables the creation of a powerful centralized model without compromising data privacy of multiple participants. While successful, it does not incorporate the case where each participant independently designs its own model. Due to intellectual property concerns and heterogeneous nature of tasks and data, this is a widespread requirement in applications of federated learning to areas such as health care and AI as a service. In this work, we use transfer learning and knowledge distillation to develop a universal framework that enables federated learning when each agent owns not only their private data, but also uniquely designed models. We test our framework on the MNIST/FEMNIST dataset and the CIFAR10/CIFAR100 dataset and observe fast improvement across all participating models. With 10 distinct participants, the final test accuracy of each model on average receives a 20% gain on top of what's possible without collaboration and is only a few percent lower than the performance each model would have obtained if all private datasets were pooled and made directly available for all participants.

Cronus: Robust and Heterogeneous Collaborative Learning with Black-Box Knowledge Transfer

Abstract: Collaborative (federated) learning enables multiple parties to train a model without sharing their private data, but through repeated sharing of the parameters of their local models. Despite its advantages, this approach has many known privacy and security weaknesses and performance overhead, in addition to being limited only to models with homogeneous architectures. Shared parameters leak a significant amount of information about the local (and supposedly private) datasets. Besides, federated learning is severely vulnerable to poisoning attacks, where some participants can adversarially influence the aggregate parameters. Large models, with high dimensional parameter vectors, are in particular highly susceptible to privacy and security attacks: curse of dimensionality in federated learning. We argue that sharing parameters is the most naive way of information exchange in collaborative learning, as they open all the internal state of the model to inference attacks, and maximize the model's malleability by stealthy poisoning attacks. We propose Cronus, a robust collaborative machine learning framework. The simple yet effective idea behind designing Cronus is to control, unify, and significantly reduce the dimensions of the exchanged information between parties, through robust knowledge transfer between their black-box local models. We evaluate all existing federated learning algorithms against poisoning attacks, and we show that Cronus is the only secure method, due to its tight robustness guarantee. Treating local models as black-box, reduces the information leakage through models, and enables us using existing privacy-preserving algorithms that mitigate the risk of information leakage through the model's output (predictions). Cronus also has a significantly lower sample complexity, compared to federated learning, which does not bind its security to the number of participants.

FedAUX: Leveraging Unlabeled Auxiliary Data in Federated Learning

Abstract: Federated distillation (FD) is a popular novel algorithmic paradigm for Federated learning (FL), which achieves training performance competitive to prior parameter averaging-based methods, while additionally allowing the clients to train different model architectures, by distilling the client predictions on an unlabeled auxiliary set of data into a student model. In this work, we propose FedAUX, an extension to FD, which, under the same set of assumptions, drastically improves the performance by deriving maximum utility from the unlabeled auxiliary data. FedAUX modifies the FD training procedure in two ways: First, unsupervised pre-training on the auxiliary data is performed to find a suitable model initialization for the distributed training. Second, $(\varepsilon, \delta)$ -differentially private certainty scoring is used to weight the ensemble predictions on the auxiliary data according to the certainty of each client model. Experiments on large-scale convolutional neural networks (CNNs) and transformer models demonstrate that our proposed method achieves remarkable performance improvements over state-of-the-art FL methods, without adding appreciable computation, communication, or privacy cost. For instance, when training ResNet8 on non-independent identically distributed (i.i.d.) subsets of CIFAR10, FedAUX raises the maximum achieved validation accuracy from 30.4% to 78.1%, further closing the gap to centralized training performance. Code is available at https://github.com/fedl-repo/fedaux.

Federated Semi-Supervised Learning with Inter-Client Consistency

Abstract: While existing federated learning approaches mostly require that clients have fully-labeled data to train on, in realistic settings, data obtained at the client side often comes without any accompanying labels. Such deficiency of labels may result from either high labeling cost, or difficulty of annotation due to requirement of expert knowledge. Thus the private data at each client may be only partly labeled, or completely unlabeled with labeled data being available only at the server, which leads us to a new problem of Federated Semi-Supervised Learning (FSSL). In this work, we study this new problem of semi-supervised learning under federated learning framework, and propose a novel method to tackle it, which we refer to as Federated Matching (FedMatch). FedMatch improves upon naive federated semi-supervised learning approaches with a new inter-client consistency loss and decomposition of the parameters into parameters for labeled and unlabeled data. Through extensive experimental validation of our method in two different scenarios, we show that our method outperforms both local semi-supervised learning and baselines which naively combine federated learning with semi-supervised learning.

DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition

Abstract: We evaluate whether features extracted from the activation of a deep convolutional network trained in a fully supervised fashion on a large, fixed set of object recognition tasks can be re-purposed to novel generic tasks. Our generic tasks may differ significantly from the originally trained tasks and there may be insufficient labeled or unlabeled data to conventionally train or adapt a deep architecture to the new tasks. We investigate and visualize the semantic clustering of deep convolutional features with respect to a variety of such tasks, including scene recognition, domain adaptation, and fine-grained recognition challenges. We compare the efficacy of relying on various network levels to define a fixed feature, and report novel results that significantly outperform the state-of-the-art on several important vision challenges. We are releasing DeCAF, an open-source implementation of these deep convolutional activation features, along with all associated network parameters to enable vision researchers to be able to conduct experimentation with deep representations across a range of visual concept learning paradigms.

CNN Features Off-the-Shelf: An Astounding Baseline for Recognition

Abstract: Recent results indicate that the generic descriptors extracted from the convolutional neural networks are very powerful. This paper adds to the mounting evidence that this is indeed the case. We report on a series of experiments conducted for different recognition tasks using the publicly available code and model of the OverFeat network which was trained to perform object classification on ILSVRC13. We use features extracted from the OverFeat network as a generic image representation to tackle the diverse range of recognition tasks of object image classification, scene recognition, fine grained recognition, attribute detection and image retrieval applied to a diverse set of datasets. We selected these tasks and datasets as they gradually move further away from the original task and data the OverFeat network was trained to solve. Astonishingly, we report consistent superior results compared to the highly tuned state-of-the-art systems in all the visual classification tasks on various datasets. For instance retrieval it consistently outperforms low memory footprint methods except for sculptures dataset. The results are achieved using a linear SVM classifier (or L2 distance in case of retrieval) applied to a feature representation of size 4096 extracted from a layer in the net. The representations are further modified using simple augmentation techniques e.g. jittering. The results strongly suggest that features obtained from deep learning with convolutional nets should be the primary candidate in most visual recognition tasks.

Rich feature hierarchies for accurate object detection and semantic segmentation

Abstract: Object detection performance, as measured on the canonical PASCAL VOC dataset, has plateaued in the last few years. The best-performing methods are complex ensemble systems that typically combine multiple low-level image features with high-level context. In this paper, we propose a simple and scalable detection algorithm that improves mean average precision (mAP) by more than 30% relative to the previous best result on VOC 2012---achieving a mAP of 53.3%. Our approach combines two key insights: (1) one can apply high-capacity convolutional neural networks (CNNs) to bottom-up region proposals in order to localize and segment objects and (2) when labeled training data is scarce, supervised pre-training for an auxiliary task, followed by domain-specific fine-tuning, yields a significant performance boost. Since we combine region proposals with CNNs, we call our method R-CNN: Regions with CNN features. We also compare R-CNN to OverFeat, a recently proposed sliding-window detector based on a similar CNN architecture. We find that R-CNN outperforms OverFeat by a large margin on the 200-class ILSVRC2013 detection dataset. Source code for the complete system is available at http://www.cs.berkeley.edu/~rbg/rcnn.

Progressive Neural Networks

Abstract: Learning to solve complex sequences of tasks--while both leveraging transfer and avoiding catastrophic forgetting--remains a key obstacle to achieving human-level intelligence. The progressive networks approach represents a step forward in this direction: they are immune to forgetting and can leverage prior knowledge via lateral connections to previously learned features. We evaluate this architecture extensively on a wide variety of reinforcement learning tasks (Atari and 3D maze games), and show that it outperforms common baselines based on pretraining and finetuning. Using a novel sensitivity measure, we demonstrate that transfer occurs at both low-level sensory and high-level control layers of the learned policy.

iCaRL: Incremental Classifier and Representation Learning

Abstract: A major open problem on the road to artificial intelligence is the development of incrementally learning systems that learn about more and more concepts over time from a stream of data. In this work, we introduce a new training strategy, iCaRL, that allows learning in such a class-incremental way: only the training data for a small number of classes has to be present at the same time and new classes can be added progressively. iCaRL learns strong classifiers and a data representation simultaneously. This distinguishes it from earlier works that were fundamentally limited to fixed data representations and therefore incompatible with deep learning architectures. We show by experiments on CIFAR-100 and ImageNet ILSVRC 2012 data that iCaRL can learn many classes incrementally over a long period of time where other strategies quickly fail.

Learning without Forgetting

Abstract: When building a unified vision system or gradually adding new capabilities to a system, the usual assumption is that training data for all tasks is always available. However, as the number of tasks grows, storing and retraining on such data becomes infeasible. A new problem arises where we add new capabilities to a Convolutional Neural Network (CNN), but the training data for its existing capabilities are unavailable. We propose our Learning without Forgetting method, which uses only new task data to train the network while preserving the original capabilities. Our method performs favorably compared to commonly used feature extraction and fine-tuning adaption techniques and performs similarly to multitask learning that uses original task data we assume unavailable. A more surprising observation is that Learning without Forgetting may be able to replace fine-tuning with similar old and new task datasets for improved new task performance.

Overview of the existing Federated and Continual Learning methods tackling, respectively, Client Drift and Catastrophic Forgetting.

For the safe and privacy-aware deployment of Deep Learning models in real-world applications, a holistic strategy must address both of these concerns, as they typically occur concurrently.
However, in the current research landscape, these problems are studied separately – as shown in Table {{table:e3ba6bf7-3503-4f2c-878c-04cedf814912}}  in the supplementary material – even though all approaches try to solve the same underlying problem of distribution shift.

In Table {{table:e3ba6bf7-3503-4f2c-878c-04cedf814912}}  we give an overview of existing approaches that address CD/CF individually and categorize them.
We start with a short introduction of the main categories from the table for approaches against CD:
One direction is to either just penalize outliers {{cite:0476653}} while other methods go further by actively compensating CD {{cite:e485539}}.
Alternatively, sharing a global dataset enables a better generalization and makes the model less prone to CD {{cite:19b96d4}}.
A very different approach is to apply Knowledge Distillation {{cite:0fbd47b}}:
First, the clients train their local model. After that, they teach the server by providing their output logits. Besides making the global model more flexible to different inputs, this intermediate representation step reduces CD.
Beyond these FL-related methods, there are separate approaches against CF in CL:
One direction is parameter tuning {{cite:57e32f4}} to balance learning new data distributions while preserving the existing knowledge.
Alternatively, some approaches extend the network {{cite:c8783fe}} to keep distribution-specific weights.
Keeping data/logits {{cite:df06858}} is a widely used approach, which we also apply in our experiments.


node2vec: Scalable Feature Learning for Networks

Abstract: Prediction tasks over nodes and edges in networks require careful effort in engineering features used by learning algorithms. Recent research in the broader field of representation learning has led to significant progress in automating prediction by learning the features themselves. However, present feature learning approaches are not expressive enough to capture the diversity of connectivity patterns observed in networks. Here we propose node2vec, an algorithmic framework for learning continuous feature representations for nodes in networks. In node2vec, we learn a mapping of nodes to a low-dimensional space of features that maximizes the likelihood of preserving network neighborhoods of nodes. We define a flexible notion of a node's network neighborhood and design a biased random walk procedure, which efficiently explores diverse neighborhoods. Our algorithm generalizes prior work which is based on rigid notions of network neighborhoods, and we argue that the added flexibility in exploring neighborhoods is the key to learning richer representations. We demonstrate the efficacy of node2vec over existing state-of-the-art techniques on multi-label classification and link prediction in several real-world networks from diverse domains. Taken together, our work represents a new way for efficiently learning state-of-the-art task-independent representations in complex networks.

LINE: Large-scale Information Network Embedding

Abstract: This paper studies the problem of embedding very large information networks into low-dimensional vector spaces, which is useful in many tasks such as visualization, node classification, and link prediction. Most existing graph embedding methods do not scale for real world information networks which usually contain millions of nodes. In this paper, we propose a novel network embedding method called the ``LINE,'' which is suitable for arbitrary types of information networks: undirected, directed, and/or weighted. The method optimizes a carefully designed objective function that preserves both the local and global network structures. An edge-sampling algorithm is proposed that addresses the limitation of the classical stochastic gradient descent and improves both the effectiveness and the efficiency of the inference. Empirical experiments prove the effectiveness of the LINE on a variety of real-world information networks, including language networks, social networks, and citation networks. The algorithm is very efficient, which is able to learn the embedding of a network with millions of vertices and billions of edges in a few hours on a typical single machine. The source code of the LINE is available online\footnote{\url{https://github.com/tangjianpku/LINE}}.

Semi-Supervised Classification with Graph Convolutional Networks

Abstract: We present a scalable approach for semi-supervised learning on graph-structured data that is based on an efficient variant of convolutional neural networks which operate directly on graphs. We motivate the choice of our convolutional architecture via a localized first-order approximation of spectral graph convolutions. Our model scales linearly in the number of graph edges and learns hidden layer representations that encode both local graph structure and features of nodes. In a number of experiments on citation networks and on a knowledge graph dataset we demonstrate that our approach outperforms related methods by a significant margin.

Inductive Representation Learning on Large Graphs

Abstract: Low-dimensional embeddings of nodes in large graphs have proved extremely useful in a variety of prediction tasks, from content recommendation to identifying protein functions. However, most existing approaches require that all nodes in the graph are present during training of the embeddings; these previous approaches are inherently transductive and do not naturally generalize to unseen nodes. Here we present GraphSAGE, a general, inductive framework that leverages node feature information (e.g., text attributes) to efficiently generate node embeddings for previously unseen data. Instead of training individual embeddings for each node, we learn a function that generates embeddings by sampling and aggregating features from a node's local neighborhood. Our algorithm outperforms strong baselines on three inductive node-classification benchmarks: we classify the category of unseen nodes in evolving information graphs based on citation and Reddit post data, and we show that our algorithm generalizes to completely unseen graphs using a multi-graph dataset of protein-protein interactions.

Graph Attention Networks

Abstract: We present graph attention networks (GATs), novel neural network architectures that operate on graph-structured data, leveraging masked self-attentional layers to address the shortcomings of prior methods based on graph convolutions or their approximations. By stacking layers in which nodes are able to attend over their neighborhoods' features, we enable (implicitly) specifying different weights to different nodes in a neighborhood, without requiring any kind of costly matrix operation (such as inversion) or depending on knowing the graph structure upfront. In this way, we address several key challenges of spectral-based graph neural networks simultaneously, and make our model readily applicable to inductive as well as transductive problems. Our GAT models have achieved or matched state-of-the-art results across four established transductive and inductive graph benchmarks: the Cora, Citeseer and Pubmed citation network datasets, as well as a protein-protein interaction dataset (wherein test graphs remain unseen during training).

Representation Learning for Attributed Multiplex Heterogeneous Network

Abstract: Network embedding (or graph embedding) has been widely used in many real-world applications. However, existing methods mainly focus on networks with single-typed nodes/edges and cannot scale well to handle large networks. Many real-world networks consist of billions of nodes and edges of multiple types, and each node is associated with different attributes. In this paper, we formalize the problem of embedding learning for the Attributed Multiplex Heterogeneous Network and propose a unified framework to address this problem. The framework supports both transductive and inductive learning. We also give the theoretical analysis of the proposed framework, showing its connection with previous works and proving its better expressiveness. We conduct systematical evaluations for the proposed framework on four different genres of challenging datasets: Amazon, YouTube, Twitter, and Alibaba. Experimental results demonstrate that with the learned embeddings from the proposed framework, we can achieve statistically significant improvements (e.g., 5.99-28.23% lift by F1 scores; p<<0.01, t-test) over previous state-of-the-art methods for link prediction. The framework has also been successfully deployed on the recommendation system of a worldwide leading e-commerce company, Alibaba Group. Results of the offline A/B tests on product recommendation further confirm the effectiveness and efficiency of the framework in practice.

DynGEM: Deep Embedding Method for Dynamic Graphs

Abstract: Embedding large graphs in low dimensional spaces has recently attracted significant interest due to its wide applications such as graph visualization, link prediction and node classification. Existing methods focus on computing the embedding for static graphs. However, many graphs in practical applications are dynamic and evolve constantly over time. Naively applying existing embedding algorithms to each snapshot of dynamic graphs independently usually leads to unsatisfactory performance in terms of stability, flexibility and efficiency. In this work, we present an efficient algorithm DynGEM based on recent advances in deep autoencoders for graph embeddings, to address this problem. The major advantages of DynGEM include: (1) the embedding is stable over time, (2) it can handle growing dynamic graphs, and (3) it has better running time than using static embedding methods on each snapshot of a dynamic graph. We test DynGEM on a variety of tasks including graph visualization, graph reconstruction, link prediction and anomaly detection (on both synthetic and real datasets). Experimental results demonstrate the superior stability and scalability of our approach.

E-LSTM-D: A Deep Learning Framework for Dynamic Network Link Prediction

Abstract: Predicting the potential relations between nodes in networks, known as link prediction, has long been a challenge in network science. However, most studies just focused on link prediction of static network, while real-world networks always evolve over time with the occurrence and vanishing of nodes and links. Dynamic network link prediction thus has been attracting more and more attention since it can better capture the evolution nature of networks, but still most algorithms fail to achieve satisfied prediction accuracy. Motivated by the excellent performance of Long Short-Term Memory (LSTM) in processing time series, in this paper, we propose a novel Encoder-LSTM-Decoder (E-LSTM-D) deep learning model to predict dynamic links end to end. It could handle long term prediction problems, and suits the networks of different scales with fine-tuned structure. To the best of our knowledge, it is the first time that LSTM, together with an encoder-decoder architecture, is applied to link prediction in dynamic networks. This new model is able to automatically learn structural and temporal features in a unified framework, which can predict the links that never appear in the network before. The extensive experiments show that our E-LSTM-D model significantly outperforms newly proposed dynamic network link prediction methods and obtain the state-of-the-art results.

Dynamic Graph Representation Learning via Self-Attention Networks

Abstract: Learning latent representations of nodes in graphs is an important and ubiquitous task with widespread applications such as link prediction, node classification, and graph visualization. Previous methods on graph representation learning mainly focus on static graphs, however, many real-world graphs are dynamic and evolve over time. In this paper, we present Dynamic Self-Attention Network (DySAT), a novel neural architecture that operates on dynamic graphs and learns node representations that capture both structural properties and temporal evolutionary patterns. Specifically, DySAT computes node representations by jointly employing self-attention layers along two dimensions: structural neighborhood and temporal dynamics. We conduct link prediction experiments on two classes of graphs: communication networks and bipartite rating networks. Our experimental results show that DySAT has a significant performance gain over several different state-of-the-art graph embedding baselines.

Summary of typical network embedding methods.

Currently, limited work has been done for network embedding of dynamic heterogeneous networks. MetaDynaMix {{cite:c056014}} integrates metapath-based topology features and latent representations to learn both heterogeneity and temporal evolution.
Change2vec {{cite:d30d9b1}} focuses on measuring changes in snapshots instead of learning the whole structural information of each snapshot, and it also uses a metapath-based model to capture heterogeneous information.
The above two methods both focus on short-term evolutionary information between adjacent snapshots of dynamic networks and thus become insufficient to capture long-term evolutionary patterns.
More recently, Sajadmanesh et al. {{cite:a552d4c}} uses a recurrent neural network model to learn long-term evolutionary patterns of dynamic networks on top of metapath-based models and proposes a non-parametric generalized linear model, NP-GLM, to predict continuous-time relationships.
Yin et al. {{cite:5160375}} propose a DHNE method, which learns both the historical and current heterogeneous information and models evolutionary patterns by constructing comprehensive historical-current networks based on consecutive snapshots. Then, DHNE performs metapath-based random walk and dynamic heterogeneous skip-gram model to capture representations of nodes.
Kong et al. {{cite:840306d}} introduce a dynamic heterogeneous information network embedding method called HA-LSTM. It uses graph convolutional network to learn heterogeneous information networks and employs attention model and long-short time memory to capture evolving information over timesteps.
Refer to Table {{table:b4f54471-1f06-43af-b80f-d8fb89611ba2}}  for a brief summary of existing network embedding methods.


Rx packets, PDR, dropped packets and delay

Intelligent Intrusion Detection of Grey Hole and Rushing Attacks in Self-Driving Vehicular Networks

Abstract: Vehicular ad hoc networks (VANETs) play a vital role in the success of self-driving and semi self-driving vehicles, where they improve safety and comfort. Such vehicles depend heavily on external communication with the surrounding environment via data control and Cooperative Awareness Messages (CAMs) exchanges. VANETs are potentially exposed to a number of attacks, such as grey hole, black hole, wormhole and rushing attacks. This work presents an intelligent Intrusion Detection System (IDS) that relies on anomaly detection to protect the external communication system from grey hole and rushing attacks. These attacks aim to disrupt the transmission between vehicles and roadside units. The IDS uses features obtained from a trace file generated in a network simulator and consists of a feed-forward neural network and a support vector machine. Additionally, the paper studies the use of a novel systematic response, employed to protect the vehicle when it encounters malicious behaviour. Our simulations of the proposed detection system show that the proposed schemes possess outstanding detection rates with a reduction in false alarms. This safe mode response system has been evaluated using four performance metrics, namely, received packets, packet delivery ratio, dropped packets and the average end to end delay, under both normal and abnormal conditions.

Robust Deep Reinforcement Learning for Security and Safety in Autonomous Vehicle Systems

Abstract: To operate effectively in tomorrow's smart cities, autonomous vehicles (AVs) must rely on intra-vehicle sensors such as camera and radar as well as inter-vehicle communication. Such dependence on sensors and communication links exposes AVs to cyber-physical (CP) attacks by adversaries that seek to take control of the AVs by manipulating their data. Thus, to ensure safe and optimal AV dynamics control, the data processing functions at AVs must be robust to such CP attacks. To this end, in this paper, the state estimation process for monitoring AV dynamics, in presence of CP attacks, is analyzed and a novel adversarial deep reinforcement learning (RL) algorithm is proposed to maximize the robustness of AV dynamics control to CP attacks. The attacker's action and the AV's reaction to CP attacks are studied in a game-theoretic framework. In the formulated game, the attacker seeks to inject faulty data to AV sensor readings so as to manipulate the inter-vehicle optimal safe spacing and potentially increase the risk of AV accidents or reduce the vehicle flow on the roads. Meanwhile, the AV, acting as a defender, seeks to minimize the deviations of spacing so as to ensure robustness to the attacker's actions. Since the AV has no information about the attacker's action and due to the infinite possibilities for data value manipulations, the outcome of the players' past interactions are fed to long-short term memory (LSTM) blocks. Each player's LSTM block learns the expected spacing deviation resulting from its own action and feeds it to its RL algorithm. Then, the the attacker's RL algorithm chooses the action which maximizes the spacing deviation, while the AV's RL algorithm tries to find the optimal action that minimizes such deviation.

Crossfire Attack Detection using Deep Learning in Software Defined ITS Networks

Abstract: Recent developments in intelligent transport systems (ITS) based on smart mobility significantly improves safety and security over roads and highways. ITS networks are comprised of the Internet-connected vehicles (mobile nodes), roadside units (RSU), cellular base stations and conventional core network routers to create a complete data transmission platform that provides real-time traffic information and enable prediction of future traffic conditions. However, the heterogeneity and complexity of the underlying ITS networks raise new challenges in intrusion prevention of mobile network nodes and detection of security attacks due to such highly vulnerable mobile nodes. In this paper, we consider a new type of security attack referred to as crossfire attack, which involves a large number of compromised nodes that generate low-intensity traffic in a temporally coordinated fashion such that target links or hosts (victims) are disconnected from the rest of the network. Detection of such attacks is challenging since the attacking traffic flows are indistinguishable from the legitimate flows. With the support of software-defined networking that enables dynamic network monitoring and traffic characteristic extraction, we develop a machine learning model that can learn the temporal correlation among traffic flows traversing in the ITS network, thus differentiating legitimate flows from coordinated attacking flows. We use different deep learning algorithms to train the model and study the performance using Mininet-WiFi emulation platform. The results show that our approach achieves a detection accuracy of at least 80%.

Summary of Literature on Attack Detection using ML

Vehicular networks are vulnerable to different types of attacks and a number of solutions are proposed in the literature to deal with such attacks{{cite:7995e4d}}. The evolution of V2R, V2V, and V2I communications demands highly-efficient, intelligent, and faster solutions. The traditional hard-coded algorithms are able to deal with the deterministic type of attack scenarios. On the other hand, the self-learning designs of deep architectures can detect a variety of attacks through experience and shared information. Therefore, ML and its sub-classified architectures are gaining popularity to detect attacks and deal with different types of security issues in V2X communications. In this section, we will discuss different types of attacks and ML-based solutions proposed in the literature to prevent these attacks. The summarized tabular comparison of solutions using ML for attack detection is presented Table. {{table:ff5f3b8f-621d-41d0-b214-08396d5c2c11}} .