Multi-Behavior Enhanced Recommendation with Cross-Interaction Collaborative Relation Modeling

Abstract: Many previous studies aim to augment collaborative filtering with deep neural network techniques, so as to achieve better recommendation performance. However, most existing deep learning-based recommender systems are designed for modeling singular type of user-item interaction behavior, which can hardly distill the heterogeneous relations between user and item. In practical recommendation scenarios, there exist multityped user behaviors, such as browse and purchase. Due to the overlook of user's multi-behavioral patterns over different items, existing recommendation methods are insufficient to capture heterogeneous collaborative signals from user multi-behavior data. Inspired by the strength of graph neural networks for structured data modeling, this work proposes a Graph Neural Multi-Behavior Enhanced Recommendation (GNMR) framework which explicitly models the dependencies between different types of user-item interactions under a graph-based message passing architecture. GNMR devises a relation aggregation network to model interaction heterogeneity, and recursively performs embedding propagation between neighboring nodes over the user-item interaction graph. Experiments on real-world recommendation datasets show that our GNMR consistently outperforms state-of-the-art methods. The source code is available at https://github.com/akaxlh/GNMR.

Graph Meta Network for Multi-Behavior Recommendation

Abstract: Modern recommender systems often embed users and items into low-dimensional latent representations, based on their observed interactions. In practical recommendation scenarios, users often exhibit various intents which drive them to interact with items with multiple behavior types (e.g., click, tag-as-favorite, purchase). However, the diversity of user behaviors is ignored in most of the existing approaches, which makes them difficult to capture heterogeneous relational structures across different types of interactive behaviors. Exploring multi-typed behavior patterns is of great importance to recommendation systems, yet is very challenging because of two aspects: i) The complex dependencies across different types of user-item interactions; ii) Diversity of such multi-behavior patterns may vary by users due to their personalized preference. To tackle the above challenges, we propose a Multi-Behavior recommendation framework with Graph Meta Network to incorporate the multi-behavior pattern modeling into a meta-learning paradigm. Our developed MB-GMN empowers the user-item interaction learning with the capability of uncovering type-dependent behavior representations, which automatically distills the behavior heterogeneity and interaction diversity for recommendations. Extensive experiments on three real-world datasets show the effectiveness of MB-GMN by significantly boosting the recommendation performance as compared to various state-of-the-art baselines. The source code is available athttps://github.com/akaxlh/MB-GMN.

Knowledge-Enhanced Hierarchical Graph Transformer Network for Multi-Behavior Recommendation

Abstract: Accurate user and item embedding learning is crucial for modern recommender systems. However, most existing recommendation techniques have thus far focused on modeling users' preferences over singular type of user-item interactions. Many practical recommendation scenarios involve multi-typed user interactive behaviors (e.g., page view, add-to-favorite and purchase), which presents unique challenges that cannot be handled by current recommendation solutions. In particular: i) complex inter-dependencies across different types of user behaviors; ii) the incorporation of knowledge-aware item relations into the multi-behavior recommendation framework; iii) dynamic characteristics of multi-typed user-item interactions. To tackle these challenges, this work proposes a Knowledge-Enhanced Hierarchical Graph Transformer Network (KHGT), to investigate multi-typed interactive patterns between users and items in recommender systems. Specifically, KHGT is built upon a graph-structured neural architecture to i) capture type-specific behavior characteristics; ii) explicitly discriminate which types of user-item interactions are more important in assisting the forecasting task on the target behavior. Additionally, we further integrate the graph attention layer with the temporal encoding strategy, to empower the learned embeddings be reflective of both dedicated multiplex user-item and item-item relations, as well as the underlying interaction dynamics. Extensive experiments conducted on three real-world datasets show that KHGT consistently outperforms many state-of-the-art recommendation methods across various evaluation settings. Our implementation code is available at https://github.com/akaxlh/KHGT.

DMBGN: Deep Multi-Behavior Graph Networks for Voucher Redemption Rate Prediction

Abstract: In E-commerce, vouchers are important marketing tools to enhance users' engagement and boost sales and revenue. The likelihood that a user redeems a voucher is a key factor in voucher distribution decision. User-item Click-Through-Rate (CTR) models are often applied to predict the user-voucher redemption rate. However, the voucher scenario involves more complicated relations among users, items and vouchers. The users' historical behavior in a voucher collection activity reflects users' voucher usage patterns, which is nevertheless overlooked by the CTR-based solutions. In this paper, we propose a Deep Multi-behavior Graph Networks (DMBGN) to shed light on this field for the voucher redemption rate prediction. The complex structural user-voucher-item relationships are captured by a User-Behavior Voucher Graph (UVG). User behavior happening both before and after voucher collection is taken into consideration, and a high-level representation is extracted by Higher-order Graph Neural Networks. On top of a sequence of UVGs, an attention network is built which can help to learn users' long-term voucher redemption preference. Extensive experiments on three large-scale production datasets demonstrate the proposed DMBGN model is effective, with 10% to 16% relative AUC improvement over Deep Neural Networks (DNN), and 2% to 4% AUC improvement over Deep Interest Network (DIN). Source code and a sample dataset are made publicly available to facilitate future research.

DGCN: Diversified Recommendation with Graph Convolutional Networks

Abstract: These years much effort has been devoted to improving the accuracy or relevance of the recommendation system. Diversity, a crucial factor which measures the dissimilarity among the recommended items, received rather little scrutiny. Directly related to user satisfaction, diversification is usually taken into consideration after generating the candidate items. However, this decoupled design of diversification and candidate generation makes the whole system suboptimal. In this paper, we aim at pushing the diversification to the upstream candidate generation stage, with the help of Graph Convolutional Networks (GCN). Although GCN based recommendation algorithms have shown great power in modeling complex collaborative filtering effect to improve the accuracy of recommendation, how diversity changes is ignored in those advanced works. We propose to perform rebalanced neighbor discovering, category-boosted negative sampling and adversarial learning on top of GCN. We conduct extensive experiments on real-world datasets. Experimental results verify the effectiveness of our proposed method on diversification. Further ablation studies validate that our proposed method significantly alleviates the accuracy-diversity dilemma.

RippleNet: Propagating User Preferences on the Knowledge Graph for Recommender Systems

Abstract: To address the sparsity and cold start problem of collaborative filtering, researchers usually make use of side information, such as social networks or item attributes, to improve recommendation performance. This paper considers the knowledge graph as the source of side information. To address the limitations of existing embedding-based and path-based methods for knowledge-graph-aware recommendation, we propose Ripple Network, an end-to-end framework that naturally incorporates the knowledge graph into recommender systems. Similar to actual ripples propagating on the surface of water, Ripple Network stimulates the propagation of user preferences over the set of knowledge entities by automatically and iteratively extending a user's potential interests along links in the knowledge graph. The multiple "ripples" activated by a user's historically clicked items are thus superposed to form the preference distribution of the user with respect to a candidate item, which could be used for predicting the final clicking probability. Through extensive experiments on real-world datasets, we demonstrate that Ripple Network achieves substantial gains in a variety of scenarios, including movie, book and news recommendation, over several state-of-the-art baselines.

Explainable Reasoning over Knowledge Graphs for Recommendation

Abstract: Incorporating knowledge graph into recommender systems has attracted increasing attention in recent years. By exploring the interlinks within a knowledge graph, the connectivity between users and items can be discovered as paths, which provide rich and complementary information to user-item interactions. Such connectivity not only reveals the semantics of entities and relations, but also helps to comprehend a user’s interest. However, existing efforts have not fully explored this connectivity to infer user preferences, especially in terms of modeling the sequential dependencies within and holistic semantics of a path.In this paper, we contribute a new model named Knowledgeaware Path Recurrent Network (KPRN) to exploit knowledge graph for recommendation. KPRN can generate path representations by composing the semantics of both entities and relations. By leveraging the sequential dependencies within a path, we allow effective reasoning on paths to infer the underlying rationale of a user-item interaction. Furthermore, we design a new weighted pooling operation to discriminate the strengths of different paths in connecting a user with an item, endowing our model with a certain level of explainability. We conduct extensive experiments on two datasets about movie and music, demonstrating significant improvements over state-of-the-art solutions Collaborative Knowledge Base Embedding and Neural Factorization Machine.

Reinforcement Knowledge Graph Reasoning for Explainable Recommendation

Abstract: Recent advances in personalized recommendation have sparked great interest in the exploitation of rich structured information provided by knowledge graphs. Unlike most existing approaches that only focus on leveraging knowledge graphs for more accurate recommendation, we perform explicit reasoning with knowledge for decision making so that the recommendations are generated and supported by an interpretable causal inference procedure. To this end, we propose a method called Policy-Guided Path Reasoning (PGPR), which couples recommendation and interpretability by providing actual paths in a knowledge graph. Our contributions include four aspects. We first highlight the significance of incorporating knowledge graphs into recommendation to formally define and interpret the reasoning process. Second, we propose a reinforcement learning (RL) approach featuring an innovative soft reward strategy, user-conditional action pruning and a multi-hop scoring function. Third, we design a policy-guided graph search algorithm to efficiently and effectively sample reasoning paths for recommendation. Finally, we extensively evaluate our method on several large-scale real-world benchmark datasets, obtaining favorable results compared with state-of-the-art methods.

KGAT: Knowledge Graph Attention Network for Recommendation

Abstract: To provide more accurate, diverse, and explainable recommendation, it is compulsory to go beyond modeling user-item interactions and take side information into account. Traditional methods like factorization machine (FM) cast it as a supervised learning problem, which assumes each interaction as an independent instance with side information encoded. Due to the overlook of the relations among instances or items (e.g., the director of a movie is also an actor of another movie), these methods are insufficient to distill the collaborative signal from the collective behaviors of users. In this work, we investigate the utility of knowledge graph (KG), which breaks down the independent interaction assumption by linking items with their attributes. We argue that in such a hybrid structure of KG and user-item graph, high-order relations --- which connect two items with one or multiple linked attributes --- are an essential factor for successful recommendation. We propose a new method named Knowledge Graph Attention Network (KGAT) which explicitly models the high-order connectivities in KG in an end-to-end fashion. It recursively propagates the embeddings from a node's neighbors (which can be users, items, or attributes) to refine the node's embedding, and employs an attention mechanism to discriminate the importance of the neighbors. Our KGAT is conceptually advantageous to existing KG-based recommendation methods, which either exploit high-order relations by extracting paths or implicitly modeling them with regularization. Empirical results on three public benchmarks show that KGAT significantly outperforms state-of-the-art methods like Neural FM and RippleNet. Further studies verify the efficacy of embedding propagation for high-order relation modeling and the interpretability benefits brought by the attention mechanism.

Temporal Meta-path Guided Explainable Recommendation

Abstract: This paper utilizes well-designed item-item path modelling between consecutive items with attention mechanisms to sequentially model dynamic user-item evolutions on dynamic knowledge graph for explainable recommendations. Compared with existing works that use heavy recurrent neural networks to model temporal information, we propose simple but effective neural networks to capture user historical item features and path-based context to characterise next purchased item. Extensive evaluations of TMER on three real-world benchmark datasets show state-of-the-art performance compared against recent strong baselines.

Learning Fair Representations for Recommendation: A Graph-based Perspective

Abstract: As a key application of artificial intelligence, recommender systems are among the most pervasive computer aided systems to help users find potential items of interests. Recently, researchers paid considerable attention to fairness issues for artificial intelligence applications. Most of these approaches assumed independence of instances, and designed sophisticated models to eliminate the sensitive information to facilitate fairness. However, recommender systems differ greatly from these approaches as users and items naturally form a user-item bipartite graph, and are collaboratively correlated in the graph structure. In this paper, we propose a novel graph based technique for ensuring fairness of any recommendation models. Here, the fairness requirements refer to not exposing sensitive feature set in the user modeling process. Specifically, given the original embeddings from any recommendation models, we learn a composition of filters that transform each user's and each item's original embeddings into a filtered embedding space based on the sensitive feature set. For each user, this transformation is achieved under the adversarial learning of a user-centric graph, in order to obfuscate each sensitive feature between both the filtered user embedding and the sub graph structures of this user. Finally, extensive experimental results clearly show the effectiveness of our proposed model for fair recommendation. We publish the source code at https://github.com/newlei/FairGo.

Say No to the Discrimination: Learning Fair Graph Neural Networks with Limited Sensitive Attribute Information

Abstract: Graph neural networks (GNNs) have shown great power in modeling graph structured data. However, similar to other machine learning models, GNNs may make predictions biased on protected sensitive attributes, e.g., skin color and gender. Because machine learning algorithms including GNNs are trained to reflect the distribution of the training data which often contains historical bias towards sensitive attributes. In addition, the discrimination in GNNs can be magnified by graph structures and the message-passing mechanism. As a result, the applications of GNNs in sensitive domains such as crime rate prediction would be largely limited. Though extensive studies of fair classification have been conducted on i.i.d data, methods to address the problem of discrimination on non-i.i.d data are rather limited. Furthermore, the practical scenario of sparse annotations in sensitive attributes is rarely considered in existing works. Therefore, we study the novel and important problem of learning fair GNNs with limited sensitive attribute information. FairGNN is proposed to eliminate the bias of GNNs whilst maintaining high node classification accuracy by leveraging graph structures and limited sensitive information. Our theoretical analysis shows that FairGNN can ensure the fairness of GNNs under mild conditions given limited nodes with known sensitive attributes. Extensive experiments on real-world datasets also demonstrate the effectiveness of FairGNN in debiasing and keeping high accuracy.

In recent years, GNN has been applied to a wide range of recommendation tasks. Here we define the taxonomy in terms of recommendation stages, scenarios, objectives, and applications, respectively. To be more specific, recommendation stages indicate the overall procedure that a recommender system is implemented in the real-world platform. The procedure includes matching for item candidate selection, ranking for capturing user preferences, and re-ranking for other criteria beyond accuracy. The recommendation scenarios include social recommendation, sequential recommendation, cross-domain recommendation, etc. The recommendation objectives incorporate accuracy, diversity, explainability, fairness, and so on, in which accuracy is of the most concern. The recommendation application refers to specific industrial applications. Table {{table:047d04de-5557-42ff-881c-0c13e2952c51}} , {{table:62d71d5c-9ff9-4427-8589-492074eebf09}} , and {{table:826a5ac4-a43a-488e-9213-79841ac17b19}}  show representative researches of GNN-based recommendation published in top-tier venues for different recommendation stages, recommendation scenarios, and recommendation objectives, respectively.


re-define distance metrics between nodes

Graph Convolutional Matrix Completion

Abstract: We consider matrix completion for recommender systems from the point of view of link prediction on graphs. Interaction data such as movie ratings can be represented by a bipartite user-item graph with labeled edges denoting observed ratings. Building on recent progress in deep learning on graph-structured data, we propose a graph auto-encoder framework based on differentiable message passing on the bipartite interaction graph. Our model shows competitive performance on standard collaborative filtering benchmarks. In settings where complimentary feature information or structured data such as a social network is available, our framework outperforms recent state-of-the-art methods.

Neural Graph Collaborative Filtering

Abstract: Learning vector representations (aka. embeddings) of users and items lies at the core of modern recommender systems. Ranging from early matrix factorization to recently emerged deep learning based methods, existing efforts typically obtain a user's (or an item's) embedding by mapping from pre-existing features that describe the user (or the item), such as ID and attributes. We argue that an inherent drawback of such methods is that, the collaborative signal, which is latent in user-item interactions, is not encoded in the embedding process. As such, the resultant embeddings may not be sufficient to capture the collaborative filtering effect. In this work, we propose to integrate the user-item interactions -- more specifically the bipartite graph structure -- into the embedding process. We develop a new recommendation framework Neural Graph Collaborative Filtering (NGCF), which exploits the user-item graph structure by propagating embeddings on it. This leads to the expressive modeling of high-order connectivity in user-item graph, effectively injecting the collaborative signal into the embedding process in an explicit manner. We conduct extensive experiments on three public benchmarks, demonstrating significant improvements over several state-of-the-art models like HOP-Rec and Collaborative Memory Network. Further analysis verifies the importance of embedding propagation for learning better user and item representations, justifying the rationality and effectiveness of NGCF. Codes are available at https://github.com/xiangwang1223/neural_graph_collaborative_filtering.

Abstract: Learning informative representations of users and items from the interaction data is of crucial importance to collaborative filtering (CF). Present embedding functions exploit user-item relationships to enrich the representations, evolving from a single user-item instance to the holistic interaction graph. Nevertheless, they largely model the relationships in a uniform manner, while neglecting the diversity of user intents on adopting the items, which could be to pass time, for interest, or shopping for others like families. Such uniform approach to model user interests easily results in suboptimal representations, failing to model diverse relationships and disentangle user intents in representations. In this work, we pay special attention to user-item relationships at the finer granularity of user intents. We hence devise a new model, Disentangled Graph Collaborative Filtering (DGCF), to disentangle these factors and yield disentangled representations. Specifically, by modeling a distribution over intents for each user-item interaction, we iteratively refine the intent-aware interaction graphs and representations. Meanwhile, we encourage independence of different intents. This leads to disentangled representations, effectively distilling information pertinent to each intent. We conduct extensive experiments on three benchmark datasets, and DGCF achieves significant improvements over several state-of-the-art models like NGCF, DisenGCN, and MacridVAE. Further analyses offer insights into the advantages of DGCF on the disentanglement of user intents and interpretability of representations. Our codes are available in https://github.com/xiangwang1223/disentangled_graph_collaborative_filtering.

LightGCN: Simplifying and Powering Graph Convolution Network for Recommendation

Abstract: Graph Convolution Network (GCN) has become new state-of-the-art for collaborative filtering. Nevertheless, the reasons of its effectiveness for recommendation are not well understood. Existing work that adapts GCN to recommendation lacks thorough ablation analyses on GCN, which is originally designed for graph classification tasks and equipped with many neural network operations. However, we empirically find that the two most common designs in GCNs -- feature transformation and nonlinear activation -- contribute little to the performance of collaborative filtering. Even worse, including them adds to the difficulty of training and degrades recommendation performance. In this work, we aim to simplify the design of GCN to make it more concise and appropriate for recommendation. We propose a new model named LightGCN, including only the most essential component in GCN -- neighborhood aggregation -- for collaborative filtering. Specifically, LightGCN learns user and item embeddings by linearly propagating them on the user-item interaction graph, and uses the weighted sum of the embeddings learned at all layers as the final embedding. Such simple, linear, and neat model is much easier to implement and train, exhibiting substantial improvements (about 16.0\% relative improvement on average) over Neural Graph Collaborative Filtering (NGCF) -- a state-of-the-art GCN-based recommender model -- under exactly the same experimental setting. Further analyses are provided towards the rationality of the simple LightGCN from both analytical and empirical perspectives.

Self-supervised Graph Learning for Recommendation

Abstract: Representation learning on user-item graph for recommendation has evolved from using single ID or interaction history to exploiting higher-order neighbors. This leads to the success of graph convolution networks (GCNs) for recommendation such as PinSage and LightGCN. Despite effectiveness, we argue that they suffer from two limitations: (1) high-degree nodes exert larger impact on the representation learning, deteriorating the recommendations of low-degree (long-tail) items; and (2) representations are vulnerable to noisy interactions, as the neighborhood aggregation scheme further enlarges the impact of observed edges. In this work, we explore self-supervised learning on user-item graph, so as to improve the accuracy and robustness of GCNs for recommendation. The idea is to supplement the classical supervised task of recommendation with an auxiliary self-supervised task, which reinforces node representation learning via self-discrimination. Specifically, we generate multiple views of a node, maximizing the agreement between different views of the same node compared to that of other nodes. We devise three operators to generate the views -- node dropout, edge dropout, and random walk -- that change the graph structure in different manners. We term this new learning paradigm as \textit{Self-supervised Graph Learning} (SGL), implementing it on the state-of-the-art model LightGCN. Through theoretical analyses, we find that SGL has the ability of automatically mining hard negatives. Empirical studies on three benchmark datasets demonstrate the effectiveness of SGL, which improves the recommendation accuracy, especially on long-tail items, and the robustness against interaction noises. Our implementations are available at \url{https://github.com/wujcan/SGL}.

Graph Convolutional Neural Networks for Web-Scale Recommender Systems

Abstract: Recent advancements in deep neural networks for graph-structured data have led to state-of-the-art performance on recommender system benchmarks. However, making these methods practical and scalable to web-scale recommendation tasks with billions of items and hundreds of millions of users remains a challenge. Here we describe a large-scale deep recommendation engine that we developed and deployed at Pinterest. We develop a data-efficient Graph Convolutional Network (GCN) algorithm PinSage, which combines efficient random walks and graph convolutions to generate embeddings of nodes (i.e., items) that incorporate both graph structure as well as node feature information. Compared to prior GCN approaches, we develop a novel method based on highly efficient random walks to structure the convolutions and design a novel training strategy that relies on harder-and-harder training examples to improve robustness and convergence of the model. We also develop an efficient MapReduce model inference algorithm to generate embeddings using a trained model. We deploy PinSage at Pinterest and train it on 7.5 billion examples on a graph with 3 billion nodes representing pins and boards, and 18 billion edges. According to offline metrics, user studies and A/B tests, PinSage generates higher-quality recommendations than comparable deep learning and graph-based alternatives. To our knowledge, this is the largest application of deep graph embeddings to date and paves the way for a new generation of web-scale recommender systems based on graph convolutional architectures.

Interest-aware Message-Passing GCN for Recommendation

Abstract: Graph Convolution Networks (GCNs) manifest great potential in recommendation. This is attributed to their capability on learning good user and item embeddings by exploiting the collaborative signals from the high-order neighbors. Like other GCN models, the GCN based recommendation models also suffer from the notorious over-smoothing problem - when stacking more layers, node embeddings become more similar and eventually indistinguishable, resulted in performance degradation. The recently proposed LightGCN and LR-GCN alleviate this problem to some extent, however, we argue that they overlook an important factor for the over-smoothing problem in recommendation, that is, high-order neighboring users with no common interests of a user can be also involved in the user's embedding learning in the graph convolution operation. As a result, the multi-layer graph convolution will make users with dissimilar interests have similar embeddings. In this paper, we propose a novel Interest-aware Message-Passing GCN (IMP-GCN) recommendation model, which performs high-order graph convolution inside subgraphs. The subgraph consists of users with similar interests and their interacted items. To form the subgraphs, we design an unsupervised subgraph generation module, which can effectively identify users with common interests by exploiting both user feature and graph structure. To this end, our model can avoid propagating negative information from high-order neighbors into embedding learning. Experimental results on three large-scale benchmark datasets show that our model can gain performance improvement by stacking more layers and outperform the state-of-the-art GCN-based recommendation models significantly.

HS-GCN: Hamming Spatial Graph Convolutional Networks for Recommendation

Abstract: An efficient solution to the large-scale recommender system is to represent users and items as binary hash codes in the Hamming space. Towards this end, existing methods tend to code users by modeling their Hamming similarities with the items they historically interact with, which are termed as the first-order similarities in this work. Despite their efficiency, these methods suffer from the suboptimal representative capacity, since they forgo the correlation established by connecting multiple first-order similarities, i.e., the relation among the indirect instances, which could be defined as the high-order similarity. To tackle this drawback, we propose to model both the first- and the high-order similarities in the Hamming space through the user-item bipartite graph. Therefore, we develop a novel learning to hash framework, namely Hamming Spatial Graph Convolutional Networks (HS-GCN), which explicitly models the Hamming similarity and embeds it into the codes of users and items. Extensive experiments on three public benchmark datasets demonstrate that our proposed model significantly outperforms several state-of-the-art hashing models, and obtains performance comparable with the real-valued recommendation models.

Details of GNN models in matching stage.

In the matching stage, efficiency is an essential problem because of the high computation complexity for candidate selection. Specifically, only hundreds of items will be selected from the item pool of million magnitudes for the following ranking stage, based on coarse-grained user preferences. Therefore, proposed models in this stage barely leverage user-item interactions as data input for modeling user preferences without introducing additional features such as user ages, item price, browsing time on the application, etc.
For matching, besides the basic challenges of graph construction and network design, there is also the challenge of CF signal extraction, as shown in Table {{table:b35ec3bc-ac90-40cf-ab96-4292ed2403c0}} .

GNN-based models in the matching stage can be regarded as embedding matching, usually designing specialized GNN architecture on the user-item bipartite graph {{cite:0eb11d8}}, {{cite:6f75dbe}}, {{cite:bbf0880}}, {{cite:2b98d4e}}, {{cite:97320c1}}. Berg et al. {{cite:0eb11d8}} proposed to pass neighborhood messages by summing and assign weight-sharing transformation channels for different relational edges (i.e., user-item ratings). Wang et al. {{cite:6f75dbe}} proposed a spatial GNN in recommendation and obtain superior performance compared with conventional CF methods like MF {{cite:67c1672}} or NCF {{cite:c66e130}}. Sun et al. {{cite:bbf0880}} argued that simple aggregation mechanisms like sum, mean, or max cannot model relational information among neighbors and proposed a neighbor interaction-aware convolution to address the issue. Wang et al. {{cite:2b98d4e}} developed disentangled GNN to capture independent user intentions, which extends the set of candidate items in matching and guarantees the accuracy simultaneously. Wu et al. {{cite:97320c1}} leverage the stability of graph structure to incorporate a contrastive learning framework to assist representation learning. These GNN-based models can capture high-order similarity among users and items as well as structural connectivity. In this way, the semantics that users with similar interactions will have similar preferences are extended through multiple times of information propagation. On the other hand, the training complexity of GNN-based models was demonstrated {{cite:6f75dbe}}, {{cite:2b98d4e}} as acceptable and comparable with non-graph models, especially when the transformation matrix is removed {{cite:d07db82}}. Besides, {{cite:3553126}} showed that the GNN-based model could be applied to web-scale recommender systems in real-world platforms efficiently and effectively, which combines random walk and GraphSAGE {{cite:c915687}} for embedding learning on a large-scale item-item graph. Table {{table:b35ec3bc-ac90-40cf-ab96-4292ed2403c0}}  shows the commonality and difference among the GNN models in matching
 {{cite:c6dacbf}} proposed first to filter promising neighbors and then adopt embedding propagation.


Privacy-preserving and Efficient Aggregation based on Blockchain for Power Grid Communications in Smart Communities

Abstract: Intelligence is one of the most important aspects in the development of our future communities. Ranging from smart home, smart building, to smart city, all these smart infrastructures must be supported by intelligent power supply. Smart grid is proposed to solve all challenges of future electricity supply. In smart grid, in order to realize optimal scheduling, a Smart Meter (SM) is installed at each home to collect the near real-time electricity consumption data, which can be used by the utilities to offer better smart home services. However, the near real-time data may disclose user's privacy. An adversary may track the application usage patterns by analyzing the user's electricity consumption profile. In this paper, we propose a privacy-preserving and efficient data aggregation scheme. We divide users into different groups and each group has a private blockchain to record its members' data. To preserve the inner privacy within a group, we use pseudonym to hide user's identity, and each user may create multiple pseudonyms and associate his/her data with different pseudonyms. In addition, the bloom filter is adopted for fast authentication. The analysis shows that the proposed scheme can meet the security requirements, and achieve a better performance than other popular methods.

Privacy-Preserving Peer-to-Peer Energy Trading in Blockchain-Enabled Smart Grids Using Functional Encryption

Abstract: Advanced smart grid technologies enable energy prosumers to trade surplus energy from their distributed renewable energy sources with other peer prosumers through peer-to-peer (P2P) energy trading. In many previous works, P2P energy trading was facilitated by blockchain technology through blockchain’s distributive nature and capacity to run smart contracts. However, the feature that all the data and transactions on a blockchain are visible to all blockchain nodes may significantly threaten the privacy of the parties participating in P2P energy trading. There are many previous works that have attempted to mitigate this problem. However, all these works focused on the anonymity of participants but did not protect the data and transactions. To address this issue, we propose a P2P energy trading system on a blockchain where all bids are encrypted and peer matching is performed on the encrypted bids by a functional encryption-based smart contract. The system guarantees that the information encoded in the encrypted bids is protected, but the peer matching transactions are performed by the nodes in a publicly verifiable manner through smart contracts. We verify the feasibility of the proposed system by implementing a prototype composed of smart meters, a distribution system operator (DSO) server, and private Ethereum blockchain.

Blockchain For Security And Privacy Protection

We summarize the design goals, key technologies, and other blockchain-related attributes of the existing solutions mentioned above. The results are reported in Table {{table:1c161b7a-260b-4183-9381-7518739913ac}} .


LRW-1000: A Naturally-Distributed Large-Scale Benchmark for Lip Reading in the Wild

Abstract: Large-scale datasets have successively proven their fundamental importance in several research fields, especially for early progress in some emerging topics. In this paper, we focus on the problem of visual speech recognition, also known as lip-reading, which has received increasing interest in recent years. We present a naturally-distributed large-scale benchmark for lip-reading in the wild, named LRW-1000, which contains 1,000 classes with 718,018 samples from more than 2,000 individual speakers. Each class corresponds to the syllables of a Mandarin word composed of one or several Chinese characters. To the best of our knowledge, it is currently the largest word-level lipreading dataset and also the only public large-scale Mandarin lip-reading dataset. This dataset aims at covering a "natural" variability over different speech modes and imaging conditions to incorporate challenges encountered in practical applications. It has shown a large variation in this benchmark in several aspects, including the number of samples in each class, video resolution, lighting conditions, and speakers’ attributes such as pose, age, gender, and make-up. Besides providing a detailed description of the dataset and its collection pipeline, we evaluate several typical popular lip-reading methods and perform a thorough analysis of the results from several aspects. The results demonstrate the consistency and challenges of our dataset, which may open up some new promising directions for future work.

Learn an Effective Lip Reading Model without Pains

Abstract: Lip reading, also known as visual speech recognition, aims to recognize the speech content from videos by analyzing the lip dynamics. There have been several appealing progress in recent years, benefiting much from the rapidly developed deep learning techniques and the recent large-scale lip-reading datasets. Most existing methods obtained high performance by constructing a complex neural network, together with several customized training strategies which were always given in a very brief description or even shown only in the source code. We find that making proper use of these strategies could always bring exciting improvements without changing much of the model. Considering the non-negligible effects of these strategies and the existing tough status to train an effective lip reading model, we perform a comprehensive quantitative study and comparative analysis, for the first time, to show the effects of several different choices for lip reading. By only introducing some easy-to-get refinements to the baseline pipeline, we obtain an obvious improvement of the performance from 83.7% to 88.4% and from 38.2% to 55.7% on two largest public available lip reading datasets, LRW and LRW-1000, respectively. They are comparable and even surpass the existing state-of-the-art results.

Deformation Flow Based Two-Stream Network for Lip Reading

Abstract: Lip reading is the task of recognizing the speech content by analyzing movements in the lip region when people are speaking. Observing on the continuity in adjacent frames in the speaking process, and the consistency of the motion patterns among different speakers when they pronounce the same phoneme, we model the lip movements in the speaking process as a sequence of apparent deformations in the lip region. Specifically, we introduce a Deformation Flow Network (DFN) to learn the deformation flow between adjacent frames, which directly captures the motion information within the lip region. The learned deformation flow is then combined with the original grayscale frames with a two-stream network to perform lip reading. Different from previous two-stream networks, we make the two streams learn from each other in the learning process by introducing a bidirectional knowledge distillation loss to train the two branches jointly. Owing to the complementary cues provided by different branches, the two-stream network shows a substantial improvement over using either single branch. A thorough experimental evaluation on two large-scale lip reading benchmarks is presented with detailed analysis. The results accord with our motivation, and show that our method achieves state-of-the-art or comparable performance on these two challenging datasets.

Lipreading Using Temporal Convolutional Networks

Abstract: Lip-reading has attracted a lot of research attention lately thanks to advances in deep learning. The current state-of-the-art model for recognition of isolated words in-the-wild consists of a residual network and Bidirectional Gated Recurrent Unit (BGRU) layers. In this work, we address the limitations of this model and we propose changes which further improve its performance. Firstly, the BGRU layers are replaced with Temporal Convolutional Networks (TCN). Secondly, we greatly simplify the training procedure, which allows us to train the model in one single stage. Thirdly, we show that the current state-of-the-art methodology produces models that do not generalize well to variations on the sequence length, and we address this issue by proposing a variable-length augmentation. We present results on the largest publicly-available datasets for isolated word recognition in English and Mandarin, LRW and LRW1000, respectively. Our proposed model1 results in an absolute improvement of 1.2% and 3.2%, respectively, in these datasets which is the new state-of-the-art performance.

LRWR benchmark

Moreover, we adapted a method from the field of action recognition – the channel-separated convolutional networks {{cite:d0bfe6f}}. The adaptation includes usage of recurrent layers as a backend, following the idea presented in article {{cite:aa6d307}}.
As another approach for backend architecture, we replace 3D convolution with (2+1)D-convolution {{cite:0e8b450}}.
The final results of methods comparison are shown in the Table {{table:09a1d639-8f6c-47af-9b94-3d5d9bf33daf}} .

Experiments have shown that the combination of LabelSmoothing, MixUp, and CosineScheduler with increased RNN hidden size positively affects the resulting performance.
We also found a general fact for LRWR: the quality of any network increases if we upsample each class up to 750 samples, while further upsampling leads to performance degradation.
Moreover, pretrained weights initialization leads to better results in all our experiments that provide promising directions for lipreading models finetuning in future research.
Finally, while D3D demonstrates the greatest performance on the LRWR dataset (Table {{table:09a1d639-8f6c-47af-9b94-3d5d9bf33daf}} ), it has the lowest accuracy on the LRW (Table {{table:5eb680a3-4107-4c98-a9d0-cf1bdddefd87}} ), which could demonstrate the differences between the benchmarked languages

LRW improvement – AttentionLipreadingNet

AttentionLipreadingNet.
Analyzing the differences between published (Table {{table:5eb680a3-4107-4c98-a9d0-cf1bdddefd87}} ) and benchmarked (Table {{table:09a1d639-8f6c-47af-9b94-3d5d9bf33daf}} ) results, we would like to propose a neural architecture that will work effectively on both datasets. Despite the fact that on LRWR dataset D3D{{cite:efc8d2e}} shows the best results, this network demonstrate lower quality on the LRW compared to networks using a combination of 3D and 2D convolutions. Therefore, to test our developments on the LRW, we need to develop new network, and apply all the techniques that led to the accuracy improvement on the LRWR.


Spatio-Temporal Graph Convolutional Networks: A Deep Learning Framework for Traffic Forecasting

Abstract: Timely accurate traffic forecast is crucial for urban traffic control and guidance. Due to the high nonlinearity and complexity of traffic flow, traditional methods cannot satisfy the requirements of mid-and-long term prediction tasks and often neglect spatial and temporal dependencies. In this paper, we propose a novel deep learning framework, Spatio-Temporal Graph Convolutional Networks (STGCN), to tackle the time series prediction problem in traffic domain. Instead of applying regular convolutional and recurrent units, we formulate the problem on graphs and build the model with complete convolutional structures, which enable much faster training speed with fewer parameters. Experiments show that our model STGCN effectively captures comprehensive spatio-temporal correlations through modeling multi-scale traffic networks and consistently outperforms state-of-the-art baselines on various real-world traffic datasets.

Deep Multi-View Spatial-Temporal Network for Taxi Demand Prediction

Abstract: Taxi demand prediction is an important building block to enabling intelligent transportation systems in a smart city. An accurate prediction model can help the city pre-allocate resources to meet travel demand and to reduce empty taxis on streets which waste energy and worsen the traffic congestion. With the increasing popularity of taxi requesting services such as Uber and Didi Chuxing (in China), we are able to collect large-scale taxi demand data continuously. How to utilize such big data to improve the demand prediction is an interesting and critical real-world problem. Traditional demand prediction methods mostly rely on time series forecasting techniques, which fail to model the complex non-linear spatial and temporal relations. Recent advances in deep learning have shown superior performance on traditionally challenging tasks such as image classification by learning the complex features and correlations from large-scale data. This breakthrough has inspired researchers to explore deep learning techniques on traffic prediction problems. However, existing methods on traffic prediction have only considered spatial relation (e.g., using CNN) or temporal relation (e.g., using LSTM) independently. We propose a Deep Multi-View Spatial-Temporal Network (DMVST-Net) framework to model both spatial and temporal relations. Specifically, our proposed model consists of three views: temporal view (modeling correlations between future demand values with near time points via LSTM), spatial view (modeling local spatial correlation via local CNN), and semantic view (modeling correlations among regions sharing similar temporal patterns). Experiments on large-scale real taxi demand data demonstrate effectiveness of our approach over state-of-the-art methods.

Revisiting Spatial-Temporal Similarity: A Deep Learning Framework for Traffic Prediction

Abstract: Traffic prediction has drawn increasing attention in AI research field due to the increasing availability of large-scale traffic data and its importance in the real world. For example, an accurate taxi demand prediction can assist taxi companies in pre-allocating taxis. The key challenge of traffic prediction lies in how to model the complex spatial dependencies and temporal dynamics. Although both factors have been considered in modeling, existing works make strong assumptions about spatial dependence and temporal dynamics, i.e., spatial dependence is stationary in time, and temporal dynamics is strictly periodical. However, in practice the spatial dependence could be dynamic (i.e., changing from time to time), and the temporal dynamics could have some perturbation from one period to another period. In this paper, we make two important observations: (1) the spatial dependencies between locations are dynamic; and (2) the temporal dependency follows daily and weekly pattern but it is not strictly periodic for its dynamic temporal shifting. To address these two issues, we propose a novel Spatial-Temporal Dynamic Network (STDN), in which a flow gating mechanism is introduced to learn the dynamic similarity between locations, and a periodically shifted attention mechanism is designed to handle long-term periodic temporal shifting. To the best of our knowledge, this is the first work that tackle both issues in a unified framework. Our experimental results on real-world traffic datasets verify the effectiveness of the proposed method.

Graph WaveNet for Deep Spatial-Temporal Graph Modeling

Abstract: Spatial-temporal graph modeling is an important task to analyze the spatial relations and temporal trends of components in a system. Existing approaches mostly capture the spatial dependency on a fixed graph structure, assuming that the underlying relation between entities is pre-determined. However, the explicit graph structure (relation) does not necessarily reflect the true dependency and genuine relation may be missing due to the incomplete connections in the data. Furthermore, existing methods are ineffective to capture the temporal trends as the RNNs or CNNs employed in these methods cannot capture long-range temporal sequences. To overcome these limitations, we propose in this paper a novel graph neural network architecture, Graph WaveNet, for spatial-temporal graph modeling. By developing a novel adaptive dependency matrix and learn it through node embedding, our model can precisely capture the hidden spatial dependency in the data. With a stacked dilated 1D convolution component whose receptive field grows exponentially as the number of layers increases, Graph WaveNet is able to handle very long sequences. These two components are integrated seamlessly in a unified framework and the whole framework is learned in an end-to-end manner. Experimental results on two public traffic network datasets, METR-LA and PEMS-BAY, demonstrate the superior performance of our algorithm.

GMAN: A Graph Multi-Attention Network for Traffic Prediction

Abstract: Long-term traffic prediction is highly challenging due to the complexity of traffic systems and the constantly changing nature of many impacting factors. In this paper, we focus on the spatio-temporal factors, and propose a graph multi-attention network (GMAN) to predict traffic conditions for time steps ahead at different locations on a road network graph. GMAN adapts an encoder-decoder architecture, where both the encoder and the decoder consist of multiple spatio-temporal attention blocks to model the impact of the spatio-temporal factors on traffic conditions. The encoder encodes the input traffic features and the decoder predicts the output sequence. Between the encoder and the decoder, a transform attention layer is applied to convert the encoded traffic features to generate the sequence representations of future time steps as the input of the decoder. The transform attention mechanism models the direct relationships between historical and future time steps that helps to alleviate the error propagation problem among prediction time steps. Experimental results on two real-world traffic prediction tasks (i.e., traffic volume prediction and traffic speed prediction) demonstrate the superiority of GMAN. In particular, in the 1 hour ahead prediction, GMAN outperforms state-of-the-art methods by up to 4% improvement in MAE measure. The source code is available at https://github.com/zhengchuanpan/GMAN.

Connecting the Dots: Multivariate Time Series Forecasting with Graph Neural Networks

Abstract: Modeling multivariate time series has long been a subject that has attracted researchers from a diverse range of fields including economics, finance, and traffic. A basic assumption behind multivariate time series forecasting is that its variables depend on one another but, upon looking closely, it is fair to say that existing methods fail to fully exploit latent spatial dependencies between pairs of variables. In recent years, meanwhile, graph neural networks (GNNs) have shown high capability in handling relational dependencies. GNNs require well-defined graph structures for information propagation which means they cannot be applied directly for multivariate time series where the dependencies are not known in advance. In this paper, we propose a general graph neural network framework designed specifically for multivariate time series data. Our approach automatically extracts the uni-directed relations among variables through a graph learning module, into which external knowledge like variable attributes can be easily integrated. A novel mix-hop propagation layer and a dilated inception layer are further proposed to capture the spatial and temporal dependencies within the time series. The graph learning, graph convolution, and temporal convolution modules are jointly learned in an end-to-end framework. Experimental results show that our proposed model outperforms the state-of-the-art baseline methods on 3 of 4 benchmark datasets and achieves on-par performance with other approaches on two traffic datasets which provide extra structural information.

Existing methods fall into the same paradigm

Graph neural networks.
More recently, graph-based deep learning models are increasingly used for spatio-temporal data analysis. Yu et al. {{cite:c5ca68a}} propose a spatio-temporal graph convolutional network based on spectral graph theory to extract features from the road network and the historical time series data. In particular, the spatial patterns are captured by graph convolutional layers while the temporal patterns are extracted by gated convolutional layers. Li et al. {{cite:e913383}} model the traffic flow as a diffusion process and capture the spatial dependencies by random walks on a graph followed by an RNN to extract temporal patterns. Yao et al. {{cite:be2c1c1}} use graph embeddings to capture the road network information and integrate Long-Short Term Memory (LSTM) and CNN with the graph embeddings for traffic prediction. Attention mechanism has also been adopted to learn spatial and temporal patterns with improved performance {{cite:ebe4bad}}, {{cite:46fbf66}}. However, all of these methods fall into the same paradigm that separate layers are applied to extract spatial patterns and temporal patterns respectively. We summarize current methods under this paradigm in Table {{table:030be659-b4b6-48c5-86d9-9d8251c5ce2e}} . Particularly, CNN, Graph Convolutional Network (GCN), or spatial attention are often used to learn spatial patterns. Gated Recurrent Unit (GRU), LSTM, gated temporal convolution (gated TCN), or temporal attention are used to learn temporal patterns.


A reinforcement learning extension to the Almgren-Chriss framework for optimal trade execution

Abstract: Reinforcement learning is explored as a candidate machine learning technique to enhance existing analytical solutions for optimal trade execution with elements from the market microstructure. Given a volume-to-trade, fixed time horizon and discrete trading periods, the aim is to adapt a given volume trajectory such that it is dynamic with respect to favourable/unfavourable conditions during realtime execution, thereby improving overall cost of trading. We consider the standard Almgren-Chriss model with linear price impact as a candidate base model. This model is popular amongst sell-side institutions as a basis for arrival price benchmark execution algorithms. By training a learning agent to modify a volume trajectory based on the market's prevailing spread and volume dynamics, we are able to improve post-trade implementation shortfall by up to 10.3% on average compared to the base model, based on a sample of stocks and trade sizes in the South African equity market.

Double Deep Q-Learning for Optimal Execution

Abstract: Optimal trade execution is an important problem faced by essentially all traders. Much research into optimal execution uses stringent model assumptions and applies continuous time stochastic control to solve them. Here, we instead take a model free approach and develop a variation of Deep Q-Learning to estimate the optimal actions of a trader. The model is a fully connected Neural Network trained using Experience Replay and Double DQN with input features given by the current state of the limit order book, other trading signals, and available execution actions, while the output is the Q-value function estimating the future rewards under an arbitrary action. We apply our model to nine different stocks and find that it outperforms the standard benchmark approach on most stocks using the measures of (i) mean and median out-performance, (ii) probability of out-performance, and (iii) gain-loss ratios.

Summary of RL for Order Execution

We present a summary of existing RL-based order execution publications in Table {{table:65dc3491-0601-4ea4-b41b-6bbac0cc4c64}} . Although there are a few successful examples using either Q-learning or PPO on order execution, existing works share a few limitations. First, most of algorithms are only tested on stock data. Their performance on different financial assets (e.g., futures and cryptocurrency) is still unclear. Second, the execution time window (e.g., 1 day) is too long, which makes the task easier. In practice, professional traders usually finish the execution process in much shorter time window (e.g., 10 minute). Third, existing works will fail when the trading volume is huge, because all of them assume there is no obvious market impact, which is impossible for large volume settings. In the real-world, the requirement of institutional investors is to execute large amount of shares in a relatively short time window. There is still a long way to go for researchers to tackle these limitations.


Link Budget Analysis for Reconfigurable Smart Surfaces in Aerial Platforms

Abstract: Non-terrestrial networks, including Unmanned Aerial Vehicles (UAVs), High Altitude Platform Station (HAPS) nodes and Low Earth Orbiting (LEO) satellites, are expected to have a pivotal role in sixth-generation wireless networks. With inherent features such as flexible placement, wide footprints, and preferred channel conditions, they can tackle several challenges faced by current terrestrial networks. However, their successful and widespread adoption relies on energy-efficient on-board communication systems. In this context, the integration of Reconfigurable Smart Surfaces (RSS) into aerial platforms is envisioned as a key enabler of energy-efficient and cost-effective aerial platform deployments. RSS consist of low-cost reflectors capable of smartly directing signals in a nearly passive way. In this paper, we investigate the link budget of RSS-assisted communications for two RSS reflection paradigms discussed in the literature, namely “specular” and “scattering” paradigms. Specifically, we analyze the characteristics of RSS-equipped aerial platforms and compare their communication performance with that of RSS-assisted terrestrial networks using standardized channel models. In addition, we derive the optimal aerial platform placements for both reflection paradigms. Our results provide important insights for the design of RSS-assisted communications. For instance, given that a HAPS has a large area for RSS, it provides superior link budget performance in most studied scenarios. In contrast, the limited RSS area on UAVs and the large propagation loss in LEO satellite communications make them unfavorable candidates for supporting terrestrial users. Finally, the optimal location of an RSS-equipped platform may depend on the platform’s altitude, coverage footprint, and type of environment

Optimization of Wireless Relaying With Flexible UAV-Borne Reflecting Surfaces

Abstract: This paper presents a theoretical framework to analyze the performance of an integrated unmanned aerial vehicle (UAV)-intelligent reflecting surface (IRS) relaying system in which the IRS provides an additional degree of freedom combined with the flexible deployment of full-duplex UAV to enhance communication between ground nodes. Our framework considers three different transmission modes: (i) UAV-only mode, (ii) IRS-only mode, and (iii) integrated UAV-IRS mode to achieve spectral and energy-efficient relaying. For the proposed modes, we provide exact and approximate expressions for the end-to-end outage probability, ergodic capacity, and energy efficiency (EE) in closed-form. We use the derived expressions to optimize key system parameters such as the UAV altitude and the number of elements on the IRS considering different modes. We formulate the problems in the form of fractional programming (e.g. single ratio, sum of multiple ratios or maximization-minimization of ratios) and devise optimal algorithms using quadratic transformations. Furthermore, we derive an analytic criterion to optimally select different transmission modes to maximize ergodic capacity and EE for a given number of IRS elements. Numerical results validate the derived expressions. The solutions obtained from the proposed optimization algorithms are compared with those obtained through exhaustive search. Insights are drawn related to the different communication modes, optimal number of IRS elements, and optimal UAV height.

IRS-Assisted UAV Communications with Imperfect Phase Compensation

Abstract:
This work presents a performance analysis on unmanned aerial vehicles (UAVs) assisted wireless

communications systems, where one of the UAVs supports intelligent reflecting surfaces (IRS). As the

estimation and compensation of the end-to-end phase for each propagation path is prone to errors, imperfect

phase compensation at the IRS is taken into consideration. The performance is derived in terms of symbol

error rate (SER) and outage probability, where the phase error is modeled using the von Mises distribution.

The air-to-air (A2A) channel for

each propagation path is modeled as a single dominant line-of-sight (LoS) component, and the results are

compared to the Rician channel model. The obtained results reveal that the considered A2A model can be

used to accurately represent the A2A channel with Rician fading.

Summary of existing performance studies of RIS-assisted NTNs

Summary: Table {{table:c66b77f6-38c6-4657-92a7-ed43bde2b00d}}  summarized the existing performance analysis works reviewed above. The majority of works derived approximations of the underlying performance indexes due to the reflection at RIS introducing product channels that are less mathematically tractable.
Moreover, important practical factors such as the near-field and the correlation between reflecting elements are hardly considered. The considered channel models are not realistic, with most of the studies relying on a Rician distributed channel. All this shows that there is still room for further analytic developments based on accurate channel models that fit with experimental data.


ARIS's position and phase shifts, transmit beamforming

ARIS's phase shifts, transmitter beamforming

UAV's trajectory, TRIS' phase shifts, RIS scheduling

UAV's trajectory, TRIS scheduling, subcarrier allocation, power allocation

TRIS's phase shift, UAV's location, power allocation, and NOMA decoding orders,

UAVs' position, TRISs' phase shifts, user scheduling, RIS scheduling

ARIS's phase shift and array partition, HAP's position, power allocation,

UAV's trajectory, TRIS's phase shifts, THz sub-bands allocation, power allocation

UAV's transmit power and location, TRIS's phase shifts,

UAV's trajectory, TRIS's phase shift, and gU's transmit power

UAV's trajectory, TRIS's phase shifts, user scheduling, power allocation

ARIS's phase shift and location, transmit precoding,

fraction programming, dual decomposition

G2A MISO gBS-multiple ABS backhual system

A2G P2P downlink and uplink secure system

Aerial Intelligent Reflecting Surface: Joint Placement and Passive Beamforming Design with 3D Beam Flattening

Abstract: This paper proposes a new three-dimensional (3D) wireless passive relaying system enabled by aerial IRS (AIRS). Compared to the conventional terrestrial IRS, AIRS enjoys more deployment flexibility as well as wider-range signal reflection, thanks to its high altitude and thus more likelihood of establishing line-of-sight (LoS) links with ground source/destination nodes. Specifically, we aim to maximize the worst-case signal-to-noise ratio (SNR) over all locations in a target area by jointly optimizing the transmit beamforming for the source node and the placement as well as 3D passive beamforming for the AIRS. The formulated problem is non-convex and thus difficult to solve. To gain useful insights, we first consider the special case of maximizing the SNR at a given target location, for which the optimal solution is obtained in closed-form. The result shows that the optimal horizontal AIRS placement only depends on the ratio between the source-destination distance and the AIRS altitude. Then for the general case of AIRS-enabled area coverage, we propose an efficient solution by decoupling the AIRS passive beamforming design to maximize the worst-case array gain, from its placement optimization by balancing the resulting angular span and the cascaded channel path loss. Our proposed solution is based on a novel 3D beam broadening and flattening technique, where the passive array of the AIRS is divided into sub-arrays of appropriate size, and their phase shifts are designed to form a flattened beam pattern with adjustable beamwidth catering to the size of the coverage area. Both the uniform linear array (ULA)-based and uniform planar array (UPA)-based AIRSs are considered in our design, which enable two-dimensional (2D) and 3D passive beamforming, respectively. Numerical results show that the proposed designs achieve significant performance gains over the benchmark schemes.

Distributional Reinforcement Learning for mmWave Communications with Intelligent Reflectors on a UAV

Abstract: In this paper, a novel communication framework that uses an unmanned aerial vehicle (UAV)-carried intelligent reflector (IR) is proposed to enhance multi-user downlink transmissions over millimeter wave (mmWave) frequencies. In order to maximize the downlink sum-rate, the optimal precoding matrix (at the base station) and reflection coefficient (at the IR) are jointly derived. Next, to address the uncertainty of mmWave channels and maintain line-of-sight links in a real-time manner, a distributional reinforcement learning approach, based on quantile regression optimization, is proposed to learn the propagation environment of mmWave communications, and, then, optimize the location of the UAV-IR so as to maximize the long-term downlink communication capacity. Simulation results show that the proposed learning-based deployment of the UAV-IR yields a significant advantage, compared to a non-learning UAV-IR, a static IR, and a direct transmission schemes, in terms of the average data rate and the achievable line-of-sight probability of downlink mmWave communications.

UAV-Assisted Intelligent Reflecting Surface Symbiotic Radio System

Abstract: This paper investigates a symbiotic unmanned aerial vehicle (UAV)-assisted intelligent reflecting surface (IRS) radio system, where the UAV is leveraged to help the IRS reflect its own signals to the base station, and meanwhile enhance the UAV transmission by passive beamforming at the IRS. First, we consider the weighted sum bit error rate (BER) minimization problem among all IRSs by jointly optimizing the UAV trajectory, IRS phase shift matrix, and IRS scheduling, subject to the minimum primary rate requirements. To tackle this complicated problem, a relaxation-based algorithm is proposed. We prove that the converged relaxation scheduling variables are binary, which means that no reconstruct strategy is needed, and thus the UAV rate constraints are automatically satisfied. Second, we consider the fairness BER optimization problem. We find that the relaxation-based method cannot solve this fairness BER problem since the minimum primary rate requirements may not be satisfied by the binary reconstruction operation. To address this issue, we first transform the binary constraints into a series of equivalent equality constraints. Then, a penalty-based algorithm is proposed to obtain a suboptimal solution. Numerical results are provided to evaluate the performance of the proposed designs under different setups, as compared with benchmarks.

Sum-Rate Maximization for IRS-Assisted UAV OFDMA Communication Systems

Abstract: In this paper, we propose the use of intelligent reflecting surface (IRS) in unmanned aerial vehicle (UAV)- based orthogonal frequency division multiple access (OFDMA) communication systems. The proposed scheme exploits both the rich beamforming gain brought by the IRS and the high mobility of UAV for improving the system sum-rate. The joint design of UAV’s trajectory, IRS scheduling, and communication resource allocation for the proposed system is formulated as a non-convex optimization problem to maximize the system sum-rate. The existence of an IRS introduces both frequencyselectivity and spatial-selectivity in the fading of the composite channel from the UAV to ground users. To facilitate the design, we first derive the expression of the composite channel gain and propose a parametric approximation approach to establish a lower bound for the formulated problem. An alternating optimization algorithm is devised to handle the lower bound optimization problem. Simulation results unveil the promising sum-rate gain achieved by the deployment of an IRS in UAVbased communication systems.

Intelligent Reflecting Surface Enhanced Multi-UAV NOMA Networks

Abstract: Intelligent reflecting surface (IRS) enhanced multi-unmanned aerial vehicle (UAV) non-orthogonal multiple access (NOMA) networks are investigated. A new transmission framework is proposed, where multiple UAV-mounted base stations employ NOMA to serve multiple groups of ground users with the aid of an IRS. The three-dimensional (3D) placement and transmit power of UAVs, the reflection matrix of the IRS, and the NOMA decoding orders among users are jointly optimized for maximization of the sum rate of considered networks. To tackle the formulated mixed-integer non-convex optimization problem with coupled variables, a block coordinate descent (BCD)-based iterative algorithm is developed. Specifically, the original problem is decomposed into three subproblems, which are alternatingly solved by exploiting the penalty method and the successive convex approximation technique. The proposed BCD-based algorithm is demonstrated to be able to obtain a stationary point of the original problem with polynomial time complexity. Numerical results show that: 1) the proposed NOMA-IRS scheme for multi-UAV networks achieves a higher sum rate compared to the benchmark schemes, i.e., orthogonal multiple access (OMA)-IRS and NOMA without IRS; 2) the use of IRS is capable of providing performance gain for multi-UAV networks by both enhancing channel qualities of UAVs to their served users and mitigating the inter-UAV interference; and 3) optimizing the UAV placement can make the sum rate gain brought by NOMA more distinct due to the flexible decoding order design.

Optimal Resource Allocation for Multi-UAV Assisted Visible Light Communication

Abstract: In this paper, the optimization of deploying unmanned aerial vehicles (UAVs) over a reconfigurable intelligent surfaces (RISs)-assisted visible light communication (VLC) system is studied. In the considered model, UAVs are required to simultaneously provide wireless services as well as illumination for ground users. To meet the traffic and illumination demands of the ground users while minimizing the energy consumption of the UAVs, one must optimize UAV deployment, phase shift of RISs, user association and RIS association. This problem is formulated as an optimization problem whose goal is to minimize the transmit power of UAVs via adjusting UAV deployment, phase shift of RISs, user association and RIS association. To solve this problem, the original optimization problem is divided into four subproblems and an alternating algorithm is proposed. Specifically, phases alignment method and semidefinite program (SDP) algorithm are proposed to optimize the phase shift of RISs. Then, the UAV deployment optimization is solved by the successive convex approximation (SCA) algorithm. Since the problems of user association and RIS association are integer programming, the fraction relaxation method is adopted before using dual method to find the optimal solution. For simplicity, a greedy algorithm is proposed as an alternative to optimize RIS association. The proposed two schemes demonstrate the superior performance of 34:85% and 32:11% energy consumption reduction over the case without RIS, respectively, through extensive numerical study.

An Energy-efficient Aerial Backhaul System with Reconfigurable Intelligent Surface

Abstract: In this paper, we propose a novel wireless architecture, mounted on a high-altitude aerial platform, which is enabled by reconfigurable intelligent surface (RIS). By installing RIS on the aerial platform, rich line-of-sight and full-area coverage can be achieved, thereby, overcoming the limitations of the conventional terrestrial RIS. We consider a scenario where a sudden increase in traffic in an urban area triggers authorities to rapidly deploy unmanned-aerial vehicle base stations (UAV-BSs) to serve the ground users. In this scenario, since the direct backhaul link from the ground source can be blocked due to several obstacles from the urban area, we propose reflecting the backhaul signal using aerial-RIS so that it successfully reaches the UAV-BSs. We jointly optimize the placement and array-partition strategies of aerial-RIS and the phases of RIS elements, which leads to an increase in energy-efficiency of every UAV-BS. We show that the complexity of our algorithm can be bounded by the quadratic order, thus implying high computational efficiency. We verify the performance of the proposed algorithm via extensive numerical evaluations and show that our method achieves an outstanding performance in terms of energy-efficiency compared to benchmark schemes.

UAV-Assisted and Intelligent Reflecting Surfaces-Supported Terahertz Communications

Abstract: In this paper, unmanned aerial vehicles (UAVs) and intelligent reflective surface (IRS) are utilized to support terahertz (THz) communications. To this end, the joint optimization of UAV's trajectory, the phase shift of IRS, the allocation of THz sub-bands, and the power control is investigated to maximize the minimum average achievable rate of all the users. An iteration algorithm based on successive Convex Approximation with the Rate constraint penalty (CAR) is developed to obtain UAV's trajectory, and the IRS phase shift is formulated as a closed-form expression with introduced pricing factors. Simulation results show that the proposed scheme significantly enhances the rate performance of the whole system.

Enabling Panoramic Full-Angle Reflection via Aerial Intelligent Reflecting Surface

Abstract: This paper proposes a new three dimensional (3D) networking architecture enabled by aerial intelligent reflecting surface (AIRS) to achieve panoramic signal reflection from the sky. Compared to the conventional terrestrial IRS, AIRS not only enjoys higher deployment flexibility, but also is able to achieve 360$^\circ$ panoramic full-angle reflection and requires fewer reflections in general due to its higher likelihood of having line of sight (LoS) links with the ground nodes. We focus on the problem to maximize the worst-case signal-to-noise ratio (SNR) in a given coverage area by jointly optimizing the transmit beamforming, AIRS placement and phase shifts. The formulated problem is non-convex and the optimization variables are coupled with each other in an intricate manner. To tackle this problem, we first consider the special case of single-location SNR maximization to gain useful insights, for which the optimal solution is obtained in closed-form. Then for the general case of area coverage, an efficient suboptimal solution is proposed by exploiting the similarity between phase shifts optimization for IRS and analog beamforming for the conventional phase array. Numerical results show that the proposed design can achieve significant performance gain than heuristic AIRS deployment schemes.

Intelligent Reflecting Surface Aided Secure UAV Communications

Abstract: In this letter, we study the secure communication problem in the unmanned aerial vehicle (UAV) enabled networks aided by an intelligent reflecting surface (IRS) from the physical-layer security perspective. Specifically, the IRS is deployed to assist the wireless transmission from the UAV to the ground user in the presence of an eavesdropper. The objective of this work is to maximize the secrecy rate by jointly optimizing the phase shifts at the IRS as well as the transmit power and location of the UAV. However, the formulated problem is difficult to solve directly due to the non-linear and non-convex objective function and constraints. By invoking fractional programming and successive convex approximation techniques, the original problem is decomposed into three subproblems, which are then transformed into convex ones. Next, a low-complexity alternating algorithm is proposed to solve the challenging non-convex problem effectively, where the closed-form expressions for transmit power and phase shifts are obtained at each iteration. Simulations results demonstrate that the designed algorithm for IRS-aided UAV communications can achieve higher secrecy rate than benchmarks.

Robust Secure UAV Communications with the Aid of Reconfigurable Intelligent Surfaces

Abstract: This paper investigates a novel unmanned aerial vehicles (UAVs) secure communication system with the assistance of reconfigurable intelligent surfaces (RISs), where an UAV and a ground user communicate with each other, while an eavesdropper tends to wiretap their information. Due to the limited capacity of UAVs, a RIS is applied to further improve the quality of the secure communications. The time division multiple access (TDMA) protocol is applied for the communications between the UAV and the ground user, namely, the downlink (DL) and the uplink (UL) communications. In particular, the channel state information (CSI) of the eavesdropping channels is assumed to be imperfect. We aim to maximize the average worst-case secrecy rate by the robust joint design of the UAV's trajectory, RIS's passive beamforming, and transmit power of the legitimate transmitters. It is challenging to solve the Joint UL/DL optimization problem due to its non-convexity. To this end, we develop an efficient algorithm based on the alternating optimization (AO) technique. Specifically, the formulated problem is divided into three sub-problems, and the successive convex approximation (SCA), S-Procedure, and semidefinite relaxation (SDR) are applied to tackle these non-convex sub-problems. Numerical results demonstrate that the proposed algorithm can considerably improve the average secrecy rate compared with the benchmark algorithms, and also confirm the robustness of the proposed algorithm.

Reflections in the Sky: Joint Trajectory and Passive Beamforming Design for Secure UAV Networks with Reconfigurable Intelligent Surface

Abstract: This paper investigates the problem of secure energy efficiency maximization for a reconfigurable intelligent surface (RIS) assisted uplink wireless communication system, where an unmanned aerial vehicle (UAV) equipped with an RIS works as a mobile relay between the base station (BS) and a group of users. We focus on maximizing the secure energy efficiency of the system via jointly optimizing the UAV's trajectory, the RIS's phase shift, users' association and transmit power. To tackle this problem, we divide the original problem into three sub-problems, and propose an efficient iterative algorithm. In particular, the successive convex approximation method (SCA) is applied to solve the nonconvex UAV trajectory, the RIS's phase shift, and transmit power optimization sub-problems. We further provide two schemes to simplify the solution of phase and trajectory sub-problem. Simulation results demonstrate that the proposed algorithm converges fast, and the proposed design can enhance the secure energy efficiency by up to 38\% gains, as compared to the traditional schemes without any RIS.

Joint Beamforming and Location Optimization for Secure Data Collection in Wireless Sensor Networks with UAV-Carried Intelligent Reflecting Surface

Abstract: This paper considers unmanned aerial vehicle (UAV)-carried intelligent reflecting surface (IRS) for secure data collection in wireless sensor networks. An eavesdropper (Eve) lurks within the vicinity of the main receiver (Bob) while several randomly placed sensor nodes beamform collaboratively to the UAV-carried IRS that reflects the signal to the main receiver (Bob). The design objective is to maximise the achievable secrecy rate in the noisy communication channel by jointly optimizing the collaborative beamforming weights of the sensor nodes, the trajectory of the UAV and the reflection coefficients of the IRS elements. By designing the IRS reflection coefficients with and without the knowledge of the eavesdropper's channel, we develop a non-iterative sub-optimal solution for the secrecy rate maximization problem. It has been shown analytically that the UAV flight time and the randomness in the distribution of the sensor nodes, obtained by varying the sensor distribution area, can greatly affect secrecy performance. In addition, the maximum allowable number of IRS elements as well as a bound on the attainable average secrecy rate of the IRS aided noisy communication channel have also been derived. Extensive simulation results demonstrate the superior performance of the proposed algorithms compared to the existing schemes.

Summary of existing literature adopting alternating algorithm

Summary: The alternating optimization is the most general method to get a favorable solutions for the jointly coupled optimization variables in different system settings. Table. {{table:02f945e7-cdf8-4384-b0d6-80881bee44f7}}  summarizes all the literature adopting the alternating optimization method. As reviewed above, although the original problems can be divided into several subproblems, it is necessary to choose an efficient method for solving each subproblem whose objectives and constraints have certain properties.


A model that learns visual concepts, words, and semantic parsing of sentences without explicit supervision on any of them. Model learns by simply looking at images and reading paired questions and answers. The specific MIT model builds an object-based scene representation and translates sentences into executable, symbolic programs. Uses CLEVR dataset.

A neuro-symbolic model for the CLEVRER dataset. CLEVRER includes 300,000 questions and answers related to the action depicted in the videos. The questions fall into four different categories: Descriptive: e.g. “What is the material of the last object to collide with the cyan cylinder? Explanatory: e.g. “What is responsible for the collision between rubber and metal cylinders?” Predictive: e.g. “What will happen next? Counterfactual: e.g. “What will happen without the cyan cylinder?”

A neuro-symbolic architecture for both inductive learning and logic reasoning. Exploits the power of both neural networks as function approximators, and logic programming as a symbolic processor for objects with properties, relations, logic connectives, and quantifiers. After being trained on small-scale tasks (such as sorting short arrays), NLM models can recover lifted rules, and generalize to large-scale tasks (such as sorting longer arrays than they were trained on)

Image Parser (Object detection/masking)

Question Parser (Natural Language Processing)

Symbolic Executor

Video Frame Parser (Object detection/masking)

Question Parser (Natural Language Processing)

Dynamics Predictor (Learned physics)

Symbolic Executor

The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervision

Abstract: We propose the Neuro-Symbolic Concept Learner (NS-CL), a model that learns visual concepts, words, and semantic parsing of sentences without explicit supervision on any of them; instead, our model learns by simply looking at images and reading paired questions and answers. Our model builds an object-based scene representation and translates sentences into executable, symbolic programs. To bridge the learning of two modules, we use a neuro-symbolic reasoning module that executes these programs on the latent scene representation. Analogical to human concept learning, the perception module learns visual concepts based on the language description of the object being referred to. Meanwhile, the learned visual concepts facilitate learning new words and parsing new sentences. We use curriculum learning to guide the searching over the large compositional space of images and language. Extensive experiments demonstrate the accuracy and efficiency of our model on learning visual concepts, word representations, and semantic parsing of sentences. Further, our method allows easy generalization to new object attributes, compositions, language concepts, scenes and questions, and even new program domains. It also empowers applications including visual question answering and bidirectional image-text retrieval.

CLEVRER: CoLlision Events for Video REpresentation and Reasoning

Abstract: The ability to reason about temporal and causal events from videos lies at the core of human intelligence. Most video reasoning benchmarks, however, focus on pattern recognition from complex visual and language input, instead of on causal structure. We study the complementary problem, exploring the temporal and causal structures behind videos of objects with simple visual appearance. To this end, we introduce the CoLlision Events for Video REpresentation and Reasoning (CLEVRER), a diagnostic video dataset for systematic evaluation of computational models on a wide range of reasoning tasks. Motivated by the theory of human casual judgment, CLEVRER includes four types of questions: descriptive (e.g., "what color"), explanatory ("what is responsible for"), predictive ("what will happen next"), and counterfactual ("what if"). We evaluate various state-of-the-art models for visual reasoning on our benchmark. While these models thrive on the perception-based task (descriptive), they perform poorly on the causal tasks (explanatory, predictive and counterfactual), suggesting that a principled approach for causal reasoning should incorporate the capability of both perceiving complex visual and language inputs, and understanding the underlying dynamics and causal relations. We also study an oracle model that explicitly combines these components via symbolic representations.

Neural Logic Machines

Abstract: We propose the Neural Logic Machine (NLM), a neural-symbolic architecture for both inductive learning and logic reasoning. NLMs exploit the power of both neural networks---as function approximators, and logic programming---as a symbolic processor for objects with properties, relations, logic connectives, and quantifiers. After being trained on small-scale tasks (such as sorting short arrays), NLMs can recover lifted rules, and generalize to large-scale tasks (such as sorting longer arrays). In our experiments, NLMs achieve perfect generalization in a number of tasks, from relational reasoning tasks on the family tree and general graphs, to decision making tasks including sorting arrays, finding shortest paths, and playing the blocks world. Most of these tasks are hard to accomplish for neural networks or inductive logic programming alone.

Overview of the three neuro-symbolic models discussed in this paper

In this paper, we analyze the inference performance characteristics of three separate neuro-symbolic models. Two of the models, the Neuro-Symbolic Concept Learner (NSCL) and Neuro-Symbolic Dynamic Reasoning (NS-DR), are composed of multiple independent "submodels", which extract problem features before passing them as input to a final symbolic submodel. The third, Neural Logic Machines (NLM), is an end-to-end model without independent submodels. The NLM model uses an object’s relations, properties, quantifiers, and logic connectives in order to accomplish the task of generalization. We provide an overview of all three models in Table {{table:a2044e28-cf78-4747-93d0-711291b653cd}} .

As described in Table {{table:a2044e28-cf78-4747-93d0-711291b653cd}} , the NSCL is internally composed of three submodels. We describe each of these submodels in more detail here.

The Neuro-Symbolic Dynamic Reasoning (NS-DR) model is a neuro-symbolic model for the CLEVRER video reasoning dataset. CLEVRER builds on CLEVR by replacing images with videos. Objects in CLEVRER may enter or exit the scene throughout the course of a (5 second, 25 frame) video, potentially colliding and rebounding with other objects. Using videos instead of images adds a new domain to the questions in CLEVRER: that of causality. As described in Table {{table:a2044e28-cf78-4747-93d0-711291b653cd}} , causal questions may range from the descriptive ("Which objects collided?") to more complex causal relations ("What caused the objects to collide? Would the objects still have collided if ..."). Existing neural models were shown to perform poorly on all but the simple descriptive questions in CLEVRER {{cite:3a6a94d}}.


Towards a Better Metric for Evaluating Question Generation Systems

Abstract: There has always been criticism for using n-gram based similarity metrics, such as BLEU, NIST, etc, for evaluating the performance of NLG systems. However, these metrics continue to remain popular and are recently being used for evaluating the performance of systems which automatically generate questions from documents, knowledge graphs, images, etc. Given the rising interest in such automatic question generation (AQG) systems, it is important to objectively examine whether these metrics are suitable for this task. In particular, it is important to verify whether such metrics used for evaluating AQG systems focus on answerability of the generated question by preferring questions which contain all relevant information such as question type (Wh-types), entities, relations, etc. In this work, we show that current automatic evaluation metrics based on n-gram similarity do not always correlate well with human judgments about answerability of a question. To alleviate this problem and as a first step towards better evaluation metrics for AQG, we introduce a scoring function to capture answerability and show that when this scoring function is integrated with existing metrics, they correlate significantly better with human judgments. The scripts and data developed as a part of this work are made publicly available.

SummEval: Re-evaluating Summarization Evaluation

Abstract: The scarcity of comprehensive up-to-date studies on evaluation metrics for text summarization and the lack of consensus regarding evaluation protocols continue to inhibit progress. We address the existing shortcomings of summarization evaluation methods along five dimensions: 1) we re-evaluate 14 automatic evaluation metrics in a comprehensive and consistent fashion using neural summarization model outputs along with expert and crowd-sourced human annotations, 2) we consistently benchmark 23 recent summarization models using the aforementioned automatic evaluation metrics, 3) we assemble the largest collection of summaries generated by models trained on the CNN/DailyMail news dataset and share it in a unified format, 4) we implement and share a toolkit that provides an extensible and unified API for evaluating summarization models across a broad range of automatic metrics, 5) we assemble and share the largest and most diverse, in terms of model types, collection of human judgments of model-generated summaries on the CNN/Daily Mail dataset annotated by both expert judges and crowd-source workers. We hope that this work will help promote a more complete evaluation protocol for text summarization as well as advance research in developing evaluation metrics that better correlate with human judgments.

What makes a good conversation? How controllable attributes affect human judgments

Abstract: A good conversation requires balance -- between simplicity and detail; staying on topic and changing it; asking questions and answering them. Although dialogue agents are commonly evaluated via human judgments of overall quality, the relationship between quality and these individual factors is less well-studied. In this work, we examine two controllable neural text generation methods, conditional training and weighted decoding, in order to control four important attributes for chitchat dialogue: repetition, specificity, response-relatedness and question-asking. We conduct a large-scale human evaluation to measure the effect of these control parameters on multi-turn interactive conversations on the PersonaChat task. We provide a detailed analysis of their relationship to high-level aspects of conversation, and show that by controlling combinations of these variables our models obtain clear improvements in human quality judgments.

From Images to Sentences through Scene Description Graphs using Commonsense Reasoning and Knowledge

Abstract: In this paper we propose the construction of linguistic descriptions of images. This is achieved through the extraction of scene description graphs (SDGs) from visual scenes using an automatically constructed knowledge base. SDGs are constructed using both vision and reasoning. Specifically, commonsense reasoning is applied on (a) detections obtained from existing perception methods on given images, (b) a "commonsense" knowledge base constructed using natural language processing of image annotations and (c) lexical ontological knowledge from resources such as WordNet. Amazon Mechanical Turk(AMT)-based evaluations on Flickr8k, Flickr30k and MS-COCO datasets show that in most cases, sentences auto-constructed from SDGs obtained by our method give a more relevant and thorough description of an image than a recent state-of-the-art image caption based approach. Our Image-Sentence Alignment Evaluation results are also comparable to that of the recent state-of-the art approaches.

Datasets with human scores on many criteria.

We use existing publicly available datasets containing human judgement scores on multiple criteria for each of the 6 tasks described earlier. For example, {{cite:c286af9}} contains 3025 samples of outputs generated by data-to-text generation systems that participated in the WebNLG 2020 challenge. For each of these samples, the organisers asked humans to rate the output based on 5 criteria, viz., fluency, data coverage, relevance, correctness and text structure. We use these scores to compute the correlations between the scores of all the {{formula:1f9dbeb4-f40b-4f53-bd3d-53574cebfbd0}}  pairs of criteria. We repeat this for the other tasks using the datasets described in Table  {{table:e505b63b-a7a4-41b8-a6e1-08428a8a85dd}}For AS, we could not find a dataset containing human judgements for the set of criteria in {{cite:eda5418}}. Hence, we use the dataset provided by {{cite:c713a40}}.
Note that all of the datasets mentioned in Table {{table:e505b63b-a7a4-41b8-a6e1-08428a8a85dd}}  were collected using well established methods to ensure that the annotations were of high quality. Some of these datasets do not explicitly report the Inter Annotator Agreement (IAA) scores whereas others {{cite:c713a40}}, {{cite:c286af9}}, {{cite:245d6d6}} report a good IAA score ranging from 0.63-0.71. Using these annotations, we compute the pairwise Kendall tau correlations between all criteria for each task as seen in Figure {{figure:31cecf9a-f886-4b1a-a09e-97c7b92461a2}} . (Refer to appendix  for pearson correlations of the criteria.)

Datasets. For the coarse-grained evaluation, we use the datasets containing human judgements as described earlier in Table {{table:e505b63b-a7a4-41b8-a6e1-08428a8a85dd}}  in Section . For the fine-grained evaluation, we use datasets containing multiple ground truth references which can then be perturbed using our templates. For MT, we use the expanded version of newstest2017 Chinese to English dataset {{cite:e367e22}} which contains two references for each sentence. For QG, we use the SQuAD dataset {{cite:5dac1fc}} which contains multiple questions for each passage. For AS, we use the curated personal narrative corpus {{cite:6cf65d7}}. For DG, we use DailyDialog++ {{cite:7d1b73a}} which contains two-speaker conversations on generic topics. For IC, we use the COCO component of the Composite dataset containing 5 reference captions for each image {{cite:aacbeef}}. Lastly, for D2T, we use the Triples-to-Text data of the WebNLG 2020 challenge dataset {{cite:c286af9}}.

{{cite:aacbeef}}