Generates minority outliers {{cite:7a80bff}}

Detection and Characterization of Physical Activity and Psychological Stress from Wristband Data

Abstract: Wearable devices continuously measure multiple physiological variables to inform users of health and behavior indicators. The computed health indicators must rely on informative signals obtained by processing the raw physiological variables with powerful noise- and artifacts-filtering algorithms. In this study, we aimed to elucidate the effects of signal processing techniques on the accuracy of detecting and discriminating physical activity (PA) and acute psychological stress (APS) using physiological measurements (blood volume pulse, heart rate, skin temperature, galvanic skin response, and accelerometer) collected from a wristband. Data from 207 experiments involving 24 subjects were used to develop signal processing, feature extraction, and machine learning (ML) algorithms that can detect and discriminate PA and APS when they occur individually or concurrently, classify different types of PA and APS, and estimate energy expenditure (EE). Training data were used to generate feature variables from the physiological variables and develop ML models (naïve Bayes, decision tree, k-nearest neighbor, linear discriminant, ensemble learning, and support vector machine). Results from an independent labeled testing data set demonstrate that PA was detected and classified with an accuracy of 99.3%, and APS was detected and classified with an accuracy of 92.7%, whereas the simultaneous occurrences of both PA and APS were detected and classified with an accuracy of 89.9% (relative to actual class labels), and EE was estimated with a low mean absolute error of 0.02 metabolic equivalent of task (MET).The data filtering and adaptive noise cancellation techniques used to mitigate the effects of noise and artifacts on the classification results increased the detection and discrimination accuracy by 0.7% and 3.0% for PA and APS, respectively, and by 18% for EE estimation. The results demonstrate the physiological measurements from wristband devices are susceptible to noise and artifacts, and elucidate the effects of signal processing and feature extraction on the accuracy of detection, classification, and estimation of PA and APS.

Continuous Stress Detection Using Wearable Sensors in Real Life: Algorithmic Programming Contest Case Study

Abstract: The negative effects of mental stress on human health has been known for decades. High-level stress must be detected at early stages to prevent these negative effects. After the emergence of wearable devices that could be part of our lives, researchers have started detecting extreme stress of individuals with them during daily routines. Initial experiments were performed in laboratory environments and recently a number of works took a step outside the laboratory environment to the real-life. We developed an automatic stress detection system using physiological signals obtained from unobtrusive smart wearable devices which can be carried during the daily life routines of individuals. This system has modality-specific artifact removal and feature extraction methods for real-life conditions. We further tested our system in a real-life setting with collected physiological data from 21 participants of an algorithmic programming contest for nine days. This event had lectures, contests as well as free time. By using heart activity, skin conductance and accelerometer signals, we successfully discriminated contest stress, relatively higher cognitive load (lecture) and relaxed time activities by using different machine learning methods.

The Effect of Person-Specific Biometrics in Improving Generic Stress Predictive Models

Abstract: —Because stress is subjective and is expressed differently from one person to another, generic stress prediction models (i.e., models that predict the stress of any person) perform crudely. Only person-speciﬁc ones (i.e., models that predict the stress of a preordained person) yield reliable predictions, but they are not adaptable and costly to deploy in real-world environments. For illustration, in an ofﬁce environment, a stress monitoring system that uses person-speciﬁc models would require collecting new data and training a new model for every employee. Moreover,once deployed,the models would deteriorate and need expensive periodic upgrades because stress is dynamic and depends on unforeseeable factors. We propose a simple, yet practical and cost-effective calibration technique that derives an accurate and personalized stress prediction model from physiological samples collected from a large population. We validate our approach on two stress datasets. The results show that our technique performs much better than a generic model. For instance, a generic model achieved only a 42.5% ± 19.9% accuracy. However, with only 100 calibration samples, we raised its accuracy to 95.2% ± 0.5% . We also propose a blueprint for a stress monitoring system based on our strategy, and we debate its merits and limitation. Finally, we made public our source code and the relevant datasets to allow other researchers to replicate our ﬁndings.

How Laboratory Experiments Can Be Exploited for Monitoring Stress in the Wild: A Bridge Between Laboratory and Daily Life

Abstract: Chronic stress leads to poor well-being, and it has effects on life quality and health. Society may have significant benefits from an automatic daily life stress detection system using unobtrusive wearable devices using physiological signals. However, the performance of these systems is not sufficiently accurate when they are used in unrestricted daily life compared to the systems tested in controlled real-life and laboratory conditions. To test our stress level detection system that preprocesses noisy physiological signals, extracts features, and applies machine learning classification techniques, we used a laboratory experiment and ecological momentary assessment based data collection with smartwatches in daily life. We investigated the effect of different labeling techniques and different training and test environments. In the laboratory environments, we had more controlled situations, and we could validate the perceived stress from self-reports. When machine learning models were trained in the laboratory instead of training them with the data coming from daily life, the accuracy of the system when tested in daily life improved significantly. The subjectivity effect coming from the self-reports in daily life could be eliminated. Our system obtained higher stress level detection accuracy results compared to most of the previous daily life studies.

A Conditional GAN for Generating Time Series Data for Stress Detection in Wearable Physiological Sensor Data

Abstract: Human-centered applications using wearable sensors in combination with machine learning have received a great deal of attention in the last couple of years. At the same time, wearable sensors have also evolved and are now able to accurately measure physiological signals and are, therefore, suitable for detecting body reactions to stress. The field of machine learning, or more precisely, deep learning, has been able to produce outstanding results. However, in order to produce these good results, large amounts of labeled data are needed, which, in the context of physiological data related to stress detection, are a great challenge to collect, as they usually require costly experiments or expert knowledge. This usually results in an imbalanced and small dataset, which makes it difficult to train a deep learning algorithm. In recent studies, this problem is tackled with data augmentation via a Generative Adversarial Network (GAN). Conditional GANs (cGAN) are particularly suitable for this as they provide the opportunity to feed auxiliary information such as a class label into the training process to generate labeled data. However, it has been found that during the training process of GANs, different problems usually occur, such as mode collapse or vanishing gradients. To tackle the problems mentioned above, we propose a Long Short-Term Memory (LSTM) network, combined with a Fully Convolutional Network (FCN) cGAN architecture, with an additional diversity term to generate synthetic physiological data, which are used to augment the training dataset to improve the performance of a binary classifier for stress detection. We evaluated the methodology on our collected physiological measurement dataset, and we were able to show that using the method, the performance of an LSTM and an FCN classifier could be improved. Further, we showed that the generated data could not be distinguished from the real data any longer.

Class balancing methods employed in reviewed studies.

As noted in Table {{table:9c03cc5a-fd08-47e8-9dd3-45391154a56a}} , neither up-sampling nor down-sampling techniques showed a substantial difference or improvement in predictive power, and this may be due to the lack of a proven strategy employed when selecting which observations to discard, potentially causing information loss {{cite:3033e2a}} of important biomarker data during the sampling process. Class balancing techniques all have varied benefits and risks, as noted in Table {{table:9c03cc5a-fd08-47e8-9dd3-45391154a56a}} , and to this extent, a number of methods have been proposed to improve class-balancing re-sampling techniques. Deng et al. {{cite:96a5d8f}} proposed a unified approach for multivariate time series classification when data is imbalanced, while Lee et al. {{cite:58e1ea3}} used a semi-supervised technique known as Active Learning to mitigate the effect of imbalanced class labels. Jiang et al. {{cite:8ad0bc9}} proposed a new oversampling method based on the classification contribution degree to deal with a number of shortcomings when using SMOTE (Synthetic Minority Oversampling Technique) {{cite:4a5f731}}, such as oversampling from noisy points. A notable drawback of reliance on class balancing when dealing with highly imbalanced datasets such as the stress biomarker datasets included in this study, where the stressed period is generally the minority class, is reproducibility and generalizability on new, unseen data that may contain significant outliers and a different class distribution, depending on the study setting and protocol used during biomarker recording. Further research is required to identify robust techniques for dealing with these class imbalances in physiological biomarker datasets.


(1) Targets a large scale IoT devices for access control processes to information, devices and services, (2) Provides fine-grained, scalable and lightweight control mechanism, (3) Local private blockchain network and smart contracts for registration, revocation of the access authorization and propagation, (3) A capability delegation mechanism to help in accessing permission, (4) Token management
strategy

(1) Improves the method of communication for IoT gateways by using trust less and decentralized nature of blockchain providing peer-to-peer network, (2) Distributed storage service

(1) Transaction throughput management, (2) Storage management details, (3) Time bound analysis

(1) Scalability and reliability issues, (2) Transaction management, (3) Transaction throughput analysis

BlendCAC: A BLockchain-ENabled Decentralized Capability-based Access Control for IoTs

Abstract: The prevalence of Internet of Things (IoTs) allows heterogeneous embedded smart devices to collaboratively provide smart services with or without human intervention. While leveraging the large scale IoT based applications like Smart Gird or Smart Cities, IoTs also incur more concerns on privacy and security. Among the top security challenges that IoTs face, access authorization is critical in resource sharing and information protection. One of the weaknesses in today's access control (AC) is the centralized authorization server, which can be the performance bottleneck or the single point of failure. In this paper, BlendCAC, a blockchain enabled decentralized capability based AC is proposed for the security of IoTs. The BlendCAC aims at an effective access control processes to devices, services and information in large scale IoT systems. Based on the blockchain network, a capability delegation mechanism is suggested for access permission propagation. A robust identity based capability token management strategy is proposed, which takes advantage of smart contract for registering, propagation and revocation of the access authorization. In the proposed BlendCAC scheme, IoT devices are their own master to control their resources instead of being supervised by a centralized authority. Implemented and tested on a Raspberry Pi device and on a local private blockchain network, our experimental results demonstrate the feasibility of the proposed BlendCAC approach to offer a decentralized, scalable, lightweight and fine grained AC solution to IoT systems.

Designing a Blockchain-Based IoT With Ethereum, Swarm, and LoRa: The Software Solution to Create High Availability With Minimal Security Risks

Abstract: Today, the number of Internet of Things (IoT) devices in all aspects of life is increasing exponentially. Our cities are getting smarter and informing us about our surroundings in a contextual manner. However, we face significant challenges in deploying, managing, and collecting data from these devices. In addition, we must address the problem of storing and mining that data for higher-quality IoT services. Blockchain technology, even in today's nascent form, has the potential to be the foundation for a common, distributed, trustless, and autonomous infrastructure system. This article describes a standardized IoT infrastructure where data are stored on a distributed storage service that is fault-tolerant and resistant to distributed denial of service (DDOS) attacks and data access is managed by a decentralized, trustless blockchain. The illustrated system used LoRa as the emerging network technology, Swarm as the distributed data storage platform, and Ethereum as the blockchain platform. Such a data back end will ensure high availability with minimal security risks while replacing traditional back-end systems with a single "smart contract."

Integration of IoT with Blockchain Concentrating on Decentralization (contd.)

In this section, different uses of blockchain in IoT are presented in details. Tables {{table:5a8b4e9b-b44c-4271-b565-a533d64e7c01}} , {{table:339cbf03-c9ca-4c02-af96-f903bc13998a}} , {{table:27e6d4f0-82e7-4a37-a68a-31ea4126f65d}} , {{table:f0ae990b-b8c4-4741-abbb-5c97b8246b77}} , {{table:5ef19492-17da-4c51-a00d-99f17cbad404}} , {{table:608220f0-c168-42fa-aaef-0cc207dbaf07}} , {{table:bff873e0-9f5a-4529-a9de-c7055dad88a9}} , {{table:01543948-7e3d-4dc2-8fab-a8c09687036d}} , and {{table:691d43e1-a3b9-4027-aa3c-3aecffb503e8}}  highlight the strengths and weaknesses of the related research works which consider the integration of the IoT with blockchain in different perspectives such as decentralization, safety, security, privacy, authentication, tamper-proof, robustness, etc.


The Remarkable Role of Similarity in Redundancy-based Program Repair

Abstract: Recently, there have been original attempts to use the concept of "code similarity" in program repair, suggesting that similarity analysis has an important role in the repair process. However, there is no dedicated work to characterize and quantify the role of similarity in redundancy-based program repair, where the patch is composed from source code taken from somewhere else. This is where our paper makes a major contribution: we perform a deep and systematic analysis of the role of code similarity during the exploration of the repair search space. We define and set up a large-scale experiment based on four code similarity metrics that capture different similarities: character, token, semantic and structure similarity. Overall, we have computed 56 million similarity score over 15 million source code components. We show that with similarity analysis, at least 90% of search space can be ignored to find the correct patch. Code similarity is capable of ranking the correct repair ingredient first in 4 - 33 % of the considered cases.

Precise Condition Synthesis for Program Repair

Abstract: Due to the difficulty of repairing defect, many research efforts have been devoted into automatic defect repair. Given a buggy program that fails some test cases, a typical automatic repair technique tries to modify the program to make all tests pass. However, since the test suites in real world projects are usually insufficient, aiming at passing the test suites often leads to incorrect patches. In this paper we aim to produce precise patches, that is, any patch we produce has a relatively high probability to be correct. More concretely, we focus on condition synthesis, which was shown to be able to repair more than half of the defects in existing approaches. Our key insight is threefold. First, it is important to know what variables in a local context should be used in an "if" condition, and we propose a sorting method based on the dependency relations between variables. Second, we observe that the API document can be used to guide the repair process, and propose document analysis technique to further filter the variables. Third, it is important to know what predicates should be performed on the set of variables, and we propose to mine a set of frequently used predicates in similar contexts from existing projects. We develop a novel program repair system, ACS, that could generate precise conditions at faulty locations. Furthermore, given the generated conditions are very precise, we can perform a repair operation that is previously deemed to be too overfitting: directly returning the test oracle to repair the defect. Using our approach, we successfully repaired 17 defects on four projects of Defects4J, which is the largest number of fully automatically repaired defects reported on the dataset so far. More importantly, the precision of our approach in the evaluation is 73.9%, which is significantly higher than previous approaches, which are usually less than 40%.

Sorting and Transforming Program Repair Ingredients via Deep Learning Code Similarities

Abstract: In the field of automated program repair, the redundancy assumption claims large programs contain the seeds of their own repair. However, most redundancy-based program repair techniques do not reason about the repair ingredients---the code that is reused to craft a patch. We aim to reason about the repair ingredients by using code similarities to prioritize and transform statements in a codebase for patch generation. Our approach, DeepRepair, relies on deep learning to reason about code similarities. Code fragments at well-defined levels of granularity in a codebase can be sorted according to their similarity to suspicious elements (i.e., code elements that contain suspicious statements) and statements can be transformed by mapping out-of-scope identifiers to similar identifiers in scope. We examined these new search strategies for patch generation with respect to effectiveness from the viewpoint of a software maintainer. Our comparative experiments were executed on six open-source Java projects including 374 buggy program revisions and consisted of 19,949 trials spanning 2,616 days of computation time. DeepRepair's search strategy using code similarities generally found compilable ingredients faster than the baseline, jGenProg, but this improvement neither yielded test-adequate patches in fewer attempts (on average) nor found significantly more patches than the baseline. Although the patch counts were not statistically different, there were notable differences between the nature of DeepRepair patches and baseline patches. The results demonstrate that our learning-based approach finds patches that cannot be found by existing redundancy-based repair techniques.

FixMiner: Mining Relevant Fix Patterns for Automated Program Repair

Abstract: Patching is a common activity in software development. It is generally performed on a source code base to address bugs or add new functionalities. In this context, given the recurrence of bugs across projects, the associated similar patches can be leveraged to extract generic fix actions. While the literature includes various approaches leveraging similarity among patches to guide program repair, these approaches often do not yield fix patterns that are tractable and reusable as actionable input to APR systems. In this paper, we propose a systematic and automated approach to mining relevant and actionable fix patterns based on an iterative clustering strategy applied to atomic changes within patches. The goal of FixMiner is thus to infer separate and reusable fix patterns that can be leveraged in other patch generation systems. Our technique, FixMiner, leverages Rich Edit Script which is a specialized tree structure of the edit scripts that captures the AST-level context of the code changes. FixMiner uses different tree representations of Rich Edit Scripts for each round of clustering to identify similar changes. These are abstract syntax trees, edit actions trees, and code context trees. We have evaluated FixMiner on thousands of software patches collected from open source projects. Preliminary results show that we are able to mine accurate patterns, efficiently exploiting change information in Rich Edit Scripts. We further integrated the mined patterns to an automated program repair prototype, PARFixMiner, with which we are able to correctly fix 26 bugs of the Defects4J benchmark. Beyond this quantitative performance, we show that the mined fix patterns are sufficiently relevant to produce patches with a high probability of correctness: 81% of PARFixMiner's generated plausible patches are correct.

Harnessing Evolution for Multi-Hunk Program Repair

Abstract: Despite significant advances in automatic program repair (APR) techniques over the past decade, practical deployment remains an elusive goal. One of the important challenges in this regard is the general inability of current APR techniques to produce patches that require edits in multiple locations, i.e., multi-hunk patches. In this work, we present a novel APR technique that generalizes single-hunk repair techniques to include an important class of multi-hunk bugs, namely bugs that may require applying a substantially similar patch at a number of locations. We term such sets of repair locations as evolutionary siblings - similar looking code, instantiated in similar contexts, that are expected to undergo similar changes. At the heart of our proposed method is an analysis to accurately identify a set of evolutionary siblings, for a given bug. This analysis leverages three distinct sources of information, namely the test-suite spectrum, a novel code similarity analysis, and the revision history of the project. The discovered siblings are then simultaneously repaired in a similar fashion. We instantiate this technique in a tool called HERCULES and demonstrate that it is able to correctly fix 46 bugs in the Defects4J dataset, the highest of any individual APR technique to date. This includes 15 multi-hunk bugs and overall 11 bugs which have not been fixed by any other technique so far.

Statistics on the collected plausible patches.

We focus our experiments on the Defects4J {{cite:ca0baa5}} benchmark since it is widely used in the literature, and we can readily collect plausible patches generated by APR tools on the programs included in the benchmark. Table {{table:7ce20f40-d8a5-4bf7-90fc-9d3b024c14d2}}  provides the statistics on the collected patches. Besides the 205 developer (correct) patches provided in the benchmark, we also leveraged the reproduced dataset from the study of 16 APR systems by Liu et al. {{cite:c47dad0}}, which we augmented with a dataset provided by Ye et al. {{cite:591c86b}}. Finally, we also scan the artifacts released in the literature towards identifying plausible patches generated by recent APR tools. Overall, we share with the community the largest dataset, so-far, of patches for Defects4J bugs, which includes hundreds of overfitting (i.e., incorrect) patches.


ALBERT: A Lite BERT for Self-supervised Learning of Language Representations

Abstract: Increasing model size when pretraining natural language representations often results in improved performance on downstream tasks. However, at some point further model increases become harder due to GPU/TPU memory limitations and longer training times. To address these problems, we present two parameter-reduction techniques to lower memory consumption and increase the training speed of BERT. Comprehensive empirical evidence shows that our proposed methods lead to models that scale much better compared to the original BERT. We also use a self-supervised loss that focuses on modeling inter-sentence coherence, and show it consistently helps downstream tasks with multi-sentence inputs. As a result, our best model establishes new state-of-the-art results on the GLUE, RACE, and \squad benchmarks while having fewer parameters compared to BERT-large. The code and the pretrained models are available at this https URL.

BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension

Abstract: We present BART, a denoising autoencoder for pretraining sequence-to-sequence models. BART is trained by (1) corrupting text with an arbitrary noising function, and (2) learning a model to reconstruct the original text. It uses a standard Tranformer-based neural machine translation architecture which, despite its simplicity, can be seen as generalizing BERT (due to the bidirectional encoder), GPT (with the left-to-right decoder), and other recent pretraining schemes. We evaluate a number of noising approaches, finding the best performance by both randomly shuffling the order of sentences and using a novel in-filling scheme, where spans of text are replaced with a single mask token. BART is particularly effective when fine tuned for text generation but also works well for comprehension tasks. It matches the performance of RoBERTa on GLUE and SQuAD, and achieves new state-of-the-art results on a range of abstractive dialogue, question answering, and summarization tasks, with gains of up to 3.5 ROUGE. BART also provides a 1.1 BLEU increase over a back-translation system for machine translation, with only target language pretraining. We also replicate other pretraining schemes within the BART framework, to understand their effect on end-task performance.

GLM: General Language Model Pretraining with Autoregressive Blank Infilling

Abstract: There have been various types of pretraining architectures including autoencoding models (e.g., BERT), autoregressive models (e.g., GPT), and encoder-decoder models (e.g., T5). However, none of the pretraining frameworks performs the best for all tasks of three main categories including natural language understanding (NLU), unconditional generation, and conditional generation. We propose a General Language Model (GLM) based on autoregressive blank infilling to address this challenge. GLM improves blank filling pretraining by adding 2D positional encodings and allowing an arbitrary order to predict spans, which results in performance gains over BERT and T5 on NLU tasks. Meanwhile, GLM can be pretrained for different types of tasks by varying the number and lengths of blanks. On a wide range of tasks across NLU, conditional and unconditional generation, GLM outperforms BERT, T5, and GPT given the same model sizes and data, and achieves the best performance from a single pretrained model with 1.25× parameters of BERT Large , demonstrating its generalizability to different downstream tasks.

a survey on GPT-3

Abstract: This paper provides an introductory survey to GPT-3. We cover some of the historical development behind this technology, some of the key features of GPT-3, and discuss the machine learning model and the datasets used. We survey both academic and commercial efforts applying GPT-3 in diverse domains such as developing conversational AI chatbots, software development, creative work, domain knowledge, and business productivity. We discuss some of the challenges that GPT-3 faces such as the problems of training complexity, bias, and hallucination/incorrect answers. We also discuss the future research opportunities in this area.

GLaM: Efficient Scaling of Language Models with Mixture-of-Experts

Abstract: Scaling language models with more data, compute and parameters has driven significant progress in natural language processing. For example, thanks to scaling, GPT-3 was able to achieve strong results on in-context learning tasks. However, training these large dense models requires significant amounts of computing resources. In this paper, we propose and develop a family of language models named GLaM (Generalist Language Model), which uses a sparsely activated mixture-of-experts architecture to scale the model capacity while also incurring substantially less training cost compared to dense variants. The largest GLaM has 1.2 trillion parameters, which is approximately 7x larger than GPT-3. It consumes only 1/3 of the energy used to train GPT-3 and requires half of the computation flops for inference, while still achieving better overall zero-shot and one-shot performance across 29 NLP tasks.

Landmark language models and their capabilities.

Several language-models have been developed and analyzed over the past decades to enable natural language understanding (NLU) and natural language generation (NLG) tasks {{cite:f493c5a}}. All existing works can be broadly classified as encoder-only models that enable machine level understanding of human language, encoder-decoder models that enable NLU and NLG combined to process human language and generate desired outcomes such as summaries or paraphrased verbiage etc. and decoder-only models such as the LLMs. Some of the language models landmarks in each category are shown in Table {{table:4fb5f5d4-4e58-42d9-a2d8-d1f42d36399d}} .


FEVER: a Large-scale Dataset for Fact Extraction and VERification

Abstract: In this paper we introduce a new publicly available dataset for verification against textual sources, FEVER: Fact Extraction and VERification. It consists of 185,445 claims generated by altering sentences extracted from Wikipedia and subsequently verified without knowledge of the sentence they were derived from. The claims are classified as Supported, Refuted or NotEnoughInfo by annotators achieving 0.6841 in Fleiss kappa. For the first two classes, the annotators also recorded the sentence(s) forming the necessary evidence for their judgment. To characterize the challenge of the dataset presented, we develop a pipeline approach and compare it to suitably designed oracles. The best accuracy we achieve on labeling a claim accompanied by the correct evidence is 31.87%, while if we ignore the evidence we achieve 50.91%. Thus we believe that FEVER is a challenging testbed that will help stimulate progress on claim verification against textual sources.

Zero-Shot Relation Extraction via Reading Comprehension

Abstract: We show that relation extraction can be reduced to answering simple reading comprehension questions, by associating one or more natural-language questions with each relation slot. This reduction has several advantages: we can (1) learn relation-extraction models by extending recent neural reading-comprehension techniques, (2) build very large training sets for those models by combining relation-specific crowd-sourced questions with distant supervision, and even (3) do zero-shot learning by extracting new relation types that are only specified at test-time, for which we have no labeled training examples. Experiments on a Wikipedia slot-filling task demonstrate that the approach can generalize to new questions for known relation types with high accuracy, and that zero-shot generalization to unseen relation types is possible, at lower accuracy levels, setting the bar for future work on this task.

HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering

Abstract: Existing question answering (QA) datasets fail to train QA systems to perform complex reasoning and provide explanations for answers. We introduce HotpotQA, a new dataset with 113k Wikipedia-based question-answer pairs with four key features: (1) the questions require finding and reasoning over multiple supporting documents to answer; (2) the questions are diverse and not constrained to any pre-existing knowledge bases or knowledge schemas; (3) we provide sentence-level supporting facts required for reasoning, allowing QA systems to reason with strong supervision and explain the predictions; (4) we offer a new type of factoid comparison questions to test QA systems’ ability to extract relevant facts and perform necessary comparison. We show that HotpotQA is challenging for the latest QA systems, and the supporting facts enable models to improve performance and make explainable predictions.

TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension

Abstract: We present TriviaQA, a challenging reading comprehension dataset containing over 650K question-answer-evidence triples. TriviaQA includes 95K question-answer pairs authored by trivia enthusiasts and independently gathered evidence documents, six per question on average, that provide high quality distant supervision for answering the questions. We show that, in comparison to other recently introduced large-scale datasets, TriviaQA (1) has relatively complex, compositional questions, (2) has considerable syntactic and lexical variability between questions and corresponding answer-evidence sentences, and (3) requires more cross sentence reasoning to find answers. We also present two baseline algorithms: a feature-based classifier and a state-of-the-art neural network, that performs well on SQuAD reading comprehension. Neither approach comes close to human performance (23% and 40% vs. 80%), suggesting that TriviaQA is a challenging testbed that is worth significant future study.

ELI5: Long Form Question Answering

Abstract: We introduce the first large-scale corpus for long form question answering, a task requiring elaborate and in-depth answers to open-ended questions. The dataset comprises 270K threads from the Reddit forum “Explain Like I’m Five” (ELI5) where an online community provides answers to questions which are comprehensible by five year olds. Compared to existing datasets, ELI5 comprises diverse questions requiring multi-sentence answers. We provide a large set of web documents to help answer the question. Automatic and human evaluations show that an abstractive model trained with a multi-task objective outperforms conventional Seq2Seq, language modeling, as well as a strong extractive baseline.However, our best model is still far from human performance since raters prefer gold responses in over 86% of cases, leaving ample opportunity for future improvement.

Wizard of Wikipedia: Knowledge-Powered Conversational agents

Abstract: In open-domain dialogue intelligent agents should exhibit the use of knowledge, however there are few convincing demonstrations of this to date. The most popular sequence to sequence models typically "generate and hope" generic utterances that can be memorized in the weights of the model when mapping from input utterance(s) to output, rather than employing recalled knowledge as context. Use of knowledge has so far proved difficult, in part because of the lack of a supervised learning benchmark task which exhibits knowledgeable open dialogue with clear grounding. To that end we collect and release a large dataset with conversations directly grounded with knowledge retrieved from Wikipedia. We then design architectures capable of retrieving knowledge, reading and conditioning on it, and finally generating natural responses. Our best performing dialogue models are able to conduct knowledgeable discussions on open-domain topics as evaluated by automatic metrics and human evaluations, while our new benchmark allows for measuring further improvements in this important research direction.

Retrieval datasets statistics of the KILT benchmark. #P denotes the average number of relevant Wikipedia pages for each query in the train, dev and test set. `-' denotes that the task does not provide a ground-truth document in the training set.

To verify the effectiveness of our method, we conduct experiments on the KILT benchmark {{cite:1332d55}} with eleven datasets spanning five distinct knowledge-intensive language tasks.
In this work, we consider the retrieval task on the KILT benchmark, in which the model should provide a set of Wikipedia pages as evidences for final prediction with respect to the input query.
Table {{table:d3f0a887-ef9c-4a95-b208-cc8a361532f0}}  shows the overall statistics of the retrieval tasks in KILT.

In this work, we fine-tune CorpusBrain via three different strategies: (1) We fine-tune CorpusBrain on specific downstream KILT task, denoted as CorpusBrain{{formula:f242e1e2-e66b-4ae5-b7f2-b0a8c9d17ebe}} .
(2) We fine-tune CorpusBrain via multi-task training on all KILT retrieval data following {{cite:d2b79ac}}, {{cite:2ae7fe2}}, {{cite:621f1f9}}, denoted as CorpusBrain{{formula:8319ade6-8439-44d2-8761-cae66d56befd}} .
Note that not all dataset available in KILT have a training set as shown in Table {{table:d3f0a887-ef9c-4a95-b208-cc8a361532f0}} . To address all tasks, multi-task training has been a common approach for the KILT benchmark.
(3) We follow the fine-tuning setting used in GENRE {{cite:2ae7fe2}} to train CorpusBrain on BLINK {{cite:1beeda4}} and all KILT data simultaneously, denoted as CorpusBrain{{formula:e8c80c98-2415-4936-ba4f-d96065703886}} .


Optimal Transport for structured data with application on graphs

Abstract: This work considers the problem of computing distances between structured objects such as undirected graphs, seen as probability distributions in a specific metric space. We consider a new transportation distance (i.e. that minimizes a total cost of transporting probability masses) that unveils the geometric nature of the structured objects space. Unlike Wasserstein or Gromov-Wasserstein metrics that focus solely and respectively on features (by considering a metric in the feature space) or structure (by seeing structure as a metric space), our new distance exploits jointly both information, and is consequently called Fused Gromov-Wasserstein (FGW). After discussing its properties and computational aspects, we show results on a graph classification task, where our method outperforms both graph kernels and deep graph convolutional networks. Exploiting further on the metric properties of FGW, interesting geometric objects such as Frechet means or barycenters of graphs are illustrated and discussed in a clustering context.

Scalable Gromov-Wasserstein Learning for Graph Partitioning and Matching

Abstract: We propose a scalable Gromov-Wasserstein learning (S-GWL) method and establish a novel and theoretically-supported paradigm for large-scale graph analysis. The proposed method is based on the fact that Gromov-Wasserstein discrepancy is a pseudometric on graphs. Given two graphs, the optimal transport associated with their Gromov-Wasserstein discrepancy provides the correspondence between their nodes and achieves graph matching. When one of the graphs has isolated but self-connected nodes ($i.e.$, a disconnected graph), the optimal transport indicates the clustering structure of the other graph and achieves graph partitioning. Using this concept, we extend our method to multi-graph partitioning and matching by learning a Gromov-Wasserstein barycenter graph for multiple observed graphs; the barycenter graph plays the role of the disconnected graph, and since it is learned, so is the clustering. Our method combines a recursive $K$-partition mechanism with a regularized proximal gradient algorithm, whose time complexity is $\mathcal{O}(K(E+V)\log_K V)$ for graphs with $V$ nodes and $E$ edges. To our knowledge, our method is the first attempt to make Gromov-Wasserstein discrepancy applicable to large-scale graph analysis and unify graph partitioning and matching into the same framework. It outperforms state-of-the-art graph partitioning and matching methods, achieving a trade-off between accuracy and efficiency.

Linear-Time Gromov Wasserstein Distances using Low Rank Couplings and Costs

Abstract: The ability to align points across two related yet incomparable point clouds (e.g. living in different spaces) plays an important role in machine learning. The Gromov-Wasserstein (GW) framework provides an increasingly popular answer to such problems, by seeking a low-distortion, geometry-preserving assignment between these points. As a non-convex, quadratic generalization of optimal transport (OT), GW is NP-hard. While practitioners often resort to solving GW approximately as a nested sequence of entropy-regularized OT problems, the cubic complexity (in the number $n$ of samples) of that approach is a roadblock. We show in this work how a recent variant of the OT problem that restricts the set of admissible couplings to those having a low-rank factorization is remarkably well suited to the resolution of GW: when applied to GW, we show that this approach is not only able to compute a stationary point of the GW problem in time $O(n^2)$, but also uniquely positioned to benefit from the knowledge that the initial cost matrices are low-rank, to yield a linear time $O(n)$ GW approximation. Our approach yields similar results, yet orders of magnitude faster computation than the SoTA entropic GW approaches, on both simulated and real data.

Wasserstein-based Graph Alignment

Abstract: We propose a novel method for comparing non-aligned graphs of different sizes, based on the Wasserstein distance between graph signal distributions induced by the respective graph Laplacian matrices. Specifically, we cast a new formulation for the one-to-many graph alignment problem, which aims at matching a node in the smaller graph with one or more nodes in the larger graph. By integrating optimal transport in our graph comparison framework, we generate both a structurally-meaningful graph distance, and a signal transportation plan that models the structure of graph data. The resulting alignment problem is solved with stochastic gradient descent, where we use a novel Dykstra operator to ensure that the solution is a one-to-many (soft) assignment matrix. We demonstrate the performance of our novel framework on graph alignment and graph classification, and we show that our method leads to significant improvements with respect to the state-of-the-art algorithms for each of these tasks.

Baseline methods summary.

In this paper, we study the multi-network alignment problem from the view of hierarchical multi-marginal optimal transport (MOT). To handle the high computational complexity, we explore the hierarchical structure of graph data and decompose the original problem into smaller cluster-level and node-level alignment subproblems. To depict high-order node relationships, a position-aware cost is first generated based on the unified random walk with restart (RWR) embeddings, and the multi-marginal fused Gromov-Wasserstein (MFGW) distance is further leveraged to align multiple networks jointly. A fast algorithm is proposed with guaranteed convergence to a local optimum, reducing both time and space complexities by an exponential factor in terms of the number of networks compared with the straightforward solution. Extensive experiments validate the effectiveness and scalability of the proposed Hot on the multi-network alignment task.
Alaux, J.; Grave, E.; Cuturi, M.; and Joulin, A. 2019.
Unsupervised Hyper-alignment for Multilingual Word Embeddings.
In International Conference on Learning Representations.
Beier, F.; Beinert, R.; and Steidl, G. 2022.
Multi-Marginal Gromov-Wasserstein Transport and Barycenters.
arXiv preprint arXiv:2205.06725.
Cang, Z.; Nie, Q.; and Zhao, Y. 2022.
Supervised Optimal Transport.
SIAM Journal on Applied Mathematics, 82(5): 1851–1877.
Chen, C.; Tong, H.; Xie, L.; Ying, L.; and He, Q. 2016.
FASCINATE: fast cross-layer dependency inference on multi-layered networks.
In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, 765–774.
Chen, L.; Gan, Z.; Cheng, Y.; Li, L.; Carin, L.; and Liu, J. 2020.
Graph optimal transport for cross-domain alignment.
In International Conference on Machine Learning, 1542–1553. PMLR.
Chen, Z.; Yu, X.; Song, B.; Gao, J.; Hu, X.; and Yang, W.-S. 2017.
Community-based network alignment for large attributed network.
In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management, 587–596.
Chu, X.; Fan, X.; Yao, D.; Zhu, Z.; Huang, J.; and Bi, J. 2019.
Cross-network embedding for multi-network alignment.
In The world wide web conference, 273–284.
Cuturi, M. 2013.
Sinkhorn distances: Lightspeed computation of optimal transport.
Advances in neural information processing systems, 26.
Cuturi, M.; and Doucet, A. 2014.
Fast computation of Wasserstein barycenters.
In International conference on machine learning, 685–693. PMLR.
Dong, Y.; and Sawin, W. 2020.
Copt: Coordinated optimal transport on graphs.
Advances in Neural Information Processing Systems, 33: 19327–19338.
Du, B.; Liu, L.; and Tong, H. 2021.
Sylvester Tensor Equation for Multi-Way Association.
In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, 311–321.
Du, B.; and Tong, H. 2018.
Fasten: Fast sylvester equation solver for graph mining.
In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 1339–1347.
Du, X.; Yan, J.; and Zha, H. 2019.
Joint Link Prediction and Network Alignment via Cross-graph Embedding.
In IJCAI, 2251–2257.
Fan, J.; Haasler, I.; Karlsson, J.; and Chen, Y. 2022.
On the complexity of the optimal transport problem with graph-structured cost.
In International conference on artificial intelligence and statistics, 9147–9165. PMLR.
Ferradans, S.; Papadakis, N.; Peyré, G.; and Aujol, J.-F. 2014.
Regularized discrete optimal transport.
SIAM Journal on Imaging Sciences, 7(3): 1853–1882.
Gao, J.; Huang, X.; and Li, J. 2021.
Unsupervised graph alignment with wasserstein distance discriminator.
In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, 426–435.
Garg, V.; and Jaakkola, T. 2019.
Solving graph compression via optimal transport.
Advances in Neural Information Processing Systems, 32.
Heimann, M.; Shen, H.; Safavi, T.; and Koutra, D. 2018.
Regal: Representation learning-based graph alignment.
In Proceedings of the 27th ACM international conference on information and knowledge management, 117–126.
Kolouri, S.; Naderializadeh, N.; Rohde, G. K.; and Hoffmann, H. 2021.
Wasserstein Embedding for Graph Learning.
In International Conference on Learning Representations.
Koutra, D.; Tong, H.; and Lubensky, D. 2013.
Big-align: Fast bipartite graph alignment.
In 2013 IEEE 13th international conference on data mining, 389–398. IEEE.
Li, C.; Wang, S.; Wang, Y.; Yu, P.; Liang, Y.; Liu, Y.; and Li, Z. 2019.
Adversarial learning for weakly-supervised social network alignment.
In Proceedings of the AAAI conference on artificial intelligence, volume 33, 996–1003.
Li, Z.; Petegrosso, R.; Smith, S.; Sterling, D.; Karypis, G.; and Kuang, R. 2021.
Scalable label propagation for multi-relational learning on the tensor product of graphs.
IEEE Transactions on Knowledge and Data Engineering.
Liao, C.-S.; Lu, K.; Baym, M.; Singh, R.; and Berger, B. 2009.
IsoRankN: spectral methods for global alignment of multiple protein networks.
Bioinformatics, 25(12): i253–i258.
Liu, H.; and Yang, Y. 2016.
Cross-graph learning of multi-relational associations.
In International Conference on Machine Learning, 2235–2243. PMLR.
Liu, L.; Cheung, W. K.; Li, X.; and Liao, L. 2016.
Aligning Users across Social Networks Using Network Embedding.
In IJCAI, 1774–1780.
Man, T.; Shen, H.; Liu, S.; Jin, X.; and Cheng, X. 2016.
Predict anchor links across social networks via an embedding approach.
In Ijcai, volume 16, 1823–1829.
Maretic, H. P.; El Gheche, M.; Chierchia, G.; and Frossard, P. 2019.
GOT: an optimal transport framework for graph comparison.
Advances in Neural Information Processing Systems, 32.
Maretic, H. P.; El Gheche, M.; Chierchia, G.; and Frossard, P. 2022.
FGOT: Graph distances based on filters and optimal transport.
In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, 7710–7718.
Maretic, H. P.; Gheche, M. E.; Minder, M.; Chierchia, G.; and Frossard, P. 2020.
Wasserstein-based graph alignment.
arXiv preprint arXiv:2003.06048.
Mémoli, F. 2011.
Gromov–Wasserstein distances and the metric approach to object matching.
Foundations of computational mathematics, 11: 417–487.
Mena, G.; Belanger, D.; Linderman, S.; and Snoek, J. 2018.
Learning latent permutations with gumbel-sinkhorn networks.
arXiv preprint arXiv:1802.08665.
Park, J.-D.; Tran, C.; Shin, W.-Y.; and Cao, X. 2022.
Grad-Align: Gradual Network Alignment via Graph Neural Networks (Student Abstract).
In Proceedings of the AAAI conference on artificial intelligence, volume 36, 13027–13028.
Pass, B. 2015.
Multi-marginal optimal transport: theory and applications.
ESAIM: Mathematical Modelling and Numerical Analysis, 49(6): 1771–1790.
Peyré, G.; Cuturi, M.; and Solomon, J. 2016.
Gromov-wasserstein averaging of kernel and distance matrices.
In International Conference on Machine Learning, 2664–2672. PMLR.
Razaviyayn, M.; Hong, M.; and Luo, Z.-Q. 2013.
A unified convergence analysis of block successive minimization methods for nonsmooth optimization.
SIAM Journal on Optimization, 23(2): 1126–1153.
Scetbon, M.; Cuturi, M.; and Peyré, G. 2021.
Low-rank sinkhorn factorization.
In International Conference on Machine Learning, 9344–9354. PMLR.
Scetbon, M.; Peyré, G.; and Cuturi, M. 2022.
Linear-time gromov wasserstein distances using low rank couplings and costs.
In International Conference on Machine Learning, 19347–19365. PMLR.
Singh, R.; Xu, J.; and Berger, B. 2008.
Global alignment of multiple protein interaction networks with application to functional orthology detection.
Proceedings of the National Academy of Sciences, 105(35): 12763–12768.
Sun, L.; Zhang, Z.; Zhang, J.; Wang, F.; Du, Y.; Su, S.; and Yu, P. S. 2020.
Perfect: A hyperbolic embedding for joint social network alignment.
In IEEE International Conference on Data Mining.
Swoboda, P.; Mokarian, A.; Theobalt, C.; Bernard, F.; et al. 2019.
A convex relaxation for multi-graph matching.
In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 11156–11165.
Tang, J.; Zhang, J.; Yao, L.; Li, J.; Zhang, L.; and Su, Z. 2008.
Arnetminer: extraction and mining of academic social networks.
In Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining, 990–998.
Titouan, V.; Courty, N.; Tavenard, R.; and Flamary, R. 2019.
Optimal transport for structured data with application on graphs.
In International Conference on Machine Learning, 6275–6284. PMLR.
Tong, H.; Faloutsos, C.; and Pan, J.-Y. 2006.
Fast random walk with restart and its applications.
In Sixth international conference on data mining (ICDM'06), 613–622. IEEE.
Vincent-Cuaz, C.; Vayer, T.; Flamary, R.; Corneli, M.; and Courty, N. 2021.
Online graph dictionary learning.
In International Conference on Machine Learning, 10564–10574. PMLR.
Xu, H.; Luo, D.; and Carin, L. 2019.
Scalable gromov-wasserstein learning for graph partitioning and matching.
Advances in neural information processing systems, 32.
Xu, H.; Luo, D.; Zha, H.; and Duke, L. C. 2019.
Gromov-wasserstein learning for graph matching and node embedding.
In International conference on machine learning, 6932–6941. PMLR.
Yan, Y.; Zhang, S.; and Tong, H. 2021.
BRIGHT: A bridging algorithm for network alignment.
In Proceedings of the Web Conference 2021, 3907–3917.
Zeng, Z.; Zhang, S.; Xia, Y.; and Tong, H. 2023a.
PARROT: Position-Aware Regularized Optimal Transport for Network Alignment.
In Proceedings of the ACM Web Conference 2023, 372–382.
Zeng, Z.; Zhu, R.; Xia, Y.; Zeng, H.; and Tong, H. 2023b.
Generative Graph Dictionary Learning.
In Krause, A.; Brunskill, E.; Cho, K.; Engelhardt, B.; Sabato, S.; and Scarlett, J., eds., Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, 40749–40769. PMLR.
Zhang, J.; and Philip, S. Y. 2015.
Multiple anonymized social networks alignment.
In 2015 IEEE International Conference on Data Mining, 599–608. IEEE.
Zhang, S.; and Tong, H. 2016.
Final: Fast attributed network alignment.
In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, 1345–1354.
Zhang, S.; Tong, H.; Jin, L.; Xia, Y.; and Guo, Y. 2021.
Balancing Consistency and Disparity in Network Alignment.
In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, 2212–2222.
Zhang, S.; Tong, H.; Maciejewski, R.; and Eliassi-Rad, T. 2019.
Multilevel network alignment.
In The World Wide Web Conference, 2344–2354.
Zhang, S.; Tong, H.; Xia, Y.; Xiong, L.; and Xu, J. 2020.
Nettrans: Neural cross-network transformation.
In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 986–996.
Zhong, E.; Fan, W.; Wang, J.; Xiao, L.; and Li, Y. 2012.
Comsoc: adaptive transfer of user behaviors over composite social network.
In Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining, 696–704.
Zhou, Y.; Ren, J.; Jin, R.; Zhang, Z.; Dou, D.; and Yan, D. 2020.
Unsupervised multiple network alignment with multinominal gan and variational inference.
In 2020 IEEE International Conference on Big Data (Big Data), 868–877. IEEE.

Detailed Algorithm
[htbp]
Cluster-level alignment
{{formula:7cee3bb8-68f6-4a21-98bb-f76da41000de}}  graphs {{formula:0c805a1e-2136-4add-8498-aea38912a83c}} , hyperparameter {{formula:585b06c1-6bac-44ed-b5f7-a85965754938}} .
The cluster-level alignments {{formula:d1683c27-bb96-4b71-8e52-6aa5a0ac2165}} .
Initialize {{formula:0446eb66-dfc3-41ee-8bf7-bad59484f09b}}
{{formula:ed8f3330-b76b-4e85-ad2c-d3958075358f}}
{{formula:f31f9b8c-15ad-462d-80b5-a19695593b89}}
Compute cross-cost {{formula:6812c92c-b10c-47f7-ab11-e12497da3995}}
Compute modified cost {{formula:16296851-bdde-4fce-91c9-ab0526ca785b}}
{{formula:4ca4603d-efab-454d-b525-311330304999}}
Compute scaling vectors {{formula:1f8625bf-1a81-4322-904f-27d728aae9f1}}  by Eq. (REF );

Update {{formula:375c5ca8-a807-43fe-9bb3-bed976dd366c}}  by Eq. (REF )

Update {{formula:f4064d44-37e2-478e-828a-5c6f4917b85c}}
Update {{formula:4e793732-c498-4159-8db1-5f14a4490bfe}}

Form cluster alignments {{formula:af2a7495-67ff-4e66-85ef-a919f85510ba}}  for {{formula:34777dcc-34e3-4252-a6ad-134d497cf634}}  such that {{formula:de4a525b-42b2-411e-bdc8-60c19126df44}}
Cluster alignments {{formula:2a160aab-c54b-42e8-b1fa-7ffb8d1406ed}} .

[htbp]
Node-level alignment
{{formula:6f6e3604-dcce-46ae-9aed-ca4a812827db}}  graphs {{formula:b3aa7915-6c65-4039-a625-45dc572b7f66}} , cross-cost tensor {{formula:3e075785-3f3f-4e90-bb1f-05457dc10436}} , hyperparameter {{formula:a2bd8bc2-aded-41f2-a180-037010dff49f}} .
The node-level alignment tensor {{formula:920e7eff-6bae-49f1-a2a3-d7f20f739dfa}} .
Initialize {{formula:6baa6baf-5dac-474e-90f4-6eacaf58c1ed}}  and a feasible solution {{formula:943b53e6-e092-4e65-bbf5-91213ec700ac}}
{{formula:d35a66cc-fc5e-4799-bad5-304cb8bc9c0d}}
Compute {{formula:5f988bc4-e52b-48bf-ab7f-41bd449318d9}}
{{formula:1e586485-472f-4b2e-9d21-37abe2f42247}}
Compute scaling vectors {{formula:ad69a073-4ea3-429f-860e-2cd0cf1911a5}}  by Eq. (REF );

Update {{formula:9e0b9083-7171-4e73-9b86-77c05399f19e}}  by Eq. (REF )

Node alignment tensor {{formula:01b3a25f-fd06-4e6f-b5be-eac68dc12282}} .

[htbp]
Hierarchical multi-marginal optimal transport (Hot)
{{formula:a8d1cbcc-6837-415c-b3ae-39710d20b229}}  graphs {{formula:8fd68eb6-3963-4cab-a4ea-1b739e7a3e65}} , anchor node sets {{formula:f7970435-1e21-4d2d-932a-2ae241b8d644}} , cluster number {{formula:9c53b1f4-6685-4d0e-b241-dc8ab464f4c2}} , hyperparameter {{formula:aeffc5c1-d209-4a1b-b62d-8e1a5a97e126}} .
The alignment tensor {{formula:ac2cb7da-ef7d-4f51-8244-97928fd1e898}} .
Compute RWR scores {{formula:5a2ffeff-5604-44e5-96c3-61191db6394f}}  by Eq. (REF )
Compute cluster-level alignment {{formula:fbcba02f-5448-4901-ad32-3f1633c9aa6b}}  by Algo.
{{formula:5edb1f66-19f7-43ce-8b0f-c868568b36a8}}
Compute cross-cost tensor {{formula:b5296a75-5b32-4e6e-9388-ab2a5e125ce3}}  for cluster {{formula:dc19daf1-34c2-4d80-b04a-16d669be7630}}  by Eq. (REF )
Compute node alignment tensor {{formula:f8c42abe-3e11-4a42-8ca4-243aad0d5744}}  by Algorithm

{{formula:f46900ae-b329-4355-9834-15a7cf6157b0}} .

Proof
*

The Wasserstein term, i.e., {{formula:34890fcf-849d-4620-a2ee-b30ddfd835e6}} , is straightforward, and we mainly focus on re-formulating the Gromov-Wasserstein part into the tensor form. The Gromov-Wasserstein term in Eq. (REF ) can be written as:
{{formula:2ac51c30-c48b-45be-ab3c-1f577986bfd9}}
We further simplify {{formula:eee4f771-177c-4130-9dde-d07384a59a5a}}  as follows:
{{formula:1f8e952c-3d8e-4b9b-bdd2-2c68f821efa5}}
*

For space complexity, thanks to the block diagonal property of {{formula:ef8645d2-1fde-46bd-9dff-0ace8c6534ee}} , only non-zero elements in the small blocks need to be stored. Storing {{formula:90c88d18-8ded-41ed-b077-a2cb7e8b11c0}}  attributed networks takes {{formula:10844a07-3674-450d-a1ee-909701164f27}}  space. Storing cost tensors {{formula:0c238094-4bec-4d3a-baad-4bca9cfc1f96}}  and alignment tensors {{formula:8d51c8b7-827e-4857-a2ee-07aec5aa7b0c}}  each takes {{formula:6e34b5c1-af2a-4290-a85e-8c3688280747}}  space. Therefore, the overall space complexity for Hot is {{formula:858dab24-1f26-42a0-91d0-7c5c0162b0fa}} , which is an exponential reduction of space in terms of the number of graphs {{formula:9d821e57-8d77-46b2-b062-ea091b05ebbe}}  compared with the {{formula:d680cbdc-1aa3-428b-91c5-647612c9a0b8}}  space complexity given by the straightforward method.
For time complexity, the complexity is {{formula:f89411bf-46b5-4233-b695-a97ea597dd0e}}The cost can be reduced to be linear in {{formula:15b9808b-fe5b-4d71-9eba-af5d458eaeea}}  with the advancement of faster Sylvester Equation solver {{cite:2d95053}}. for the RWR calculation in Eq. (REF )and {{formula:eade1a59-fd27-4059-8147-bd3a8144c38b}}  for the cost tensor calculation in Eq. (REF ). For cluster-level alignment calculation, the complexities are {{formula:16fc46ef-b443-4bc5-888b-c3a907ba8aa6}}  for the optimal couplings {{formula:4232018a-4b4f-4c7c-919b-7c0ad5ba4741}}  in Eq. (REF ), {{formula:8e7544e6-e777-41be-a392-be861dc73803}}  for adjacency matrices {{formula:2086912d-a6f9-487b-a28b-cff5fcfc0291}}  in Eq. (REF ) and {{formula:06c0fed1-1a41-4204-937e-209f186d0ac2}}  for node attribute matrices {{formula:95e991e0-70a9-462c-8f48-ada2e6220a12}}  in Eq. (REF ) {{cite:08cd552}}. Thus, the complexity of cluster-level alignment is {{formula:7c7bcf7f-75df-4dd8-b362-daf51817094c}} . For node-level alignment, the complexity lies in computing {{formula:b531b39d-39fd-4f68-a7ce-2735514d8763}} , {{formula:a22b91b2-1813-4224-8ef5-6e0500beeb39}}  and {{formula:2f8735cc-45f5-48a4-8cac-1fe478c2d4d9}}  (Eqs. (REF )-(REF )) iteratively and equals {{formula:18e2eeb9-fe0d-4ebf-8600-21d4b5d86514}} . Therefore, the overall time complexity of Hot is {{formula:333f29f9-de4e-402e-b959-791c41ab7a5c}} .
*

We denote the objective function of MFGW distance under {{formula:ea57144f-9cb2-4963-b41d-b490c8684207}}  as:
{{formula:8bec82b0-05b7-4de2-af22-ac8b88104edf}}

where {{formula:e4582319-383c-48d9-b107-0e4592443293}} . The proximal point method decompose the above non-convex problem into a sequence of entropy-regularized multi-margianl OT problem as follows:
{{formula:a18a8f09-8792-4b12-a6ea-0c317d462755}}
where {{formula:4c0ba2c8-6f72-489a-af2a-d93bdedf9e67}}  denotes {{formula:ae326cb6-d9cf-4345-a065-837683696116}}  with {{formula:08fb91cf-0782-463a-90ff-d805fd73ffaf}} . Note that the Gromov-Wasserstein term {{formula:0e872db9-fdfa-458b-bb28-3ce79ba7c969}}  is a constant tensor in the above equation as {{formula:b77fc636-d9b2-48c1-bfcc-1d93a23f718a}}  is fixed during the optimization. Besides, it is obvious that the solution space {{formula:bc0bdee0-8646-434e-be18-133bc0613a30}}  is a closed convex set. We then evaluate the following conditions to hold the global convergence of the proximal point method based on Theorem 1 in {{cite:ab6b6ab}}:

C1: {{formula:e993909b-da6a-414c-93ad-63c46777637e}} .

C2: {{formula:7281c9f4-64e0-4f2c-a3b6-bff4e9fb14c6}} .

C3: {{formula:760087f0-3c76-4647-b24f-5d27fcc25829}}  with {{formula:1f787471-c49d-4cac-a5b3-7f25b3344625}} .

C4: {{formula:b5b41a20-4ff5-4cb2-b5ba-91679fa2a7d6}}  is continuous in {{formula:8d28cf95-0788-49eb-b936-29629bc7fcff}} .

For C1, it is obvious that {{formula:acc14675-f39c-4182-b3ab-6f7300a136a9}} .
For C2, since {{formula:e370b9ba-809f-4ffd-983e-e4c64bd839a3}}  and the equation holds when {{formula:65fec254-41dc-4b97-8869-e95d8483b7c1}} , we have {{formula:3b620b6d-3916-4704-b5b0-345d034bc7d4}} .
For C3, according to Proposition 1 in {{cite:ab6b6ab}}, C3 holds when C1 and C2 hold.
For C4, it is easy to validate that {{formula:2611087d-96e1-4b75-90ca-e5c4d55f2567}}  is continuous.
Therefore, the proximal point method has global convergence to a stationary point of the MFGW problem.
*

For clarity, we slightly abuse the notation {{formula:0e3fa050-c537-4864-b308-f11baada92f5}}  to denote the value of FGW distance under coupling {{formula:12a167c7-3285-44d4-a5a0-32693852e401}} . We first focus on the Wasserstein distance part in the MFGW distance in Eq. (REF ), which can be reformulated as follows:
{{formula:ea348e6d-0f11-4732-a17f-8084b269ee8c}}
We then consider the Gromov-Wasserstein distance pair in the MFGW distance in Eq. (REF ), which can be reformulated as follows:
{{formula:13fff5c3-7ed0-4e44-b75e-f6c305385bf0}}
Combining Eqs. (REF ) and (REF ), the MFGW distance can be reformulated as follows:
{{formula:4aec1952-56fc-4795-a92c-7d1ebe2dd9e3}}
Note that {{formula:af8035bf-3725-4d19-8564-4fdab09bca6f}}  is a suboptimal coupling of {{formula:8484052a-bef3-4f3e-85b3-fba693fc2a60}} , that is:
{{formula:5cd79281-1531-4a07-9544-e44950a2ce86}}
Therefore, we can prove the proposed upper bound as follows:
{{formula:6dbf952c-3445-4c0a-b613-f8f8e131e140}}

Additional Experiments
We carried out additional experiments to evaluate our proposed Hot from different aspects.
Alignment results on plain networks: (a) DBLP-500; (b) DBLP-1000.
{{figure:06d09afd-39e0-4e43-874d-1efcb6a7cc5c}}Alignment results on attributed networks: (a) ACM(A)-500; (b) DBLP(A)-500.
{{figure:2cd69c4d-ffad-4236-8489-312a7441d466}}More effectiveness results.
We provide more experimental results on plain networks: DBLP-500/DBLP-1000, and attributed networks: DBLP(A)-500/ACM(A)-500. Results are shown in Figures {{figure:06d09afd-39e0-4e43-874d-1efcb6a7cc5c}}  and {{figure:2cd69c4d-ffad-4236-8489-312a7441d466}} , respectively. Our proposed Hot consistently outperforms all baselines on these datasets in both pairwise and high-order Hits@10 metrics. Comparing the performance of Hot on DBLP-500 and DBLP(A)-500, Hot achieves nearly 100% pairwise Hits@K on DBLP-500 even without the node attributes, which is comparable to the performance on DBLP(A)-500. This validates that the position-aware cost tensor design extracts the essential positional information and is quite beneficial to plain network alignment.
We also test the proposed Hot on larger graphs, including the plain DBLP and attributed DBLP networks with up to 6000 nodes. We set the cluster number as {{formula:b4b3fb5e-dba9-436a-a3b4-e985c67a4393}} . Experiment results are shown in Tables {{table:52d6c07a-ab54-4783-bf68-8899280958a5}}  and {{table:910ce900-d78c-4ea0-b947-3f07d2367fc7}} . It is shown that the proposed Hot can achieves excellent performance when aligning large multiple networks. Besides, as stated before, all baseline methods constructing a dense alignment tensor fail to handle all the large multi-network alignment problems (with more than 2000 nodes) due to OOM.
Experiments on aligning larger DBLP plain networks.
2*#nodes
Pairwise Hits
High-order Hits
2*MRR
(lr)2-11
@1
@5
@10
@30
@50
@1
@5
@10
@30
@50

2000
78.0{{formula:cd64460f-ed5b-4390-8d47-099a4969a348}} 1.3
91.2{{formula:a73c22af-f059-4d95-9ce3-a26412fb206c}} 1.0
92.9{{formula:54ba7595-e6ef-4940-a4d1-53f62cb36e74}} 1.0
94.2{{formula:625080ae-4c67-463a-b16b-27955be6b5e5}} 0.9
94.5{{formula:d08317af-100f-4ecc-b86a-2c0c5d99e6ac}} 0.9
55.1{{formula:ab8f6bbf-4548-43d6-b140-9904bc1cbb72}} 2.1
67.8{{formula:3ae12026-7f1b-420c-ba51-e7d3fc58ffdb}} 1.6
71.7{{formula:d41976f0-182e-4ed6-9e65-86e5f70f0d85}} 1.5
76.8{{formula:13bd8f0c-6059-458f-b463-93be7e3d663a}} 1.7
78.2{{formula:872dad60-b171-434d-818a-b21c13f878f4}} 1.7
60.7{{formula:7645fcdd-85fe-4a61-b7cf-b69b10f60ca8}} 1.8
3000
69.5{{formula:a7938db6-eed4-46fe-b997-2b406519650b}} 1.9
82.8{{formula:fc004df7-1c57-4d93-bdcc-43281d3510db}} 1.8
84.8{{formula:8206b9d3-248c-44ca-9c59-a6385c5262c4}} 1.7
86.5{{formula:d48c6b8d-5963-4ffe-9171-cf660067c063}} 1.4
86.9{{formula:acd1afec-1ba7-4be2-bff1-29895550c1ed}} 1.3
40.9{{formula:744d0a21-ba43-421a-a496-94f634d522b2}} 1.4
52.6{{formula:a5847e88-db2e-421a-a788-7134cd2ba212}} 1.2
56.7{{formula:cbae126e-107e-4af7-a54e-107494dbddff}} 1.3
61.3{{formula:edc1c822-3a51-4d50-afa3-fc9cc40c9d40}} 1.4
62.5{{formula:c497de6c-a3d9-4c49-8761-21cc68d5d5ed}} 1.4
46.3{{formula:35f8d817-41bc-4d1b-a021-b4f5092702a9}} 1.3
4000
65.1{{formula:a14b4faa-85bc-4e65-ba3f-b4d5033028f0}} 0.8
80.7{{formula:b7866926-c1c3-402e-821a-d952119ec1f8}} 0.9
83.2{{formula:1f075181-6388-4187-be26-e4499a5368bb}} 0.8
85.0{{formula:23f24c71-7098-4c1b-b064-9ac483dcab07}} 0.8
85.5{{formula:86406723-449c-491f-bc43-8a0d35e86811}} 0.8
38.0{{formula:7bf66fe1-01a8-400b-9fd7-27733b1702d6}} 0.6
49.6{{formula:c111437a-d1c9-4236-b8ba-649c8c5e4edf}} 0.8
54.0{{formula:07865c3c-dd5b-4161-9aac-92b70240d960}} 1.0
59.6{{formula:0c064846-2cfa-4c15-980d-e6f069c8e4ce}} 0.7
61.4{{formula:5fb8b02b-9b65-461f-b763-e9160bfb0f01}} 0.8
43.4{{formula:c5b56870-7dfc-482b-990a-d9123d87c5ee}} 0.7
5000
76.9{{formula:9d95cb43-9c98-488d-9d0f-7321a1ac7697}} 0.9
91.5{{formula:6440deac-6228-4e43-9f73-d62c2cdac156}} 0.6
93.5{{formula:88d2b8e5-b964-4c4a-93e0-4750e4a7f4ac}} 0.5
94.5{{formula:f34bb549-230a-42d9-bd34-7e91775f1b21}} 0.5
94.6{{formula:8af04160-dccb-44c1-932b-2f05c46935f5}} 0.5
53.9{{formula:5bdb597c-bf41-4e70-a6de-a7d8bedcda55}} 0.9
67.2{{formula:2bcbf6f8-4725-4016-b0db-702aa99443a3}} 0.9
72.2{{formula:2987b3b1-19fd-4703-8e69-d965fce53d47}} 1.0
77.6{{formula:873f1d1c-7300-418a-8bff-081ab7a7fb39}} 1.0
79.1{{formula:bdef916a-bf16-4ee1-93b0-f48f91537324}} 1.0
59.8{{formula:f60ee998-7a9b-4884-8d33-ea7f298a274b}} 0.9
6000
64.7{{formula:dbbe3bf3-712c-441f-a019-137b48e7ff07}} 0.9
81.1{{formula:8995f1c5-76c8-4adf-b32b-60d2ee987452}} 0.8
83.7{{formula:69c23444-ca07-469c-b434-320fdfd9fd01}} 0.7
85.6{{formula:20359d75-db7d-46b5-9e84-639f8717cca4}} 0.6
86.1{{formula:42e1d390-acac-482a-beb7-8d41e62759e2}} 0.6
37.5{{formula:1f2d3f3f-882f-4d1d-adc6-660a0b111ad9}} 0.8
49.3{{formula:4a2f22d8-68a7-4b2d-8817-2c92610f36ba}} 0.8
54.2{{formula:94727604-595c-4396-b868-3dd7f72c77f1}} 0.8
60.3{{formula:f60a3724-8b2d-433f-8987-eeef6630b515}} 1.0
62.3{{formula:42977f6f-4c92-4992-886c-75389225b00a}} 0.9
42.9{{formula:70428fd7-e44a-4a28-9cd9-9ed6e9d08ed3}} 0.8
{{table:67617f3c-6068-4905-be39-d7ffcaf8ed48}}{{table:52d6c07a-ab54-4783-bf68-8899280958a5}}Experiments on aligning larger DBLP attributed networks.
2*#nodes
Pairwise Hits
High-order Hits
2*MRR
(lr)2-11
@1
@5
@10
@30
@50
@1
@5
@10
@30
@50

2000
91.7{{formula:d3525381-9e42-403e-a919-baa32af08baa}} 0.6
98.1{{formula:b8dc6fb0-62fc-4b5b-8c6c-f99a4a5a0552}} 0.3
98.6{{formula:db4aa6d6-b8ee-47b9-92d2-cd3a0f6aeeb3}} 0.3
98.9{{formula:8147ca6b-65df-4ff4-a7ef-a74e1a6150d4}} 0.3
99.0{{formula:6ae6a010-e303-4c4f-acf0-24632eba2d31}} 0.3
82.2{{formula:33ce09d6-c1af-4c1c-a627-046b08ef2fa9}} 0.8
88.8{{formula:26563a67-8c2d-4184-81c3-ec2d4bd856cf}} 0.7
91.6{{formula:85ad49df-fa0c-47a2-b630-31962fe9bad7}} 0.6
94.4{{formula:6cee59ab-caa4-463c-8f21-7e465363abc1}} 0.6
94.9{{formula:0c65c802-2956-4bb8-b63d-a4905ffa9913}} 0.6
85.1{{formula:616ff609-cfdb-4257-b7a6-7d565b62e0da}} 0.6
3000
91.0{{formula:1a93671f-1051-4233-885c-731542c82d61}} 0.4
97.0{{formula:11d3f141-6ab1-4a1b-939b-39e03c9709c6}} 0.4
97.4{{formula:047e10af-0362-43fa-8b58-b09931038433}} 0.4
97.6{{formula:c1cbca20-bb18-4863-9c82-960e4e4a329c}} 0.4
97.7{{formula:7bf3a7e5-d8bb-4e3c-a897-5b60bd536bd3}} 0.4
78.6{{formula:549a06b5-e597-477d-9907-cb35b09e9b9c}} 0.7
85.5{{formula:92117625-c216-4f21-9626-8cf249010682}} 0.8
88.0{{formula:667bbb22-6aa3-4744-b6ab-46aeb6408052}} 0.8
89.6{{formula:bc4ad4c6-32d0-4375-881d-f7bde4d14059}} 0.6
89.9{{formula:341fa9ab-f93d-4681-be45-f2b2cd4b9f29}} 0.6
81.6{{formula:c37cede2-0ea7-46f2-9010-98b34b180dad}} 0.7
4000
88.1{{formula:4efa28bc-56a2-49d3-bee5-c3e57dce5093}} 0.4
96.3{{formula:44411dd7-6a41-4373-a2e2-6d0cd4ec4d34}} 0.3
97.1{{formula:cc4c39c1-b26a-4dfb-b510-4df4dbfaaf30}} 0.3
97.4{{formula:647d8acf-e868-48e1-88df-d02ac19ec9bd}} 0.3
97.4{{formula:3cdfa890-42f0-4264-817c-32ded00ea718}} 0.3
74.4{{formula:97b49032-2485-42c4-94af-9fa18e3e9196}} 0.8
82.4{{formula:87605d5c-7bf4-4ae7-921b-43b71f386004}} 0.7
85.3{{formula:471ffc79-e076-4e10-b2d2-7941902f06ee}} 0.6
88.3{{formula:981410e5-b7cd-452f-bc2a-12d1efb51e0d}} 0.7
89.1{{formula:665e838f-0192-47bb-a913-15040f105b38}} 0.7
77.9{{formula:9fbafeed-df1f-4fd3-a12d-6aa0e939f5e9}} 0.8
5000
88.9{{formula:444037b1-938f-41f1-8eee-444f7bf400e6}} 0.3
97.8{{formula:51266249-a6eb-46e4-b92b-e45a0f32893e}} 0.2
98.7{{formula:9f0b3069-672d-4d13-a248-1c026ebeec05}} 0.2
99.0{{formula:8ae37fe7-68c4-46ad-a47c-7b1aecca30e4}} 0.2
99.1{{formula:d1d80c46-c975-43d2-bada-e40ab61b6aa6}} 0.2
78.2{{formula:a99ad3ee-f39c-4748-82a9-b9be86ffbabc}} 0.4
86.5{{formula:a392888a-2324-4595-b4cd-53adaacd36d7}} 0.4
90.1{{formula:cdf72839-df39-4a8f-94a2-c6fd261ae564}} 0.4
93.8{{formula:7dbe88bb-c422-4f74-bdd1-896a8cc01420}} 0.5
94.6{{formula:315f152e-08ea-4aa6-b2b6-4da2e7fa5a79}} 0.5
81.9{{formula:42924f4c-668e-46ce-8058-93f511dbe546}} 0.4
6000
86.9{{formula:2349fab4-1663-4a83-8091-d253dbfc5dac}} 0.5
96.3{{formula:de92dd14-b2f3-4561-995f-fcb8ad65ed20}} 0.4
97.2{{formula:6f7d31fc-d8de-45a1-a0fb-8fd75294b9dd}} 0.4
97.6{{formula:315cc55a-2870-4335-aa83-7f4669befec7}} 0.4
97.6{{formula:0551085e-4939-4f28-ac8b-68a1bbce3fd1}} 0.3
73.0{{formula:54086695-be20-4a88-b02d-5f22430f2346}} 0.7
81.7{{formula:31c11102-7905-4cd7-9831-619cab8f0ad0}} 0.7
85.0{{formula:31ee208b-7f13-454c-a9b6-7abd534f6fd9}} 0.6
88.7{{formula:035e507a-4b45-438f-ae05-f30a97963a3c}} 0.6
89.7{{formula:59ff8cf1-7fb1-4438-beca-c653e5fe53ae}} 0.6
76.8{{formula:4b2b79b1-428d-40ed-a2b6-1a72cf9bcc77}} 0.7
{{table:b6710366-933c-42d8-b8ff-29f90904ac6b}}{{table:910ce900-d78c-4ea0-b947-3f07d2367fc7}}
Effectiveness vs scalability.
We conducted an analysis to determine how the number of clusters ({{formula:0a5659ef-6dcf-4295-aa42-a0b088691d2f}} ) impacts the effectiveness and scalability of Hot, and the results are shown in Figure {{figure:d8f51f0f-0a3b-4670-93ca-375cb9b8e8b9}} . In general, as the number of clusters {{formula:6a500e00-d2ea-46c3-9ce7-16853d84749b}}  increases, the alignment performance drops slightly, but the running time reduces dramatically. However, the alignment performance is still relatively stable when {{formula:3350700f-8714-46a0-a28b-c13006a55aa3}}  changes, with an average standard deviation of 2.6%. The effect of {{formula:f34a97fb-60d0-4abd-a8ec-d7443ceadb42}}  on the alignment accuracy is two-fold. On one hand, increasing {{formula:33515151-eda1-4418-84d9-b813949a70bb}}  may result in aligned nodes being assigned to different clusters, which can be detrimental to the overall performance and may explain the decreasing trend of high-order Hits@10. On the other hand, if the cluster-level alignment is accurate, a higher value of {{formula:9239b16b-92a5-4d1f-ba1c-24ac7b3924b7}}  can lead to a significant reduction in the search space, which may explain the intermediate performance increase.
Furthermore, comparing the running time with different {{formula:9e296b25-5454-45d4-9547-678d6fd6aadc}} , the running time exhibits a huge drop when {{formula:5054b0eb-7761-4da4-8b75-f0d5da54d460}}  increases from 5 to 20, and stays relatively stable when {{formula:bb37e881-45a8-4b47-b0af-13d46f9cb523}}  further increases from 20 to 40. These results are consistent with our time complexity analysis in Section REF  and scalability results in Figure {{figure:6cd79ac9-91dd-4fd8-b986-11505673d1a0}} . Specifically, when {{formula:84269aeb-acfc-4e98-9746-c16df304e13f}}  is relatively small, the node-level alignment dominates the overall time complexity, resulting in an exponential decrease in running time. Conversely, when {{formula:3d47cfbb-fbb8-4a8b-bfb2-db43bcca870e}}  is relatively large, the cluster-level alignment dominates the overall time complexity, leading to a slow increase in running time.
High-order Hits@10 vs running time: (a) ACM(A)-500, (b) DBLP(A)-500. Number of clusters {{formula:f4b208f3-ebfa-410f-99ee-b3ad8a0d0f18}}  achieves a trade-off between effectiveness and scalability.
ACM(A)-500
DBLP(A)-500
{{figure:d8f51f0f-0a3b-4670-93ca-375cb9b8e8b9}}
Reproducibility
Baseline methods summary.
Methods
Types
Scenarios
Settings
Hierarchical
IsoRank {{cite:51b9e0b}}
consistency
plain
pairwise
✗
FINAL {{cite:dc4f300}}
consistency
plain&attributed
pairwise
✗
MOANA {{cite:c4fbdbb}}
consistency
attributed
pairwise
✓
SYTE {{cite:7163756}}
consistency
plain&attributed
multiple
✗
CrossMNA {{cite:4a54acc}}
embedding
plain
multiple
✗
NetTrans {{cite:dbb6816}}
embedding
attributed
pairwise
✗
NeXtAlign {{cite:66c289d}}
embedding
plain&attributed
pairwise
✗
Grad-Align {{cite:1fc2a85}}
embedding
attributed
pairwise
✗
GW {{cite:713c1e8}}
OT
plain
pairwise
✗
FGW {{cite:08cd552}}
OT
attributed
pairwise
✗
S-GWL {{cite:ac2de2d}}
OT
attributed
pairwise
✓
LOT {{cite:3af81ea}}
OT
plain
pairwise
✗
WAlign {{cite:44825ad}}
OT
attributed
pairwise
✗
{{table:c82376d7-2f9c-440b-81ba-f1bd47cb94e4}}Dataset summary.
1mm
-5pt
Scenarios
Networks
# nodes
# edges
# attributes
4*Plain
ER
500/500/500
1813/2160/1982
0

Douban
230/230/242
336/344/356
0

DBLP-500
500/500/500
1710/1788/1785
0

DBLP-1000
1000/1000/1000
3627/3807/3807
0
4*Attributed
ACM(A)-500
500/ 500/ 500
1813/2160/1982
17

ACM(A)-1000
1000/1000/1000
3790/4539/4159
17

DBLP(A)-500
500/500/500
1710/1788/1785
17

DBLP(A)-1000
1000/1000/1000
3627/3807/3807
17
{{table:ace088de-3700-4cb7-af66-a578c1924220}}{{table:6797329c-b4cc-4090-8bf8-56fc0ce77fcd}}Baseline methods descriptions.
A brief overview of baseline methods is given in Table {{table:c82376d7-2f9c-440b-81ba-f1bd47cb94e4}} . We categorize baseline methods based on three properties: method type (consistency-based, embedding-based and OT-based), alignment scenarios (plain and attributed networks), setting (pairwise and multi-network alignment) and whether it is a hierarchical approach or not.

Dataset descriptions.
The datasets used in our experiments include:

ER: Synthetic Erdös-Rényi random graphs. Each network is a permutation of the base network with noise added by first inserting 10% edges and then removing 15% edges.

Douban {{cite:1c2eaac}}: Social networks modeling both online and offline activities. Nodes represent users and edges model friendships between two users. The original dataset contains 50k users and 5M edges.

ACM {{cite:1d10635}}: Co-authorship network of ACM Digital library. Nodes represent authors and an edge exists between two authors if they are co-author for at least one publication. Node attributes indicate the number of papers published in different venues by the node/author. The original dataset includes 9,916 nodes and 44,808 edges.

DBLP {{cite:1d10635}}: Co-authorship network of DBLP bibliography. Nodes represent authors and an edge exists between two authors if they are co-author for at least one publication. Node attributes indicate the number of papers published in different venues by the node/author. The original dataset includes 9,872 nodes and 39,561 edges.

Owing to the high time and space complexities of many baseline methods, we randomly sample connected subgraphs with 500 and 1000 nodes from the ACM and DBLP dataset as the benchmark dataset. Detailed dataset statistics can be found in Table {{table:6797329c-b4cc-4090-8bf8-56fc0ce77fcd}} .

Machine configuration and code.
The proposed method is implemented in Python. Experiments are conducted on an Apple M1 CPU with 16 GB RAM and a NVIDIA Tesla V100 SXM2 GPU. We will release the source code and the datasets after the paper is published.

Impacts, Limitations and Future Works
In this paper, we study the multi-network alignment problem from the view of multi-marginal optimal transport. This work could benefit various downstream applications, including computer vision {{cite:3094220}}, high-order recommendation {{cite:7465104}} and drug discovery {{cite:84808ad}}. We emphasize that our work has no negative social impact and discuss the limitations and possible future directions as follows:

Multi-level alignment: the current work only considers two level: node-level and cluster-level. A possible future direction is to generalize the current framework to multiple levels by recursively applying the FGW barycenter to form aligned clusters at multiple levels, through which the time and space complexities can be further reduced dramatically.

Leveraging supervision: In this paper, we incorporate the supervision, i.e., anchor node sets, by generating a unified positional embedding based on RWR. Though quite effective, we can still leverage supervision from other aspects. For example, enforcing the one-hot alignment scores for anchor node sets based on the supervised optimal transport {{cite:7fe6f4e}}, or adding alignment preference regularization {{cite:55dd44e}} to penalize the inconsistency between alignment results and supervision.

Accelerating with low-rank approximation: Although the proposed Hot achieves an exponential reduction in both time and space complexities via the hierarchical approach, but the complexity is still exponential w.r.t. number of graphs. We may reduce the time and space complexities by exploring the low-rank property of graph data based on the low-rank Sinkhorn factorization {{cite:4306c07}}.


Open Graph Benchmark: Datasets for Machine Learning on Graphs

Abstract: We present the Open Graph Benchmark (OGB), a diverse set of challenging and realistic benchmark datasets to facilitate scalable, robust, and reproducible graph machine learning (ML) research. OGB datasets are large-scale (up to 100+ million nodes and 1+ billion edges), encompass multiple important graph ML tasks, and cover a diverse range of domains, ranging from social and information networks to biological networks, molecular graphs, source code ASTs, and knowledge graphs. For each dataset, we provide a unified evaluation protocol using meaningful application-specific data splits and evaluation metrics. In addition to building the datasets, we also perform extensive benchmark experiments for each dataset. Our experiments suggest that OGB datasets present significant challenges of scalability to large-scale graphs and out-of-distribution generalization under realistic data splits, indicating fruitful opportunities for future research. Finally, OGB provides an automated end-to-end graph ML pipeline that simplifies and standardizes the process of graph data loading, experimental setup, and model evaluation. OGB will be regularly updated and welcomes inputs from the community. OGB datasets as well as data loaders, evaluation scripts, baseline code, and leaderboards are publicly available at this https URL .

Inductive Representation Learning on Large Graphs

Abstract: Low-dimensional embeddings of nodes in large graphs have proved extremely useful in a variety of prediction tasks, from content recommendation to identifying protein functions. However, most existing approaches require that all nodes in the graph are present during training of the embeddings; these previous approaches are inherently transductive and do not naturally generalize to unseen nodes. Here we present GraphSAGE, a general, inductive framework that leverages node feature information (e.g., text attributes) to efficiently generate node embeddings for previously unseen data. Instead of training individual embeddings for each node, we learn a function that generates embeddings by sampling and aggregating features from a node's local neighborhood. Our algorithm outperforms strong baselines on three inductive node-classification benchmarks: we classify the category of unseen nodes in evolving information graphs based on citation and Reddit post data, and we show that our algorithm generalizes to completely unseen graphs using a multi-graph dataset of protein-protein interactions.

Benchmarking Graph Neural Networks

Abstract: Graph neural networks (GNNs) have become the standard toolkit for analyzing and learning from data on graphs. As the field grows, it becomes critical to identify key architectures and validate new ideas that generalize to larger, more complex datasets. Unfortunately, it has been increasingly difficult to gauge the effectiveness of new models in the absence of a standardized benchmark with consistent experimental settings. In this paper, we introduce a reproducible GNN benchmarking framework, with the facility for researchers to add new models conveniently for arbitrary datasets. We demonstrate the usefulness of our framework by presenting a principled investigation into the recent Weisfeiler-Lehman GNNs (WL-GNNs) compared to message passing-based graph convolutional networks (GCNs) for a variety of graph tasks, i.e. graph regression/classification and node/link prediction, with medium-scale datasets.

Benchmarking Graph Neural Networks

Benchmarking Graph Neural Networks

Benchmarking Graph Neural Networks

Benchmarking Graph Neural Networks

Summary of selected datasets for GNNs conducting the node- and graph-level tasks.

In this section, we introduce the main learning tasks for GNNs and PGMs that are studied in this survey. For GNN-related tasks, we have four main categories including structured prediction for node and graph classification, GNN explanation, graph structure learning; and for PGM-related tasks, we have two main categories including exact inference and DAG learning.
For each category and task, we summarize the representative models in Table {{table:4b88e3e9-5e4b-4a06-9ac9-c549461e12c5}} , and introduce the relevant datasets in Table {{table:678a486e-e462-4a78-8105-ae29a70940ec}} .

We summarize selected node benchmark datasets in Table {{table:678a486e-e462-4a78-8105-ae29a70940ec}} ; node-level tasks include node classification and node regression.
In node-level tasks, most methods
follow full-supervised splits of train/valid/test or semi-supervised splits on benchmark datasets including citation networks, commercial networks, social networks, proteins graphs, mathematical modeling graphs: Cora, Citeseer, Pubmed, NELL, PATTERN, CLUSTER, Reddit, ogbn-products, ogbn-proteins, ogbn-arxiv, ogbn-papers100M, ogbn-mag. And it is required to report the number of model parameters, the total training time in second or millisecond, the number of training epochs per second or millisecond, and the average accuracy or ROC-AUC with standard deviation on the test data over at least five runtimes.

We summarize selected graph benchmark datasets in Table {{table:678a486e-e462-4a78-8105-ae29a70940ec}} , graph-level tasks include graph classification and graph regression.
In graph-level tasks, most methods
follow full-supervised splits of train/valid/test on benchmark datasets including computer vision graphs, chemistry graphs, molecular graphs, protein graphs, social networks: MUTAG, DD, NCI1,
PROTEINS, REDDIT-BINARY, ZINC, MNIST, CIFAR10, ogbg-molhiv, ogbg-molpcba, ogbg-ppa. And it is required to report the number of model parameters, the total training time in second or millisecond, the number of training epochs per second or millisecond, the average accuracy, AP, MAE, or ROC-AUC with standard deviation on the test data over at least five runtimes.

Researchers present experiments to evaluate example-level or model-level explanations for node classification and graph classification tasks on either synthetic datasets or real-world datasets, e.g., benchmarking datasets listed in Table {{table:678a486e-e462-4a78-8105-ae29a70940ec}}  can also be used in GNN explanation task. Or more often,
the trust weighted signed networks, Bitcoin-Alpha and Bitcoin-OTC {{cite:3502b71}}, are two datasets for evaluating GNN explanations on node-level tasks; the computer vision graphs MNIST {{cite:cb39884}}, the molecular graphs MUTAG, and the social networks Reddit-Binary {{cite:cf12906}}, are three datasets for evaluating GNN explanations on graph-level tasks. A GNN is initially trained on these datasets to generate some representations or label distributions, then each explainable method is compared by the explanation accuracy.

Like the GNN explanation task, benchmarking datasets listed in Table {{table:678a486e-e462-4a78-8105-ae29a70940ec}}  can be used in graph structure learning for GNNs as well. {{cite:5750f7a}} evaluate their model on incomplete Cora and Citeseer networks where they construct graphs
with missing edges by randomly sampling 25%, 50%, and 75% of the edges. Authors of vGCN {{cite:7ba4c65}} also test on incomplete Cora and Citeseer graphs with a different setting.

Experiments for explainability, structure learning, and exact inference demonstrate successes on small graphs and graphical datasets.
However, existing datasets for explainable GNNs, graph structure learning, inference, and DAG learning still remain limited in terms of their scale and accessibility. Graph datasets (see Table {{table:678a486e-e462-4a78-8105-ae29a70940ec}} ) used in classification/regression tasks are also often used in other tasks, however, each task should have its specific datasets and benchmarks for evaluation and comparison. Future experiments should consider training and testing on larger and more diverse real-world networks, as well as on broader classes of graphical models with more interesting sufficient statistics for nodes. Therefore we encourage future studies to collect more data on large diverse real-world graphs for training, and proper baselines and benchmarks for fair evaluation and comparison.


Subgraph Federated Learning with Missing Neighbor Generation

Abstract: Graphs have been widely used in data mining and machine learning due to their unique representation of real-world objects and their interactions. As graphs are getting bigger and bigger nowadays, it is common to see their subgraphs separately collected and stored in multiple local systems. Therefore, it is natural to consider the subgraph federated learning setting, where each local system holds a small subgraph that may be biased from the distribution of the whole graph. Hence, the subgraph federated learning aims to collaboratively train a powerful and generalizable graph mining model without directly sharing their graph data. In this work, towards the novel yet realistic setting of subgraph federated learning, we propose two major techniques: (1) FedSage, which trains a GraphSage model based on FedAvg to integrate node features, link structures, and task labels on multiple local subgraphs; (2) FedSage+, which trains a missing neighbor generator along FedSage to deal with missing links across local subgraphs. Empirical results on four real-world graph datasets with synthesized subgraph federated learning settings demonstrate the effectiveness and efficiency of our proposed techniques. At the same time, consistent theoretical implications are made towards their generalization ability on the global graphs.

Federated Graph Classification over Non-IID Graphs

Abstract: Federated learning has emerged as an important paradigm for training machine learning models in different domains. For graph-level tasks such as graph classification, graphs can also be regarded as a special type of data samples, which can be collected and stored in separate local systems. Similar to other domains, multiple local systems, each holding a small set of graphs, may benefit from collaboratively training a powerful graph mining model, such as the popular graph neural networks (GNNs). To provide more motivation towards such endeavors, we analyze real-world graphs from different domains to confirm that they indeed share certain graph properties that are statistically significant compared with random graphs. However, we also find that different sets of graphs, even from the same domain or same dataset, are non-IID regarding both graph structures and node features. To handle this, we propose a graph clustered federated learning (GCFL) framework that dynamically finds clusters of local systems based on the gradients of GNNs, and theoretically justify that such clusters can reduce the structure and feature heterogeneity among graphs owned by the local systems. Moreover, we observe the gradients of GNNs to be rather fluctuating in GCFL which impedes high-quality clustering, and design a gradient sequence-based clustering mechanism based on dynamic time warping (GCFL+). Extensive experimental results and in-depth analysis demonstrate the effectiveness of our proposed frameworks.

A summary of three representative FGL algorithms, where we only list the behaviors other than ordinary local updates and aggregation.

Requirements of Federated Graph Learning

We first review the existing FGL algorithms and summarize their uniqueness against general FL. As shown in Table {{table:c9f1a4fb-211c-438d-997d-84cc7b4a7160}} , the three very recent FGL works, targeting different tasks, need to exchange heterogeneous data across the participants. In contrast, in each round of a general FL procedure (e.g., using FedAvg), only homogeneous data are exchanged for one pass from server to clients and one pass back. As a result of this difference, the participant of an FGL course often executes rich kinds of subroutines to handle the received data and prepare what to send. In contrast, the participant of a general FL course has canonical behaviors, i.e., local updates or aggregation.


Kvasir-SEG: A Segmented Polyp Dataset

Abstract: Pixel-wise image segmentation is a highly demanding task in medical-image analysis. In practice, it is difficult to find annotated medical images with corresponding segmentation masks. In this paper, we present Kvasir-SEG: an open-access dataset of gastrointestinal polyp images and corresponding segmentation masks, manually annotated by a medical doctor and then verified by an experienced gastroenterologist. Moreover, we also generated the bounding boxes of the polyp regions with the help of segmentation masks. We demonstrate the use of our dataset with a traditional segmentation approach and a modern deep-learning based Convolutional Neural Network (CNN) approach. The dataset will be of value for researchers to reproduce results and compare methods. By adding segmentation masks to the Kvasir dataset, which only provide frame-wise annotations, we enable multimedia and computer vision researchers to contribute in the field of polyp segmentation and automatic analysis of colonoscopy images.

A Benchmark for Endoluminal Scene Segmentation of Colonoscopy Images

Abstract: Colorectal cancer (CRC) is the third cause of cancer death worldwide. Currently, the standard approach to reduce CRC-related mortality is to perform regular screening in search for polyps and colonoscopy is the screening tool of choice. The main limitations of this screening procedure are polyp miss rate and the inability to perform visual assessment of polyp malignancy. These drawbacks can be reduced by designing decision support systems (DSS) aiming to help clinicians in the different stages of the procedure by providing endoluminal scene segmentation. Thus, in this paper, we introduce an extended benchmark of colonoscopy image segmentation, with the hope of establishing a new strong benchmark for colonoscopy image analysis research. The proposed dataset consists of 4 relevant classes to inspect the endoluminal scene, targeting different clinical needs. Together with the dataset and taking advantage of advances in semantic segmentation literature, we provide new baselines by training standard fully convolutional networks (FCNs). We perform a comparative study to show that FCNs significantly outperform, without any further postprocessing, prior results in endoluminal scene segmentation, especially with respect to polyp segmentation and localization.

Kvasir-Instrument: Diagnostic and therapeutic tool segmentation dataset in gastrointestinal endoscopy

Abstract: Gastrointestinal (GI) pathologies are periodically screened, biopsied, and resected using surgical tools. Usually the procedures and the treated or resected areas are not specifically tracked or analysed during or after colonoscopies. Information regarding disease borders, development and amount and size of the resected area get lost. This can lead to poor follow-up and bothersome reassessment difficulties post-treatment. To improve the current standard and also to foster more research on the topic we have released the ``Kvasir-Instrument'' dataset which consists of $590$ annotated frames containing GI procedure tools such as snares, balloons and biopsy forceps, etc. Beside of the images, the dataset includes ground truth masks and bounding boxes and has been verified by two expert GI endoscopists. Additionally, we provide a baseline for the segmentation of the GI tools to promote research and algorithm development. We obtained a dice coefficient score of 0.9158 and a Jaccard index of 0.8578 using a classical U-Net architecture. A similar dice coefficient score was observed for DoubleUNet. The qualitative results showed that the model did not work for the images with specularity and the frames with multiple instruments, while the best result for both methods was observed on all other types of images. Both, qualitative and quantitative results show that the model performs reasonably good, but there is a large potential for further improvements. Benchmarking using the dataset provides an opportunity for researchers to contribute to the field of automatic endoscopic diagnostic and therapeutic tool segmentation for GI endoscopy.

Table shows the datasets we used to evaluate our model on several medical image segmentation tasks. “# images", “Image Size" and “Application" represent how many images there are in the corresponding dataset, the width, and height information of the images, and the application domain, respectively.

The datasets we used to evaluate our model and the dataset properties are summarized in Table {{table:89c230f2-099e-4b43-9150-b24861b3050e}} :


Attacks and Intrusion Detection in Cloud Computing Using Neural Networks and Particle Swarm Optimization Algorithms

Abstract: Today, cloud computing has become popular among users in organizations and companies. Security and efficiency are the two major issues facing cloud service providers and their customers. Since cloud computing is a virtual pool of resources provided in an open environment (Internet), cloud-based services entail security risks. Detection of intrusions and attacks through unauthorized users is one of the biggest challenges for both cloud service providers and cloud users. In the present study, artificial intelligence techniques, e.g. MLP Neural Network sand particle swarm optimization algorithm, were used to detect intrusion and attacks. The methods were tested for NSL-KDD, KDD-CUP datasets. The results showed improved accuracy in detecting attacks and intrusions by unauthorized users.

Deep Learning Based Joint Resource Scheduling Algorithms for Hybrid MEC Networks

Abstract: In this paper, we consider a hybrid mobile edge computing (H-MEC) platform, which includes ground stations (GSs), ground vehicles (GVs) and unmanned aerial vehicle (UAVs), all with mobile edge cloud installed to enable user equipments (UEs) or Internet of thing (IoT) devices with intensive computing tasks to offload. Our objective is to obtain an online offloading algorithm to minimize the energy consumption of all the UEs, by jointly optimizing the positions of GVs and UAVs, user association and resource allocation in real-time, while considering the dynamic environment. To this end, we propose a hybrid deep learning based online offloading (H2O) framework where a large-scale path-loss fuzzy c-means (LSFCM) algorithm is first proposed and used to predict the optimal positions of GVs and UAVs. Secondly, a fuzzy membership matrix U-based particle swarm optimization (U-PSO) algorithm is applied to solve the mixed integer nonlinear programming (MINLP) problems and generate the sample datasets for the deep neural network (DNN) where the fuzzy membership matrix can capture the small-scale fading effects and the information of mutual interference. Thirdly, a DNN with the scheduling layer is introduced to provide user association and computing resource allocation under the practical latency requirement of the tasks and limited available computing resource of H-MEC. In addition, different from traditional DNN predictor, we only input one UE information to the DNN at one time, which will be suitable for the scenarios where the number of UE is varying and avoid the curse of dimensionality in DNN.

HS for addressing issues in CC and EC.

This section discusses the works on the use of HS for solving issues in CC and EC, which are summarized in Table {{table:1bf89a2b-a39f-4deb-93f0-95dc4f0e3f6b}} .