D2A: A Dataset Built for AI-Based Vulnerability Detection Methods Using Differential Analysis

Abstract: Static analysis tools are widely used for vulnerability detection as they understand programs with complex behavior and millions of lines of code. Despite their popularity, static analysis tools are known to generate an excess of false positives. The recent ability of Machine Learning models to understand programming languages opens new possibilities when applied to static analysis. However, existing datasets to train models for vulnerability identification suffer from multiple limitations such as limited bug context, limited size, and synthetic and unrealistic source code. We propose D2A, a differential analysis based approach to label issues reported by static analysis tools. The D2A dataset is built by analyzing version pairs from multiple open source projects. From each project, we select bug fixing commits and we run static analysis on the versions before and after such commits. If some issues detected in a before-commit version disappear in the corresponding after-commit version, they are very likely to be real bugs that got fixed by the commit. We use D2A to generate a large labeled dataset to train models for vulnerability identification. We show that the dataset can be used to build a classifier to identify possible false alarms among the issues reported by static analysis, hence helping developers prioritize and investigate potential true positives first.

Devign: Effective Vulnerability Identification by Learning Comprehensive Program Semantics via Graph Neural Networks

Abstract: Vulnerability identification is crucial to protect the software systems from attacks for cyber security. It is especially important to localize the vulnerable functions among the source code to facilitate the fix. However, it is a challenging and tedious process, and also requires specialized security expertise. Inspired by the work on manually-defined patterns of vulnerabilities from various code representation graphs and the recent advance on graph neural networks, we propose Devign, a general graph neural network based model for graph-level classification through learning on a rich set of code semantic representations. It includes a novel Conv module to efficiently extract useful features in the learned rich node representations for graph-level classification. The model is trained over manually labeled datasets built on 4 diversified large-scale open-source C projects that incorporate high complexity and variety of real source code instead of synthesis code used in previous works. The results of the extensive evaluation on the datasets demonstrate that Devign outperforms the state of the arts significantly with an average of 10.51% higher accuracy and 8.68% F1 score, increases averagely 4.66% accuracy and 6.37% F1 by the Conv module.

Dataset information

We fine-tuned the models on the following datasets of bugs in C and C++ source code, which we also use later in our studies.
We chose these datasets because they are widely used and represent diverse dataset collection methods. Table {{table:c1a83708-e851-44e6-bfd7-df1f7880e173}}  lists the details of each dataset.
D2A {{cite:4ff05de}} used the Infer static analyzer and differential analysis to collect buggy and the corresponding fixed functions.
We used the function-level leaderboard datasethttps://developer.ibm.com/exchanges/data/all/d2a/, which is balanced.
Devign {{cite:a78f13f}} used commit filtering and further manual analysis to gather a balanced dataset of buggy and non-buggy functions.
We used the partitions released by the CodeXGLUE dataset {{cite:09c10c8}}.
Big-Vul {{cite:b955121}} crawled the Common Vulnerabilities and Exposures (CVE) database to collect an imbalanced dataset of buggy and non-buggy functions.
We used the partitions released by Fu et al. {{cite:cbae45b}}.
ReVeal {{cite:bc6542b}} crawled public bug repositories for Chrome and Debian to produce an imbalanced dataset of buggy/fixed functions and unrelated functions in the same commit, which are assumed non-vulnerable.


This Paper Is Included in the Proceedings of the 12th Usenix Symposium on Operating Systems Design and Implementation (osdi '16). Tensorflow: a System for Large-scale Machine Learning Tensorflow: a System for Large-scale Machine Learning

Abstract: TensorFlow is a machine learning system that operates at large scale and in heterogeneous environments. Tensor-Flow uses dataflow graphs to represent computation, shared state, and the operations that mutate that state. It maps the nodes of a dataflow graph across many machines in a cluster, and within a machine across multiple computational devices, including multicore CPUs, general-purpose GPUs, and custom-designed ASICs known as Tensor Processing Units (TPUs). This architecture gives flexibility to the application developer: whereas in previous " parameter server " designs the management of shared state is built into the system, TensorFlow enables developers to experiment with novel optimizations and training algorithms. TensorFlow supports a variety of applications, with a focus on training and inference on deep neural networks. Several Google services use TensorFlow in production , we have released it as an open-source project, and it has become widely used for machine learning research. In this paper, we describe the TensorFlow dataflow model and demonstrate the compelling performance that Tensor-Flow achieves for several real-world applications.

Caffe: Convolutional Architecture for Fast Feature Embedding

Abstract: Caffe provides multimedia scientists and practitioners with a clean and modifiable framework for state-of-the-art deep learning algorithms and a collection of reference models. The framework is a BSD-licensed C++ library with Python and MATLAB bindings for training and deploying general-purpose convolutional neural networks and other deep models efficiently on commodity architectures. Caffe fits industry and internet-scale media needs by CUDA GPU computation, processing over 40 million images a day on a single K40 or Titan GPU (approx 2 ms per image). By separating model representation from actual implementation, Caffe allows experimentation and seamless switching among platforms for ease of development and deployment from prototyping machines to cloud environments. Caffe is maintained and developed by the Berkeley Vision and Learning Center (BVLC) with the help of an active community of contributors on GitHub. It powers ongoing research projects, large-scale industrial applications, and startup prototypes in vision, speech, and multimedia.

MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems

Abstract: MXNet is a multi-language machine learning (ML) library to ease the development of ML algorithms, especially for deep neural networks. Embedded in the host language, it blends declarative symbolic expression with imperative tensor computation. It offers auto differentiation to derive gradients. MXNet is computation and memory efficient and runs on various heterogeneous systems, ranging from mobile devices to distributed GPU clusters. This paper describes both the API design and the system implementation of MXNet, and explains how embedding of both symbolic expression and tensor operation is handled in a unified fashion. Our preliminary experiments reveal promising results on large scale deep neural network applications using multiple GPU machines.

CNNdroid: GPU-Accelerated Execution of Trained Deep Convolutional Neural Networks on Android

Abstract: Many mobile applications running on smartphones and wearable devices would potentially benefit from the accuracy and scalability of deep CNN-based machine learning algorithms. However, performance and energy consumption limitations make the execution of such computationally intensive algorithms on mobile devices prohibitive. We present a GPU-accelerated library, dubbed CNNdroid, for execution of trained deep CNNs on Android-based mobile devices. Empirical evaluations show that CNNdroid achieves up to 60X speedup and 130X energy saving on current mobile devices. The CNNdroid open source library is available for download at https://github.com/ENCP/CNNdroid

PyTorch: An Imperative Style, High-Performance Deep Learning Library

Abstract: Deep learning frameworks have often focused on either usability or speed, but not both. PyTorch is a machine learning library that shows that these two goals are in fact compatible: it provides an imperative and Pythonic programming style that supports code as a model, makes debugging easy and is consistent with other popular scientific computing libraries, while remaining efficient and supporting hardware accelerators such as GPUs. In this paper, we detail the principles that drove the implementation of PyTorch and how they are reflected in its architecture. We emphasize that every aspect of PyTorch is a regular Python program under the full control of its user. We also explain how the careful and pragmatic implementation of the key components of its runtime enables them to work together to achieve compelling performance. We demonstrate the efficiency of individual subsystems, as well as the overall speed of PyTorch on several common benchmarks.

Mlpack 3: a Fast, Flexible Machine Learning Library

Abstract: In the past several years, the field of machine learning has seen an explosion of interest and excitement, with hundreds or thousands of algorithms developed for different tasks every year. But a primary problem faced by the field is the ability to scale to larger and larger data—since it is known that training on larger datasets typically produces better results (Halevy, Norvig, and Pereira 2009). Therefore, the development of new algorithms for the continued growth of the field depends largely on the existence of good tooling and libraries that enable researchers and practitioners to quickly prototype and develop solutions (Sonnenburg et al. 2007). Simultaneously, useful libraries must also be efficient and well-implemented. This has motivated our development of mlpack.

XGBoost: A Scalable Tree Boosting System

Abstract: Tree boosting is a highly effective and widely used machine learning method. In this paper, we describe a scalable end-to-end tree boosting system called XGBoost, which is used widely by data scientists to achieve state-of-the-art results on many machine learning challenges. We propose a novel sparsity-aware algorithm for sparse data and weighted quantile sketch for approximate tree learning. More importantly, we provide insights on cache access patterns, data compression and sharding to build a scalable tree boosting system. By combining these insights, XGBoost scales beyond billions of examples using far fewer resources than existing systems.

AI Frameworks

AI Discriminator for AI app identification

To alleviate the issues that some non-AI words contain simple AI keywords (e.g., LSTM and CNN), we remove those terms from the keyword dictionary. Furthermore, we collect AI service-related keywords based on the study of Xu et al. {{cite:d8d2ba4}}. We present the collected AI framework information in Table {{table:a16a3e4a-52c7-4b4f-b6f6-ac058a4fe559}} .


not present, all access to the page has been revoked

Read-only Bypass: bypasses read/write permission checks to transiently write over read-only data within the current privilege level. May be used, for example, to bypass the hardware-enforced isolation of software-based sandboxes.

Write Transient Forwarding (WTF): store buffer

FPU Register Bypass: bypasses isolation of floating point unit or SIMD registers across context switches, to leak register contents.

Speculative Buffer Overflows: Attacks and Defenses

Abstract:
Practical attacks that exploit speculative execution can leak confidential information via microarchitectural side channels. The recently-demonstrated Spectre attacks leverage speculative loads which circumvent access checks to read memory-resident secrets, transmitting them to an attacker using cache timing or other covert communication channels.
We introduce Spectre1.1, a new Spectre-v1 variant that leverages speculative stores to create speculative buffer overflows. Much like classic buffer overflows, speculative out-of-bounds stores can modify data and code pointers. Data-value attacks can bypass some Spectre-v1 mitigations, either directly or by redirecting control flow. Control-flow attacks enable arbitrary speculative code execution, which can bypass fence instructions and all other software mitigations for previous speculative-execution attacks. It is easy to construct return-oriented-programming (ROP) gadgets that can be used to build alternative attack payloads.
We also present Spectre1.2: on CPUs that do not enforce read/write protections, speculative stores can overwrite read-only data and code pointers to breach sandboxes.
We highlight new risks posed by these vulnerabilities, discuss possible software mitigations, and sketch microarchitectural mechanisms that could serve as hardware defenses. We have not yet evaluated the performance impact of our proposed software and hardware mitigations. We describe the salient vulnerability features and additional hypothetical attack scenarios only to the detail necessary to guide hardware and software vendors in threat analysis and mitigations. We advise users to refer to more user-friendly vendor recommendations for mitigations against speculative buffer overflows or available patches.

Fallout: Reading Kernel Writes From User Space

Abstract:
Recently, out-of-order execution, an important performance optimization in modern high-end processors, has been revealed to pose a significant security threat, allowing information leaks across security domains. In particular, the Meltdown attack leaks information from the operating system kernel to user space, completely eroding the security of the system. To address this and similar attacks, without incurring the performance costs of software countermeasures, Intel includes hardware-based defenses in its recent Coffee Lake R processors.
In this work, we show that the recent hardware defenses are not sufficient. Specifically, we present Fallout, a new transient execution attack that leaks information from a previously unexplored microarchitectural component called the store buffer. We show how unprivileged user processes can exploit Fallout to reconstruct privileged information recently written by the kernel. We further show how Fallout can be used to bypass kernel address space randomization. Finally, we identify and explore microcode assists as a hitherto ignored cause of transient execution.
Fallout affects all processor generations we have tested. However, we notice a worrying regression, where the newer Coffee Lake R processors are more vulnerable to Fallout than older generations.

LazyFP: Leaking FPU Register State using Microarchitectural Side-Channels

Abstract: Modern processors utilize an increasingly large register set to facilitate efficient floating point and SIMD computation. This large register set is a burden for operating systems, as its content needs to be saved and restored when the operating system context switches between tasks. As an optimization, the operating system can defer the context switch of the FPU and SIMD register set until the first instruction is executed that needs access to these registers. Meanwhile, the old content is left in place with the hope that the current task might not use these registers at all. This optimization is commonly called lazy FPU context switching. To make it possible, a processor offers the ability to toggle the availability of instructions utilizing floating point and SIMD registers. If the instructions are turned off, any attempt of executing them will generate a fault. In this paper, we present an attack that exploits lazy FPU context switching and allows an adversary to recover the FPU and SIMD register set of arbitrary processes or VMs. The attack works on processors that transiently execute FPU or SIMD instructions that follow an instruction generating the fault indicating the first use of FPU or SIMD instructions. On operating systems using lazy FPU context switching, the FPU and SIMD register content of other processes or virtual machines can then be reconstructed via cache side effects. With SIMD registers not only being used for cryptographic computation, but also increasingly for simple operations, such as copying memory, we argue that lazy FPU context switching is a dangerous optimization that needs to be turned off in all operating systems, if there is a chance that they run on affected processors.

Meltdown variants by exception

The primary way of categorizing Meltdown-type variants, shown in Table {{table:ad6fc9b8-f002-4d8f-884b-2e359574a1e3}} , is by the exception used in the preparation phase. A secondary way of categorizing Meltdown-type variants is by the microarchitectural states used in the transmission phase of attack—Table {{table:81f02b6c-f516-472c-8489-e216c4e4206e}}  shows some of the highlights. There have been fewer attempts to replicate Meltdown variants across a diverse collection of different side channels, because it quickly became clear that it was feasible to block Meltdown in the preparation and access phases of the attack, so the side channel used in the transmission phase is less interesting.

While AMD was not vulnerable to earlier variants of Meltdown, it was vulnerable to the Meltdown-BND variant {{cite:9174454}} in Table {{table:ad6fc9b8-f002-4d8f-884b-2e359574a1e3}}  and to new variants reported by Xiao et al. {{cite:0407b18}} in Table {{table:81f02b6c-f516-472c-8489-e216c4e4206e}} .


89% (line coverage), 90% (branch coverage)

Unit Test Case Generation with Transformers

Abstract: Automated Unit Test Case generation has been the focus of extensive literature within the research community. Existing approaches are usually guided by the test coverage criteria, generating synthetic test cases that are often difficult to read or understand for developers. In this paper we propose AthenaTest, an approach that aims at generating unit test cases by learning from real-world, developer-written test cases. Our approach relies on a state-of-the-art sequence-to-sequence transformer model which is able to write useful test cases for a given method under test (i.e., focal method). We also introduce methods2test - the largest publicly available supervised parallel corpus of unit test case methods and corresponding focal methods in Java, which comprises 630k test cases mined from 70k open-source repositories hosted on GitHub. We use this dataset to train a transformer model to translate focal methods into the corresponding test cases. We evaluate the ability of our model in generating test cases using natural language processing as well as code-specific criteria. First, we assess the quality of the translation compared to the target test case, then we analyze properties of the test case such as syntactic correctness and number and variety of testing APIs (e.g., asserts). We execute the test cases, collect test coverage information, and compare them with test cases generated by EvoSuite and GPT-3. Finally, we survey professional developers on their preference in terms of readability, understandability, and testing effectiveness of the generated test cases.

ChatUniTest: a ChatGPT-based automated unit test generation tool

Abstract: Unit testing is a crucial, yet often tedious and time-consuming task. To relieve developers from this burden, automated unit test generation techniques are developed. Existing automated unit test generation tools, such as program-analysis-based tools like EvoSuite and Randoop, lack program comprehension, resulting in unit tests with poor readability and limited assertions. Language-model-based tools, such as AthenaTest and A3Test, have limitations in the generation of correct unit tests. In this paper, we introduce ChatUniTest, a ChatGPT-based automated unit test generation tool developed under the Generation-Validation-Repair framework. ChatUniTest generates tests by parsing the project, extracting essential information, and creating an adaptive focal context that includes the focal method and its dependencies within the pre-defined maximum prompt token limit. The context is incorporated into a prompt and subsequently submitted to ChatGPT. Once ChatGPT's response is received, ChatUniTest proceeds to extract the raw test from the response. It then validates the test and employs rule-based repair to fix syntactic and simple compile errors, followed by ChatGPT-based repair to address challenging errors. Our rigorous evaluation demonstrates that ChatUniTest outperforms EvoSuite in branch and line coverage, surpasses AthenaTest and A3Test in focal method coverage, and effectively generates assertions while utilizing mock objects and reflection to achieve test objectives.

No More Manual Tests? Evaluating and Improving ChatGPT for Unit Test Generation

Abstract: Unit testing is essential in detecting bugs in functionally-discrete program units. Manually writing high-quality unit tests is time-consuming and laborious. Although traditional techniques can generate tests with reasonable coverage, they exhibit low readability and cannot be directly adopted by developers. Recent work has shown the large potential of large language models (LLMs) in unit test generation, which can generate more human-like and meaningful test code. ChatGPT, the latest LLM incorporating instruction tuning and reinforcement learning, has performed well in various domains. However, It remains unclear how effective ChatGPT is in unit test generation. In this work, we perform the first empirical study to evaluate ChatGPT's capability of unit test generation. Specifically, we conduct a quantitative analysis and a user study to systematically investigate the quality of its generated tests regarding the correctness, sufficiency, readability, and usability. The tests generated by ChatGPT still suffer from correctness issues, including diverse compilation errors and execution failures. Still, the passing tests generated by ChatGPT resemble manually-written tests by achieving comparable coverage, readability, and even sometimes developers' preference. Our findings indicate that generating unit tests with ChatGPT could be very promising if the correctness of its generated tests could be further improved. Inspired by our findings above, we propose ChatTESTER, a novel ChatGPT-based unit test generation approach, which leverages ChatGPT itself to improve the quality of its generated tests. ChatTESTER incorporates an initial test generator and an iterative test refiner. Our evaluation demonstrates the effectiveness of ChatTESTER by generating 34.3% more compilable tests and 18.7% more tests with correct assertions than the default ChatGPT.

Performance of unit test case generation.
Since the aforementioned studies of unit test case generation are based different datasets, one can hardly derive a fair comparison and we present the details in Table {{table:3738a277-164c-4805-97a5-4537c3adb53d}}  to let the readers obtain a general view.
We can see that in SF110 benchmark, all three evaluated LLMs have quite low performance, i.e., 2% coverage {{cite:2c0eab8}}.
This SF110 dataset, which is an Evosuite (a search-based unit test case generation technique) benchmark consisting of 111 open-source Java projects retrieved from SourceForge, contains 23,886 classes, over 800,000 bytecode-level branches, and 6.6 million lines of code.
The authors did not present the detailed reasons for the low performance, and can be further explored in future.

More Solid Benchmarks and Rigorous Evaluations

The second is the limited number of benchmarks.
For unit test case generation, there is even no widely recognized benchmarks, and different studies would utilize different datasets for performance evaluation, as demonstrated in Table {{table:3738a277-164c-4805-97a5-4537c3adb53d}} .
For program repair, there are only two well-known and commonly-used benchmarks, i.e., Defect4J and QuixBugs, as demonstrated in Table {{table:76a8eda9-3745-4e5e-bcb4-53ac2425dba7}} .
Furthermore, these datasets are not specially designed for testing the LLMs.
For example, as reported in {{cite:2de43d4}}, 39 out of 40 Python bugs in QuixBugs dataset can be fixed by Codex, yet in real-world practice, the successful fix rate can be nowhere near as high.
This motivates to build more specialized and diversified benchmark as mentioned in the first limitation.

The third is the performance variation across datasets.
This is not only the problem for LLMs, but also for traditional machine learning and deep learning models.
For example for unit test case generation, as shown in Table {{table:3738a277-164c-4805-97a5-4537c3adb53d}} , with the same technique {{cite:2c0eab8}}, on HumanEval dataset it can generate 78% correct unit test cases, and the line coverage is 87%, yet on SF110 dataset, both the correctness of generated tests and the line coverage is merely 2%.
The performance variation among different datasets indicates the fragile of current unit test case generation techniques.
This performance variation is also frequently observed in traditional machine learning tasks, e.g., in defect prediction {{cite:6bad4c2}}, {{cite:44ac32c}}.
Typically solution evolves more reliable benchmark and extension comparison as mentioned in the first and second limitations.


Introduction to the CoNLL-2003 Shared Task: Language-Independent Named Entity Recognition

Abstract: We describe the CoNLL-2003 shared task: language-independent named entity recognition. We give background information on the data sets (English and German) and the evaluation method, present a general overview of the systems that have taken part in the task and discuss their performance.

A Finnish News Corpus for Named Entity Recognition

Abstract: We present a corpus of Finnish news articles with a manually prepared named entity annotation. The corpus consists of 953 articles (193,742 word tokens) with six named entity classes (organization, location, person, product, event, and date). The articles are extracted from the archives of Digitoday, a Finnish online technology news source. The corpus is available for research purposes. We present baseline experiments on the corpus using a rule-based and two deep learning systems on two, in-domain and out-of-domain, test sets.

The collected datasets for the NER task and their properties: number of sentences, number of tagged words, availability, and link to the corpus location.

Named entity recognition (NER) is an information extraction task that seeks to locate and classify named entities mentioned in unstructured text into pre-defined categories such as the person names, organizations, locations, medical codes, time expressions, quantities, monetary values, etc. The labels in the used NER datasets are simplified to a common label set of four labels present in all the addressed languages. These labels are "person", "location", "organization", and "other". The latter encompasses all named entities that do not fall in one of the three mentioned classes and all the tokens that are not named entities.
The datasets used in the evaluation on the NER task are shown in Table {{table:6439708b-4fda-4df4-8610-96f9aa2a7927}} , along with some basic statistics of the datasets.


Han's conjecture and Hochschild homology for null-square projective algebras

Abstract: Let $\mathcal H$ be the class of algebras verifying Han's conjecture. In this paper we analyse two types of algebras with the aim of providing an inductive step towards the proof of this conjecture. Firstly we show that if an algebra $\Lambda$ is triangular with respect to a system of non necessarily primitive idempotents, and if the algebras at the idempotents belong to $\mathcal H$, then $\Lambda$ is in $\mathcal H$. Secondly we consider a $2\times 2$ matrix algebra, with two algebras on the diagonal, two projective bimodules in the corners, and zero corner products. They are not triangular with respect to the system of the two diagonal idempotents. However, the analogous result holds, namely if both algebras on the diagonal belong to $\mathcal H$, then the algebra itself is in $\mathcal H$.

Homological properties of extensions of abstract and pseudocompact algebras

Abstract: We consider a class of extensions of both abstract and pseudocompact algebras, which we refer to as “strongly proj-bounded extensions”. We prove that the ﬁniteness of the left global dimension and the support of the Hochschild homology is preserved by strongly proj-bounded extensions, generalizing results of Cibils, Lanzillota, Marcos and Solotar. Moreover, we show that the ﬁniteness of the big left ﬁnitistic dimension is preserved by strongly proj-bounded extensions. In order to construct examples, we describe a new class of extensions of algebras of ﬁnite relative global dimension, which may be of independent interest.

Extensions of finite-dimensional algebras which preserves Han's property. The list is organized in chronological order of the references.

Preservation of Han's property by Extensions

Recently, some authors gave contributions for the understanding of Han's conjecture in a distinct way from above. Having in mind, for example, a possible inductive step in order to prove the conjecture, many efforts were given in the following direction: encountering extensions of algebras that preserves Han's property, i.e. pairs of algebras {{formula:9a3e46e6-7ee5-45c2-9ad0-d4cb85f22bdd}}  such that, if {{formula:52bb182b-2cf3-4b8a-b045-1cea39fb2a06}}  satisfy Han's property, then {{formula:15793142-3c4c-41c2-bea5-dc3ea96c070a}}  also satisfy it. We summarize these in table {{table:566a9f05-0f0f-477e-ae4e-20c1ad12eb30}} . For instance, with such results, one can construct from the previous examples many other algebras satisfying Han's property.


Differentially Private Mixture of Generative Neural Networks

Abstract: Generative models are used in a wide range of applications building on large amounts of contextually rich information. Due to possible privacy violations of the individuals whose data is used to train these models, however, publishing or sharing generative models is not always viable. In this paper, we present a novel technique for privately releasing generative models and entire high-dimensional datasets produced by these models. We model the generator distribution of the training data with a mixture of $k$ generative neural networks. These are trained together and collectively learn the generator distribution of a dataset. Data is divided into $k$ clusters, using a novel differentially private kernel $k$-means, then each cluster is given to separate generative neural networks, such as Restricted Boltzmann Machines or Variational Autoencoders, which are trained only on their own cluster using differentially private gradient descent. We evaluate our approach using the MNIST dataset, as well as call detail records and transit datasets, showing that it produces realistic synthetic samples, which can also be used to accurately compute arbitrary number of counting queries.

GS-WGAN: A Gradient-Sanitized Approach for Learning Differentially Private Generators

Abstract: The wide-spread availability of rich data has fueled the growth of machine learning applications in numerous domains. However, growth in domains with highly-sensitive data (e.g., medical) is largely hindered as the private nature of data prohibits it from being shared. To this end, we propose Gradient-sanitized Wasserstein Generative Adversarial Networks (GS-WGAN), which allows releasing a sanitized form of the sensitive data with rigorous privacy guarantees. In contrast to prior work, our approach is able to distort gradient information more precisely, and thereby enabling training deeper models which generate more informative samples. Moreover, our formulation naturally allows for training GANs in both centralized and federated (i.e., decentralized) data scenarios. Through extensive experiments, we find our approach consistently outperforms state-of-the-art approaches across multiple metrics (e.g., sample quality) and datasets.

Differentially Private Synthetic Medical Data Generation using Convolutional GANs

Abstract: Deep learning models have demonstrated superior performance in several application problems, such as image classification and speech processing. However, creating a deep learning model using health record data requires addressing certain privacy challenges that bring unique concerns to researchers working in this domain. One effective way to handle such private data issues is to generate realistic synthetic data that can provide practically acceptable data quality and correspondingly the model performance. To tackle this challenge, we develop a differentially private framework for synthetic data generation using R\'enyi differential privacy. Our approach builds on convolutional autoencoders and convolutional generative adversarial networks to preserve some of the critical characteristics of the generated synthetic data. In addition, our model can also capture the temporal information and feature correlations that might be present in the original data. We demonstrate that our model outperforms existing state-of-the-art models under the same privacy budget using several publicly available benchmark medical datasets in both supervised and unsupervised settings.

Summary of differentially private generative models. The methods are either based on (variational) autoencoders or Generative Adversarial Networks (GAN). The DP algorithm DP-EM refers to differentially private expectation maximization {{cite:559f823}}.

Differentially private generative models

Generative models are a class of machine learning models that aim to generate new data samples similar to the data samples from the training set. The synthetic data created with generative models are often seen as privacy-preserving as they are not directly linked to real entities or individuals. However, similarly to other machine learning models, generative models can memorize sensitive information and be vulnerable to privacy attacks. This section gives an overview of recent works regarding the generation of differentially private synthetic data with generative models (see Table {{table:c2900d33-e3fb-40dd-ae21-35c9e444a1f2}} ).


FedCurv uses the EWC algorithm to prevent catastrophic forgetting when transferring tasks.

pFedME utilizes the Moreau envelope function as a regularized loss function.

Similarities between the MAML setting and the personalized objectives of HFL are pointed out.

A system-aware optimization framework for FMTL is built.

It may ignore the differences in the extent to which clients are affected by catastrophic forgetting.

Tuning the regularization parameters may be labour intensive.

The two stages of meta-training and meta-testing may introduce additional communication overhead.

It cannot be applied to non-convex deep learning models.

Overcoming Forgetting in Federated Learning on Non-IID Data

Abstract: We tackle the problem of Federated Learning in the non i.i.d. case, in which local models drift apart, inhibiting learning. Building on an analogy with Lifelong Learning, we adapt a solution for catastrophic forgetting to Federated Learning. We add a penalty term to the loss function, compelling all local models to converge to a shared optimum. We show that this can be done efficiently for communication (adding no further privacy risks), scaling with the number of nodes in the distributed setting. Our experiments show that this method is superior to competing ones for image recognition on the MNIST dataset.

Personalized Federated Learning with Moreau Envelopes

Abstract: Federated learning (FL) is a decentralized and privacy-preserving machine learning technique in which a group of clients collaborate with a server to learn a global model without sharing clients' data. One challenge associated with FL is statistical diversity among clients, which restricts the global model from delivering good performance on each client's task. To address this, we propose an algorithm for personalized FL (pFedMe) using Moreau envelopes as clients' regularized loss functions, which help decouple personalized model optimization from the global model learning in a bi-level problem stylized for personalized FL. Theoretically, we show that pFedMe's convergence rate is state-of-the-art: achieving quadratic speedup for strongly convex and sublinear speedup of order 2/3 for smooth nonconvex objectives. Experimentally, we verify that pFedMe excels at empirical performance compared with the vanilla FedAvg and Per-FedAvg, a meta-learning based personalized FL algorithm.

Improving Federated Learning Personalization via Model Agnostic Meta Learning

Abstract: Federated Learning (FL) refers to learning a high quality global model based on decentralized data storage, without ever copying the raw data. A natural scenario arises with data created on mobile phones by the activity of their users. Given the typical data heterogeneity in such situations, it is natural to ask how can the global model be personalized for every such device, individually. In this work, we point out that the setting of Model Agnostic Meta Learning (MAML), where one optimizes for a fast, gradient-based, few-shot adaptation to a heterogeneous distribution of tasks, has a number of similarities with the objective of personalization for FL. We present FL as a natural source of practical applications for MAML algorithms, and make the following observations. 1) The popular FL algorithm, Federated Averaging, can be interpreted as a meta learning algorithm. 2) Careful fine-tuning can yield a global model with higher accuracy, which is at the same time easier to personalize. However, solely optimizing for the global model accuracy yields a weaker personalization result. 3) A model trained using a standard datacenter optimization method is much harder to personalize, compared to one trained using Federated Averaging, supporting the first claim. These results raise new questions for FL, MAML, and broader ML research.

Federated Multi-Task Learning

Abstract: Federated learning poses new statistical and systems challenges in training machine learning models over distributed networks of devices. In this work, we show that multi-task learning is naturally suited to handle the statistical challenges of this setting, and propose a novel systems-aware optimization method, MOCHA, that is robust to practical systems issues. Our method and theory for the first time consider issues of high communication cost, stragglers, and fault tolerance for distributed multi-task learning. The resulting method achieves significant speedups compared to alternatives in the federated setting, as we demonstrate through simulations on real-world federated datasets.

Parameterized Knowledge Transfer for Personalized Federated Learning

Abstract: In recent years, personalized federated learning (pFL) has attracted increasing attention for its potential in dealing with statistical heterogeneity among clients. However, the state-of-the-art pFL methods rely on model parameters aggregation at the server side, which require all models to have the same structure and size, and thus limits the application for more heterogeneous scenarios. To deal with such model constraints, we exploit the potentials of heterogeneous model settings and propose a novel training framework to employ personalized models for different clients. Specifically, we formulate the aggregation procedure in original pFL into a personalized group knowledge transfer training algorithm, namely, KT-pFL, which enables each client to maintain a personalized soft prediction at the server side to guide the others' local training. KT-pFL updates the personalized soft prediction of each client by a linear combination of all local soft predictions using a knowledge coefficient matrix, which can adaptively reinforce the collaboration among clients who own similar data distribution. Furthermore, to quantify the contributions of each client to others' personalized training, the knowledge coefficient matrix is parameterized so that it can be trained simultaneously with the models. The knowledge coefficient matrix and the model parameters are alternatively updated in each round following the gradient descent way. Extensive experiments on various datasets (EMNIST, Fashion\_MNIST, CIFAR-10) are conducted under different settings (heterogeneous models and data distributions). It is demonstrated that the proposed framework is the first federated learning paradigm that realizes personalized model training via parameterized group knowledge transfer while achieving significant performance gain comparing with state-of-the-art algorithms.

Federated Learning with Personalization Layers

Abstract: The emerging paradigm of federated learning strives to enable collaborative training of machine learning models on the network edge without centrally aggregating raw data and hence, improving data privacy. This sharply deviates from traditional machine learning and necessitates the design of algorithms robust to various sources of heterogeneity. Specifically, statistical heterogeneity of data across user devices can severely degrade the performance of standard federated averaging for traditional machine learning applications like personalization with deep learning. This paper pro-posesFedPer, a base + personalization layer approach for federated training of deep feedforward neural networks, which can combat the ill-effects of statistical heterogeneity. We demonstrate effectiveness ofFedPerfor non-identical data partitions ofCIFARdatasetsand on a personalized image aesthetics dataset from Flickr.

Heterogeneous Federated Learning: State-of-the-art and Research Challenges

Abstract: Federated learning (FL) has drawn increasing attention owing to its potential use in large-scale industrial applications. Existing FL works mainly focus on model homogeneous settings. However, practical FL typically faces the heterogeneity of data distributions, model architectures, network environments, and hardware devices among participant clients. Heterogeneous Federated Learning (HFL) is much more challenging, and corresponding solutions are diverse and complex. Therefore, a systematic survey on this topic about the research challenges and state-of-the-art is essential. In this survey, we firstly summarize the various research challenges in HFL from five aspects: statistical heterogeneity, model heterogeneity, communication heterogeneity, device heterogeneity, and additional challenges. In addition, recent advances in HFL are reviewed and a new taxonomy of existing HFL methods is proposed with an in-depth analysis of their pros and cons. We classify existing methods from three different levels according to the HFL procedure: data-level, model-level, and server-level. Finally, several critical and promising future research directions in HFL are discussed, which may facilitate further developments in this field. A periodically updated collection on HFL is available at https://github.com/marswhu/HFL_Survey.

Model-level methods.

blackIn this subsection, we categorize model-level methods, introduce representative methods in each category, and discuss their contributions and limitations, as shown in Tab. {{table:f42bd3f4-f0b3-4b06-8128-3757f41e2f97}} . Model-level methods represent methods for innovative design at the model level, mainly including federated optimization, knowledge transfer across models, and architecture sharing. Federated optimization aims to adapt the model to the local distribution while learning the global information. They can effectively realize local model personalization under statistical heterogeneity. Knowledge transfer across models enables multi-party collaboration in a model-agnostic manner and thus is usually used to solve model and communication heterogeneity. Architecture sharing realizes personalized federated learning by sharing part of the model structure and can solve statistical, model and device heterogeneity simultaneously to some extent. The data-level methods can solve the problem associated with large differences in data distribution to a certain extent and accelerate convergence by smoothing the statistical heterogeneity among data.
In contrast, the model-level methods aim to learn a local model for each client that adapts to its private data distribution, and thus such methods have been extensively researched.


Character-level Convolutional Networks for Text Classification

Abstract: This article offers an empirical exploration on the use of character-level convolutional networks (ConvNets) for text classification. We constructed several large-scale datasets to show that character-level convolutional networks could achieve state-of-the-art or competitive results. Comparisons are offered against traditional models such as bag of words, n-grams and their TFIDF variants, and deep learning models such as word-based ConvNets and recurrent neural networks.

A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference

Abstract: This paper introduces the Multi-Genre Natural Language Inference (MultiNLI) corpus, a dataset designed for use in the development and evaluation of machine learning models for sentence understanding. At 433k examples, this resource is one of the largest corpora available for natural language inference (a.k.a. recognizing textual entailment), improving upon available resources in both its coverage and difficulty. MultiNLI accomplishes this by offering data from ten distinct genres of written and spoken English, making it possible to evaluate systems on nearly the full complexity of the language, while supplying an explicit setting for evaluating cross-genre domain adaptation. In addition, an evaluation using existing machine learning models designed for the Stanford NLI corpus shows that it represents a substantially more difficult task than does that corpus, despite the two showing similar levels of inter-annotator agreement.

Evaluation datasets.Label quantity of each class follows prior work {{cite:4306b7f}} where {{formula:63dd8f84-f663-4815-8204-062e308766c1}} .

Dataset and few-shot setting
We experiment with four popular NLP datasets and promptsWe attempt 6, 2, 6, 4 different prompts that are widely used in prior work for each dataset, and use the one with highest accuracy.
The verbalizers are the same as the previous literature {{cite:29d7fd3}}.,
as shown in Table {{table:4fc443cb-9dbc-4a0d-b981-3fcda3d432fa}} .
(1) AGNEWS {{cite:d0d73cc}} is a news classification dataset. Given headline and text body, news needs to be classified into one of the four categories: World, Sports, Business or Science/Tech.
(2) MNLI {{cite:397aabe}} is a sentence understanding dataset. Given text pairs x = (a, b), the task is to find out whether a implies b, a and b contradict each other, or neither.
(3) YELP {{cite:d0d73cc}} is a restaurant rating dataset. Given a customer's review, text should be estimated on a 1-5 star scale.
(4) YAHOO {{cite:d0d73cc}} is a text classification dataset. Given a question a and an answer b, one of ten possible categories needs to be assigned.
For each dataset, we follow prior work {{cite:380aaeb}}, {{cite:4a138c3}} to randomly select gold labels.
By default, the labels form a skewed distribution across clients to be more realistic to real-world situation as discussed in {{formula:2cade633-d961-4413-8f9e-a305ff4a6136}}REF .
For each dataset, we generate 3 different few-shot settings, on which we repeat the experiments and report the mean results.


The Joint Automated Repository for Various Integrated Simulations (JARVIS) for data-driven materials design

Abstract: The Joint Automated Repository for Various Integrated Simulations (JARVIS) is an integrated infrastructure to accelerate materials discovery and design using density functional theory (DFT), classical force-fields (FF), and machine learning (ML) techniques. JARVIS is motivated by the Materials Genome Initiative (MGI) principles of developing open-access databases and tools to reduce the cost and development time of materials discovery, optimization, and deployment. The major features of JARVIS are: JARVIS-DFT, JARVIS-FF, JARVIS-ML, and JARVIS-Tools. To date, JARVIS consists of 40,000 materials and 1 million calculated properties in JARVIS-DFT, 1,500 materials and 110 force-fields in JARVIS-FF, and 25 ML models for material-property predictions in JARVIS-ML, all of which are continuously expanding. JARVIS-Tools provides scripts and workflows for running and analyzing various simulations. We compare our computational data to experiments or high-fidelity computational methods wherever applicable to evaluate error/uncertainty in predictions. In addition to the existing workflows, the infrastructure can support a wide variety of other technologically important applications as part of the data-driven materials design paradigm. The JARVIS datasets and tools are publicly available at the website: https://jarvis.nist.gov .

Atomistic Line Graph Neural Network for Improved Materials Property Predictions

Abstract: Graph neural networks (GNN) have been shown to provide substantial performance improvements for atomistic material representation and modeling compared with descriptor-based machine learning models. While most existing GNN models for atomistic predictions are based on atomic distance information, they do not explicitly incorporate bond angles, which are critical for distinguishing many atomic structures. Furthermore, many material properties are known to be sensitive to slight changes in bond angles. We present an Atomistic Line Graph Neural Network (ALIGNN), a GNN architecture that performs message passing on both the interatomic bond graph and its line graph corresponding to bond angles. We demonstrate that angle information can be explicitly and efficiently included, leading to improved performance on multiple atomistic prediction tasks. We ALIGNN models for predicting 52 solid-state and molecular properties available in the JARVIS-DFT, Materials project, and QM9 databases. ALIGNN can outperform some previously reported GNN models on atomistic prediction tasks by up to 85% in accuracy with better or comparable model training speed.

AtomVision: A machine vision library for atomistic images

Abstract: Computer vision techniques have immense potential for materials design applications. In this work, we introduce an integrated and general-purpose AtomVision library that can be used to generate, curate scanning tunneling microscopy (STM) and scanning transmission electron microscopy (STEM) datasets and apply machine learning techniques. To demonstrate the applicability of this library, we 1) generate and curate an atomistic image dataset of about 10000 materials, 2) develop and compare convolutional and graph neural network models to classify the Bravais lattices, 3) develop fully convolutional neural network using U-Net architecture to pixelwise classify atom vs background, 4) use generative adversarial network for super-resolution, 5) curate a natural language processing based image dataset using open-access arXiv dataset, and 6) integrate the computational framework with experimental microscopy tools. AtomVision library is available at https://github.com/usnistgov/atomvision.

ChemNLP: A Natural Language-Processing-Based Library for Materials Chemistry Text Data

Abstract: In this work, we present the ChemNLP library that can be used for 1) curating open access datasets for materials and chemistry literature, developing and comparing traditional machine learning, transformers and graph neural network models for 2) classifying and clustering texts, 3) named entity recognition for large-scale text-mining, 4) abstractive summarization for generating titles of articles from abstracts, 5) text generation for suggesting abstracts from titles, 6) integration with density functional theory dataset for identifying potential candidate materials such as superconductors, and 7) web-interface development for text and reference query. We primarily use the publicly available arXiv and Pubchem datasets but the tools can be used for other datasets as well. Moreover, as new models are developed, they can be easily integrated in the library. ChemNLP is available at the websites: https://github.com/usnistgov/chemnlp and https://jarvis.nist.gov/jarvischemnlp.

Quantum computation for predicting electron and phonon properties of solids

Abstract: Quantum chemistry is one of the most promising near-term applications of quantum computers. Quantum algorithms such as variational quantum eigen solver (VQE) and variational quantum deflation (VQD) algorithms have been mainly applied for molecular systems and there is a need to implement such methods for periodic solids. Using Wannier tight-binding Hamiltonian (WTBH) approaches, we demonstrate the application of VQE and VQD to accurately predict both electronic and phonon bandstructure properties of several elemental as well as multi-component solid-state materials. We apply VQE–VQD calculations for 307 spin–orbit coupling based electronic WTBHs and 933 finite-difference based phonon WTBHs. Also, we discuss a workflow for using VQD with lattice Green’s function that can be used for solving dynamical mean-field theory problems. The WTBH model solvers can be used for testing other quantum algorithms and models also.

A summary of the publicly available JARVIS tools.

The Joint Automated Repository for Various Integrated Simulations (JARVIS){{cite:32ae0f1}} is an integrated infrastructure to accelerate materials
discovery and design. The JARVIS infrastructure can be separated into electronic structure methods (density functional theory (DFT) {{cite:2943077}}, tight binding {{cite:eb6b1d4}}, dynamical mean field theory (DMFT) {{cite:48e08b0}}, many-body perturbation theory (GW) {{cite:7f0fffe}}, and Quantum Monte Carlo (QMC) {{cite:721e0b0}}) {{cite:ddad05c}}, classical force-fields (FF) {{cite:31bb092}}, machine learning (ML) techniques {{cite:6233d2d}}, quantum computation algorithms {{cite:bf46193}} and experiments {{cite:b243723}}. JARVIS is motivated by the Materials Genome Initiative (MGI) {{cite:b5141ec}} principles of developing open-access databases and tools to reduce the cost and development time of materials discovery, optimization, and deployment. A depiction of the major areas of ongoing research as part of the JARVIS infrastructure is depicted in Fig. {{figure:ce4ef9ec-3e1a-4822-92d0-f23e80e448eb}}  and the publicly available JARVIS tools are listed in Table {{table:523a023e-b9e0-4d91-a3ff-a28f3b4f40d1}} .


Using clustering methods or deep learning-based methods

Fitting 3D primitives or deep learning methods

Fully convolutional networks for semantic segmentation

Abstract: Convolutional networks are powerful visual models that yield hierarchies of features. We show that convolutional networks by themselves, trained end-to-end, pixels-to-pixels, exceed the state-of-the-art in semantic segmentation. Our key insight is to build “fully convolutional” networks that take input of arbitrary size and produce correspondingly-sized output with efficient inference and learning. We define and detail the space of fully convolutional networks, explain their application to spatially dense prediction tasks, and draw connections to prior models. We adapt contemporary classification networks (AlexNet [20], the VGG net [31], and GoogLeNet [32]) into fully convolutional networks and transfer their learned representations by fine-tuning [3] to the segmentation task. We then define a skip architecture that combines semantic information from a deep, coarse layer with appearance information from a shallow, fine layer to produce accurate and detailed segmentations. Our fully convolutional network achieves state-of-the-art segmentation of PASCAL VOC (20% relative improvement to 62.2% mean IU on 2012), NYUDv2, and SIFT Flow, while inference takes less than one fifth of a second for a typical image.

PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation

Abstract: Point cloud is an important type of geometric data structure. Due to its irregular format, most researchers transform such data to regular 3D voxel grids or collections of images. This, however, renders data unnecessarily voluminous and causes issues. In this paper, we design a novel type of neural network that directly consumes point clouds, which well respects the permutation invariance of points in the input. Our network, named PointNet, provides a unified architecture for applications ranging from object classification, part segmentation, to scene semantic parsing. Though simple, PointNet is highly efficient and effective. Empirically, it shows strong performance on par or even better than state of the art. Theoretically, we provide analysis towards understanding of what the network has learnt and why the network is robust with respect to input perturbation and corruption.

Summary of object localization methods.

Most grasping approaches require the computation of the target object's location in the input image data first. This involves object detection and segmentation techniques. Object detection provides the rectangular bounding box of the target object, and object segmentation provides the precise boundary of the target object. The latter provides more accurate descriptions of the object area, while its computation is more time consuming. The representative works of the two methods are shown in Table {{table:ad925c44-7ea9-4e4c-b9da-c4c11381abe0}} .


Low complexity

Power efficient

Low cost

Economic Loss due to replacement of existing equipment

Reconfigurable 3-D Slot Antenna Design for 4G and Sub-6G Smartphones with Metallic Casing

Abstract: The design of a reconfigurable three-dimensional (3-D) slot antenna for 4G and sub-6G smartphone application is presented in this paper. The antenna is located at the bottom of the smartphone and integrated with a metallic casing. Positive-Intrinsic-Negative (PIN) diodes are loaded at the dual-open slot and the folded U-shaped slot, respectively, which are used to realize four working states. The antenna has a compact volume of 42 × 6 × 6 mm3, which can cover the long term evolution (LTE) bands of 698–960 MHz and 1710–2690 MHz, and the sub-6G bands of 3300–3600 MHz & 4800–5000 MHz. The design processes are presented and the structure is optimized, fabricated and measured. The comparison to other state-of-the-art antennas shows that the proposed design has multiband characteristics with small size.

Cooperative Non-Orthogonal Multiple Access in 5G Systems

Abstract: Non-orthogonal multiple access (NOMA) has recently received considerable attention as a promising candidate for 5G systems. A key feature of NOMA is that users with better channel conditions have prior information about the messages of the other users. This prior knowledge is fully exploited in this paper, where a cooperative NOMA scheme is proposed. Outage probability and diversity order achieved by this cooperative NOMA scheme are analyzed, and an approach based on user pairing is also proposed to reduce system complexity in practice.

A critical analyses of different techniques proposed for B5G/6G systems.

Table {{table:9f67cd5f-e31b-452f-a5dd-e6a6c65bb4cd}}  gives a critical analysis of the potential 6G technologies, which will enable communications in the B5G/6G era. Advantages and disadvantages along with the research initiative in these technologies are also described.


Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales

Abstract: We address the rating-inference problem, wherein rather than simply decide whether a review is "thumbs up" or "thumbs down", as in previous sentiment analysis work, one must determine an author's evaluation with respect to a multi-point scale (e.g., one to five "stars"). This task represents an interesting twist on standard multi-class text categorization because there are several different degrees of similarity between class labels; for example, "three stars" is intuitively closer to "four stars" than to "one star". We first evaluate human performance at the task. Then, we apply a meta-algorithm, based on a metric labeling formulation of the problem, that alters a given n-ary classifier's output in an explicit attempt to ensure that similar items receive similar labels. We show that the meta-algorithm can provide significant improvements over both multi-class and regression versions of SVMs when we employ a novel similarity measure appropriate to the problem.

A Sentimental Education: Sentiment Analysis Using Subjectivity Summarization Based on Minimum Cuts

Abstract: Sentiment analysis seeks to identify the viewpoint(s) underlying a text span; an example application is classifying a movie review as "thumbs up" or "thumbs down". To determine this sentiment polarity, we propose a novel machine-learning method that applies text-categorization techniques to just the subjective portions of the document. Extracting these portions can be implemented using efficient techniques for finding minimum cuts in graphs; this greatly facilitates incorporation of cross-sentence contextual constraints.

SentEval Transfer Task datasets used for our experiments.

The datasets on which we tested the models come from the well-known SentEval Transfer Task, which includes classification and textual entailment tasks {{cite:2ab748b}}, giving a fair picture of overall model performance (see Table {{table:979de858-275f-4710-ad15-006cab68f35d}} ).


TIE: A Framework for Embedding-based Incremental Temporal Knowledge Graph Completion

Abstract: Reasoning in a temporal knowledge graph (TKG) is a critical task for information retrieval and semantic search. It is particularly challenging when the TKG is updated frequently. The model has to adapt to changes in the TKG for efficient training and inference while preserving its performance on historical knowledge. Recent work approaches TKG completion (TKGC) by augmenting the encoder-decoder framework with a time-aware encoding function. However, naively fine-tuning the model at every time step using these methods does not address the problems of 1) catastrophic forgetting, 2) the model's inability to identify the change of facts (e.g., the change of the political affiliation and end of a marriage), and 3) the lack of training efficiency. To address these challenges, we present the Time-aware Incremental Embedding (TIE) framework, which combines TKG representation learning, experience replay, and temporal regularization. We introduce a set of metrics that characterizes the intransigence of the model and propose a constraint that associates the deleted facts with negative labels. Experimental results on Wikidata12k and YAGO11k datasets demonstrate that the proposed TIE framework reduces training time by about ten times and improves on the proposed metrics compared to vanilla full-batch training. It comes without a significant loss in performance for any traditional measures. Extensive ablation studies reveal performance trade-offs among different evaluation metrics, which is essential for decision-making around real-world TKG applications.

Learning Sequence Encoders for Temporal Knowledge Graph Completion

Abstract: Research on link prediction in knowledge graphs has mainly focused on static multi-relational data. In this work we consider temporal knowledge graphs where relations between entities may only hold for a time interval or a specific point in time. In line with previous work on static knowledge graphs, we propose to address this problem by learning latent entity and relation type representations. To incorporate temporal information, we utilize recurrent neural networks to learn time-aware representations of relation types which can be used in conjunction with existing latent factorization methods. The proposed approach is shown to be robust to common challenges in real-world KGs: the sparsity and heterogeneity of temporal expressions. Experiments show the benefits of our approach on four temporal KGs. The data sets are available under a permissive BSD-3 license 1.

TempoQR: Temporal Question Reasoning over Knowledge Graphs

Abstract: Knowledge Graph Question Answering (KGQA) involves retrieving facts from a Knowledge Graph (KG) using natural language queries. A KG is a curated set of facts consisting of entities linked by relations. Certain facts include also temporal information forming a Temporal KG (TKG). Although many natural questions involve explicit or implicit time constraints, question answering (QA) over TKGs has been a relatively unexplored area. Existing solutions are mainly designed for simple temporal questions that can be answered directly by a single TKG fact. This paper puts forth a comprehensive embedding-based framework for answering complex questions over TKGs. Our method termed temporal question reasoning (TempoQR) exploits TKG embeddings to ground the question to the specific entities and time scope it refers to. It does so by augmenting the question embeddings with context, entity and time-aware information by employing three specialized modules. The first computes a textual representation of a given question, the second combines it with the entity embeddings for entities involved in the question, and the third generates question-specific time embeddings. Finally, a transformer-based encoder learns to fuse the generated temporal information with the question representation, which is used for answer predictions. Extensive experiments show that TempoQR improves accuracy by 25--45 percentage points on complex temporal questions over state-of-the-art approaches and it generalizes better to unseen question types.

Typical benchmark datasets for temporal knowledge graph reasoning.

Eighteen Typical temporal KGR datasets are summarized. The statistic is presented in Table {{table:e929ed7f-af5c-4129-a523-2de66bb7f4d7}} , and descriptions are listed below.


Price DOES Matter!: Modeling Price and Interest Preferences in Session-based Recommendation

Abstract: Session-based recommendation aims to predict items that an anonymous user would like to purchase based on her short behavior sequence. The current approaches towards session-based recommendation only focus on modeling users' interest preferences, while they all ignore a key attribute of an item, i.e., the price. Many marketing studies have shown that the price factor significantly influences users' behaviors and the purchase decisions of users are determined by both price and interest preferences simultaneously. However, it is nontrivial to incorporate price preferences for session-based recommendation. Firstly, it is hard to handle heterogeneous information from various features of items to capture users' price preferences. Secondly, it is difficult to model the complex relations between price and interest preferences in determining user choices. To address the above challenges, we propose a novel method Co-guided Heterogeneous Hypergraph Network (CoHHN) for session-based recommendation. Towards the first challenge, we devise a heterogeneous hypergraph to represent heterogeneous information and rich relations among them. A dual-channel aggregating mechanism is then designed to aggregate various information in the heterogeneous hypergraph. After that, we extract users' price preferences and interest preferences via attention layers. As to the second challenge, a co-guided learning scheme is designed to model the relations between price and interest preferences and enhance the learning of each other. Finally, we predict user actions based on item features and users' price and interest preferences. Extensive experiments on three real-world datasets demonstrate the effectiveness of the proposed CoHHN. Further analysis reveals the significance of price for session-based recommendation.

Towards Content Provider Aware Recommender Systems: A Simulation Study on the Interplay between User and Provider Utilities

Abstract: Most existing recommender systems focus primarily on matching users (content consumers) to content which maximizes user satisfaction on the platform. It is increasingly obvious, however, that content providers have a critical influence on user satisfaction through content creation, largely determining the content pool available for recommendation. A natural question thus arises: can we design recommenders taking into account the long-term utility of both users and content providers? By doing so, we hope to sustain more content providers and a more diverse content pool for long-term user satisfaction. Understanding the full impact of recommendations on both user and content provider groups is challenging. This paper aims to serve as a research investigation of one approach toward building a content provider aware recommender, and evaluating its impact in a simulated setup. To characterize the user-recommender-provider interdependence, we complement user modeling by formalizing provider dynamics as well. The resulting joint dynamical system gives rise to a weakly-coupled partially observable Markov decision process driven by recommender actions and user feedback to providers. We then build a REINFORCE recommender agent, coined EcoAgent, to optimize a joint objective of user utility and the counterfactual utility lift of the content provider associated with the recommended content, which we show to be equivalent to maximizing overall user utility and the utilities of all content providers on the platform under some mild assumptions. To evaluate our approach, we introduce a simulation environment capturing the key interactions among users, providers, and the recommender. We offer a number of simulated experiments that shed light on both the benefits and the limitations of our approach. These results help understand how and when a content provider aware recommender agent is of benefit in building multi-stakeholder recommender systems.

Price-aware Recommendation with Graph Convolutional Networks

Abstract: In recent years, much research effort on recommendation has been devoted to mining user behaviors, i.e., collaborative filtering, along with the general information which describes users or items, e.g., textual attributes, categorical demographics, product images, and so on. Price, an important factor in marketing — which determines whether a user will make the final purchase decision on an item — surprisingly, has received relatively little scrutiny.In this work, we aim at developing an effective method to predict user purchase intention with the focus on the price factor in recommender systems. The main difficulties are twofold: 1) the preference and sensitivity of a user on item price are unknown, which are only implicitly reflected in the items that the user has purchased, and 2) how the item price affects a user’s intention depends largely on the product category, that is, the perception and affordability of a user on item price could vary significantly across categories. Towards the first difficulty, we propose to model the transitive relationship between user-to-item and item-to-price, taking the inspiration from the recently developed Graph Convolution Networks (GCN). The key idea is to propagate the influence of price on users with items as the bridge, so as to make the learned user representations be price-aware. For the second difficulty, we further integrate item categories into the propagation progress and model the possible pairwise interactions for predicting user-item interactions. We conduct extensive experiments on two real-world datasets, demonstrating the effectiveness of our GCN-based method in learning the price-aware preference of users. Further analysis reveals that modeling the price awareness is particularly useful for predicting user preference on items of unexplored categories.

E-commerce Recommendation with Weighted Expected Utility

Abstract: Different from shopping at retail stores, consumers on e-commerce platforms usually cannot touch or try products before purchasing, which means that they have to make decisions when they are uncertain about the outcome (e.g., satisfaction level) of purchasing a product. To study people's preferences with regard to choices that have uncertain outcomes, economics researchers have proposed the hypothesis of Expected Utility (EU) that models the subject value associated with an individual's choice as the statistical expectations of that individual's valuations of the outcomes of this choice. Despite its success in studies of game theory and decision theory, the effectiveness of EU, however, is mostly unknown in e-commerce recommendation systems. Previous research on e-commerce recommendation interprets the utility of purchase decisions either as a function of the consumed quantity of the product or as the gain of sellers/buyers in the monetary sense. As most consumers just purchase one unit of a product at a time and most alternatives have similar prices, such modeling of purchase utility is likely to be inaccurate in practice. In this paper, we interpret purchase utility as the satisfaction level a consumer gets from a product and propose a recommendation framework using EU to model consumers' behavioral patterns. We assume that consumer estimates the expected utilities of all the alternatives and choose products with maximum expected utility for each purchase. To deal with the potential psychological biases of each consumer, we introduce the usage of Probability Weight Function (PWF) and design our algorithm based on Weighted Expected Utility (WEU). Empirical study on real-world e-commerce datasets shows that our proposed ranking-based recommendation framework achieves statistically significant improvement against both classical Collaborative Filtering/Latent Factor Models and state-of-the-art deep models in top-K recommendation.

Learning Personalized Risk Preferences for Recommendation

Abstract: The rapid growth of e-commerce has made people accustomed to shopping online. Before making purchases on e-commerce websites, most consumers tend to rely on rating scores and review information to make purchase decisions. With this information, they can infer the quality of products to reduce the risk of purchase. Specifically, items with high rating scores and good reviews tend to be less risky, while items with low rating scores and bad reviews might be risky to purchase. On the other hand, the purchase behaviors will also be influenced by consumers' tolerance of risks, known as the risk attitudes. Economists have studied risk attitudes for decades. These studies reveal that people are not always rational enough when making decisions, and their risk attitudes may vary in different circumstances. Most existing works over recommendation systems do not consider users' risk attitudes in modeling, which may lead to inappropriate recommendations to users. For example, suggesting a risky item to a risk-averse person or a conservative item to a risk-seeking person may result in the reduction of user experience. In this paper, we propose a novel risk-aware recommendation framework that integrates machine learning and behavioral economics to uncover the risk mechanism behind users' purchasing behaviors. Concretely, we first develop statistical methods to estimate the risk distribution of each item and then draw the Nobel-award winning Prospect Theory into our model to learn how users choose from probabilistic alternatives that involve risks, where the probabilities of the outcomes are uncertain. Experiments on several e-commerce datasets demonstrate that by taking user risk preferences into consideration, our approach can achieve better performance than many classical recommendation approaches, and further analyses also verify the advantages of risk-aware recommendation beyond accuracy.

U-rank: Utility-oriented Learning to Rank with Implicit Feedback

Abstract: Learning to rank with implicit feedback is one of the most important tasks in many real-world information systems where the objective is some specific utility, e.g., clicks and revenue. However, we point out that existing methods based on probabilistic ranking principle do not necessarily achieve the highest utility. To this end, we propose a novel ranking framework called U-rank that directly optimizes the expected utility of the ranking list. With a position-aware deep click-through rate prediction model, we address the attention bias considering both query-level and item-level features. Due to the item-specific attention bias modeling, the optimization for expected utility corresponds to a maximum weight matching on the item-position bipartite graph. We base the optimization of this objective in an efficient Lambdaloss framework, which is supported by both theoretical and empirical analysis. We conduct extensive experiments for both web search and recommender systems over three benchmark datasets and two proprietary datasets, where the performance gain of U-rank over state-of-the-arts is demonstrated. Moreover, our proposed U-rank has been deployed on a large-scale commercial recommender and a large improvement over the production baseline has been observed in an online A/B testing.

Studies in the surveyed literature that provide the code.

In particular, by reviewing the ECRSs literature, we note several limitations concerning the reproducibility of the studies.
As reported in Table {{table:ffbc4135-9bb5-4007-b0a8-57a7e4e2cb09}} , only a very small subset of 15 articles, out of 133 ({{formula:41109bac-4000-478d-b7b3-42f3781560e9}} %) identified by the present systematic review share the implementation codeWe did not dive into the code details because even if the code is shared, it was found earlier in the RSs literature {{cite:a9703ab}}, {{cite:70f5084}}, {{cite:22fabb6}} that in many cases important information is missing to ensure reproducibility (e.g., pre-processing code)..
Notably, as can be seen from the table, we find no article that publicly share the code prior to 2019.
In addition, the level of reproducibility is quite uneven when considering the different subdomains of ECRSs.
In particular, we note the following critical issues:
there are many articles published in the profit-awareness subdomain but only two of them share the code;
all the articles published in the field of promotional strategies refer to relevance-based bundling methods (i.e., there is no code shared about brand-awareness and pricing methods);
the code of articles concerning price-sensitivity and long-term value methods is published only for the most recent and advanced GNN- and RL-based algorithms.
Consequently, it would be beneficial and significantly accelerate progress in this field if researchers would pay special attention to increasing the level of reproducibility.


X-rays, MR, and Dermoscopic (Skin lesion)

PGGAN has a complex architecture setting. It is easy to implement and shows efficient performance for all resolutions of images except 1024x1024. It is generalized for several biomedical imaging modalities.

SAGAN uses a self-attention in the architecture of DCGAN. It is easy to implement and shows efficient performance for all low resolutions of images. It is generalized for several biomedical imaging modalities.

StyleGAN uses a style generator to control the generation of synthetic images with salient features. It shows efficient performance for all resolutions of images except 1024x1024. It is generalized for several biomedical imaging modalities.

SRGAN shows a stabilized training and efficient performance for all resolutions of images.

CycleGAN has a stabilized training for image-to-image translation task.

PGGAN uses high memory and computational cost due to its complex architecture compared to DCGAN.

SAGAN uses high memory and computational cost due to its complex architecture compared to DCGAN. It shows unstable training for several high-resolution imaging modalities.

StyleGAN uses high memory and computational cost due to its complex architecture compared to DCGAN and PGGAN.

SRGAN uses high memory and computational cost due to its complex architecture.

It uses high memory and computational cost due to its complex architecture and shows unstable training for high-resolution and fine-grained biomedical imaging modalities such as PET and histopathology images.

Progressive Growing of GANs for Improved Quality, Stability, and Variation

Abstract: We describe a new training methodology for generative adversarial networks. The key idea is to grow both the generator and discriminator progressively: starting from a low resolution, we add new layers that model increasingly fine details as training progresses. This both speeds the training up and greatly stabilizes it, allowing us to produce images of unprecedented quality, e.g., CelebA images at 1024^2. We also propose a simple way to increase the variation in generated images, and achieve a record inception score of 8.80 in unsupervised CIFAR10. Additionally, we describe several implementation details that are important for discouraging unhealthy competition between the generator and discriminator. Finally, we suggest a new metric for evaluating GAN results, both in terms of image quality and variation. As an additional contribution, we construct a higher-quality version of the CelebA dataset.

Self-Attention Generative Adversarial Networks

Abstract: In this paper, we propose the Self-Attention Generative Adversarial Network (SAGAN) which allows attention-driven, long-range dependency modeling for image generation tasks. Traditional convolutional GANs generate high-resolution details as a function of only spatially local points in lower-resolution feature maps. In SAGAN, details can be generated using cues from all feature locations. Moreover, the discriminator can check that highly detailed features in distant portions of the image are consistent with each other. Furthermore, recent work has shown that generator conditioning affects GAN performance. Leveraging this insight, we apply spectral normalization to the GAN generator and find that this improves training dynamics. The proposed SAGAN achieves the state-of-the-art results, boosting the best published Inception score from 36.8 to 52.52 and reducing Frechet Inception distance from 27.62 to 18.65 on the challenging ImageNet dataset. Visualization of the attention layers shows that the generator leverages neighborhoods that correspond to object shapes rather than local regions of fixed shape.

A Style-Based Generator Architecture for Generative Adversarial Networks

Abstract: We propose an alternative generator architecture for generative adversarial networks, borrowing from style transfer literature. The new architecture leads to an automatically learned, unsupervised separation of high-level attributes (e.g., pose and identity when trained on human faces) and stochastic variation in the generated images (e.g., freckles, hair), and it enables intuitive, scale-specific control of the synthesis. The new generator improves the state-of-the-art in terms of traditional distribution quality metrics, leads to demonstrably better interpolation properties, and also better disentangles the latent factors of variation. To quantify interpolation quality and disentanglement, we propose two new, automated methods that are applicable to any generator architecture. Finally, we introduce a new, highly varied and high-quality dataset of human faces.

Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network

Abstract: Despite the breakthroughs in accuracy and speed of single image super-resolution using faster and deeper convolutional neural networks, one central problem remains largely unsolved: how do we recover the finer texture details when we super-resolve at large upscaling factors? The behavior of optimization-based super-resolution methods is principally driven by the choice of the objective function. Recent work has largely focused on minimizing the mean squared reconstruction error. The resulting estimates have high peak signal-to-noise ratios, but they are often lacking high-frequency details and are perceptually unsatisfying in the sense that they fail to match the fidelity expected at the higher resolution. In this paper, we present SRGAN, a generative adversarial network (GAN) for image super-resolution (SR). To our knowledge, it is the first framework capable of inferring photo-realistic natural images for 4x upscaling factors. To achieve this, we propose a perceptual loss function which consists of an adversarial loss and a content loss. The adversarial loss pushes our solution to the natural image manifold using a discriminator network that is trained to differentiate between the super-resolved images and original photo-realistic images. In addition, we use a content loss motivated by perceptual similarity instead of similarity in pixel space. Our deep residual network is able to recover photo-realistic textures from heavily downsampled images on public benchmarks. An extensive mean-opinion-score (MOS) test shows hugely significant gains in perceptual quality using SRGAN. The MOS scores obtained with SRGAN are closer to those of the original high-resolution images than to those obtained with any state-of-the-art method.

Normalization of breast MRIs using Cycle-Consistent Generative Adversarial Networks

Abstract: Dynamic Contrast Enhanced-Magnetic Resonance Imaging (DCE-MRI) is widely used to complement ultrasound examinations and x-ray mammography during the early detection and diagnosis of breast cancer. However, images generated by various MRI scanners (e.g. GE Healthcare vs Siemens) differ both in intensity and noise distribution, preventing algorithms trained on MRIs from one scanner to generalize to data from other scanners successfully. We propose a method for image normalization to solve this problem. MRI normalization is challenging because it requires both normalizing intensity values and mapping between the noise distributions of different scanners. We utilize a cycle-consistent generative adversarial network to learn a bidirectional mapping between MRIs produced by GE Healthcare and Siemens scanners. This allows us learning the mapping between two different scanner types without matched data, which is not commonly available. To ensure the preservation of breast shape and structures within the breast, we propose two technical innovations. First, we incorporate a mutual information loss with the CycleGAN architecture to ensure that the structure of the breast is maintained. Second, we propose a modified discriminator architecture which utilizes a smaller field-of-view to ensure the preservation of finer details in the breast tissue. Quantitative and qualitative evaluations show that the second proposed method was able to consistently preserve a high level of detail in the breast structure while also performing the proper intensity normalization and noise mapping. Our results demonstrate that the proposed model can successfully learn a bidirectional mapping between MRIs produced by different vendors, potentially enabling improved accuracy of downstream computational algorithms for diagnosis and detection of breast cancer. All the data used in this study are publicly available.

An overview of experimental effects of benchmark GAN variants for biomedical image generation.

Challenges and Future Research Directions

The implementation effects such as computational cost, memory consumption, and pros and cons for alternate biomedical imaging modalities of benchmark GANs architectures have been added in Table {{table:36923abe-b30f-4d7b-b138-e5e9ef24d4c7}} . Table {{table:36923abe-b30f-4d7b-b138-e5e9ef24d4c7}}  provides a comprehensive overview of experimental effects that impact during the re-implementation of these architectures for diverse biomedical imaging modalities. This table will guide a reader to find the best appropriate GAN architecture for targeted biomedical images.