Faithful to the Original: Fact Aware Neural Abstractive Summarization

Abstract: Unlike extractive summarization, abstractive summarization has to fuse different parts of the source text, which inclines to create fake facts. Our preliminary study reveals nearly 30% of the outputs from a state-of-the-art neural summarization system suffer from this problem. While previous abstractive summarization approaches usually focus on the improvement of informativeness, we argue that faithfulness is also a vital prerequisite for a practical abstractive summarization system. To avoid generating fake facts in a summary, we leverage open information extraction and dependency parse technologies to extract actual fact descriptions from the source text. The dual-attention sequence-to-sequence framework is then proposed to force the generation conditioned on both the source text and the extracted fact descriptions. Experiments on the Gigaword benchmark dataset demonstrate that our model can greatly reduce fake summaries by 80%. Notably, the fact descriptions also bring significant improvement on informativeness since they often condense the meaning of the source text.

Boosting Factual Correctness of Abstractive Summarization with Knowledge Graph

Abstract: A commonly observed problem with abstractive summarization is the distortion or fabrication of factual information in the article. This inconsistency between summary and original text has led to various concerns over its applicability. In this paper, we propose a Fact-Aware Summarization model, FASum, which extracts factual relations from the article to build a knowledge graph and integrates it into the neural decoding process. Then, we propose a Factual Corrector model, FC, that can modify abstractive summaries generated by any summarization model to improve factual correctness. Empirical results show that FASum can generate summaries with higher factual correctness compared with state-of-the-art abstractive summarization systems. And FC improves the factual correctness of summaries generated by various models via only modifying several entity tokens.

Knowledge Graph-Augmented Abstractive Summarization with Semantic-Driven Cloze Reward

Abstract: Sequence-to-sequence models for abstractive summarization have been studied extensively, yet the generated summaries commonly suffer from fabricated content, and are often found to be near-extractive. We argue that, to address these issues, the summarizer should acquire semantic interpretation over input, e.g., via structured representation, to allow the generation of more informative summaries. In this paper, we present ASGARD, a novel framework for Abstractive Summarization with Graph-Augmentation and semantic-driven RewarD. We propose the use of dual encoders—a sequential document encoder and a graph-structured encoder—to maintain the global context and local characteristics of entities, complementing each other. We further design a reward based on a multiple choice cloze test to drive the model to better capture entity interactions. Results show that our models produce significantly higher ROUGE scores than a variant without knowledge graph as input on both New York Times and CNN/Daily Mail datasets. We also obtain better or comparable performance compared to systems that are fine-tuned from large pretrained language models. Human judges further rate our model outputs as more informative and containing fewer unfaithful errors.

Mind The Facts: Knowledge-Boosted Coherent Abstractive Text Summarization

Abstract: Neural models have become successful at producing abstractive summaries that are human-readable and fluent. However, these models have two critical shortcomings: they often don't respect the facts that are either included in the source article or are known to humans as commonsense knowledge, and they don't produce coherent summaries when the source article is long. In this work, we propose a novel architecture that extends Transformer encoder-decoder architecture in order to improve on these shortcomings. First, we incorporate entity-level knowledge from the Wikidata knowledge graph into the encoder-decoder architecture. Injecting structural world knowledge from Wikidata helps our abstractive summarization model to be more fact-aware. Second, we utilize the ideas used in Transformer-XL language model in our proposed encoder-decoder architecture. This helps our model with producing coherent summaries even when the source article is long. We test our model on CNN/Daily Mail summarization dataset and show improvements on ROUGE scores over the baseline Transformer model. We also include model predictions for which our model accurately conveys the facts, while the baseline Transformer model doesn't.

Multi-Fact Correction in Abstractive Text Summarization

Abstract: Pre-trained neural abstractive summarization systems have dominated extractive strategies on news summarization performance, at least in terms of ROUGE. However, system-generated abstractive summaries often face the pitfall of factual inconsistency: generating incorrect facts with respect to the source text. To address this challenge, we propose Span-Fact, a suite of two factual correction models that leverages knowledge learned from question answering models to make corrections in system-generated summaries via span selection. Our models employ single or multi-masking strategies to either iteratively or auto-regressively replace entities in order to ensure semantic consistency w.r.t. the source text, while retaining the syntactic structure of summaries generated by abstractive summarization models. Experiments show that our models significantly boost the factual consistency of system-generated summaries without sacrificing summary quality in terms of both automatic metrics and human evaluation.

Factual Error Correction for Abstractive Summarization Models

Abstract: Neural abstractive summarization systems have achieved promising progress, thanks to the availability of large-scale datasets and models pre-trained with self-supervised methods. However, ensuring the factual consistency of the generated summaries for abstractive summarization systems is a challenge. We propose a post-editing corrector module to address this issue by identifying and correcting factual errors in generated summaries. The neural corrector model is pre-trained on artificial examples that are created by applying a series of heuristic transformations on reference summaries. These transformations are inspired by an error analysis of state-of-the-art summarization model outputs. Experimental results show that our model is able to correct factual errors in summaries generated by other neural summarization models and outperforms previous models on factual consistency evaluation on the CNN/DailyMail dataset. We also find that transferring from artificial error correction to downstream settings is still very challenging.

Improving Truthfulness of Headline Generation

Abstract: Most studies on abstractive summarization report ROUGE scores between system and reference summaries. However, we have a concern about the truthfulness of generated summaries: whether all facts of a generated summary are mentioned in the source text. This paper explores improving the truthfulness in headline generation on two popular datasets. Analyzing headlines generated by the state-of-the-art encoder-decoder model, we show that the model sometimes generates untruthful headlines. We conjecture that one of the reasons lies in untruthful supervision data used for training the model. In order to quantify the truthfulness of article-headline pairs, we consider the textual entailment of whether an article entails its headline. After confirming quite a few untruthful instances in the datasets, this study hypothesizes that removing untruthful instances from the supervision data may remedy the problem of the untruthful behaviors of the model. Building a binary classifier that predicts an entailment relation between an article and its headline, we filter out untruthful instances from the supervision data. Experimental results demonstrate that the headline generation model trained on filtered supervision data shows no clear difference in ROUGE scores but remarkable improvements in automatic and manual evaluations of the generated headlines.

Constrained Abstractive Summarization: Preserving Factual Consistency with Constrained Generation

Abstract: Despite significant progress, state-of-the-art abstractive summarization methods are still prone to hallucinate content inconsistent with the source document. In this paper, we propose Constrained Abstractive Summarization (CAS), a general setup that preserves the factual consistency of abstractive summarization by specifying tokens as constraints that must be present in the summary. We adopt lexically constrained decoding, a technique generally applicable to autoregressive generative models, to fulfill CAS and conduct experiments in two scenarios: (1) automatic summarization without human involvement, where keyphrases are extracted from the source document and used as constraints; (2) human-guided interactive summarization, where human feedback in the form of manual constraints are used to guide summary generation. Automatic and human evaluations on two benchmark datasets demonstrate that CAS improves both lexical overlap (ROUGE) and factual consistency of abstractive summarization. In particular, we observe up to 13.8 ROUGE-2 gains when only one manual constraint is used in interactive summarization.

List of factual consistency optimization methods. links to corresponding resource location.

In this section, we provide an overview of approaches to optimizing summarization systems towards factual consistency. As illustrated in Figure {{figure:a74accaf-0300-4e3e-a3b8-2986689bcd05}} , we divide existing methods roughly into 4 classes according to principles that each method bases on: fact encode-based, textual entailment-based, post-editing-based, and other methods. Besides, we organize these methods into Table {{table:db6575d3-d9f7-4520-9118-ca35d99e831b}} .


Privacy preservation in distributed learning

An end-to-end latency analysis for on-device FL with blockchain.

A communication analysis model for the edge computing plane in digital twin-based MEC networks with blockchain.

A DRL-based approach for resource management in FLchain.

A DRL-based approach for resource allocation in global model transmissions.

An incentive protocol for supporting FLchain clients.

A scalable privacy architecture for blockchain-based distributed learning.

Other parameters such as block arrival rate, rewards have not been considered in system analysis.

Comparison between DPoS with other consensus schemes has not been performed.

Resource allocation for block mining at miners has not been considered.

The role of blockchain in FL training has not been verified.

Training performance of FL has not been evaluated.

System costs, such as latency, incurred from privacy preservation has not been considered.

Blockchained On-Device Federated Learning

Abstract: By leveraging blockchain, this letter proposes a blockchained federated learning (BlockFL) architecture where local learning model updates are exchanged and verified. This enables on-device machine learning without any centralized training data or coordination by utilizing a consensus mechanism in blockchain. Moreover, we analyze an end-to-end latency model of BlockFL and characterize the optimal block generation rate by considering communication, computation, and consensus delays.

Low-latency Federated Learning and Blockchain for Edge Association in Digital Twin empowered 6G Networks

Abstract: Emerging technologies such as digital twins and 6th Generation mobile networks (6G) have accelerated the realization of edge intelligence in Industrial Internet of Things (IIoT). The integration of digital twin and 6G bridges the physical system with digital space and enables robust instant wireless connectivity. With increasing concerns on data privacy, federated learning has been regarded as a promising solution for deploying distributed data processing and learning in wireless networks. However, unreliable communication channels, limited resources, and lack of trust among users, hinder the effective application of federated learning in IIoT. In this paper, we introduce the Digital Twin Wireless Networks (DTWN) by incorporating digital twins into wireless networks, to migrate real-time data processing and computation to the edge plane. Then, we propose a blockchain empowered federated learning framework running in the DTWN for collaborative computing, which improves the reliability and security of the system, and enhances data privacy. Moreover, to balance the learning accuracy and time cost of the proposed scheme, we formulate an optimization problem for edge association by jointly considering digital twin association, training data batch size, and bandwidth allocation. We exploit multi-agent reinforcement learning to find an optimal solution to the problem. Numerical results on real-world dataset show that the proposed scheme yields improved efficiency and reduced cost compared to benchmark learning method.

Resource Management for Blockchain-enabled Federated Learning: A Deep Reinforcement Learning Approach

Abstract: Blockchain-enabled Federated Learning (BFL) enables mobile devices to collaboratively train neural network models required by a Machine Learning Model Owner (MLMO) while keeping data on the mobile devices. Then, the model updates are stored in the blockchain in a decentralized and reliable manner. However, the issue of BFL is that the mobile devices have energy and CPU constraints that may reduce the system lifetime and training efficiency. The other issue is that the training latency may increase due to the blockchain mining process. To address these issues, the MLMO needs to (i) decide how much data and energy that the mobile devices use for the training and (ii) determine the block generation rate to minimize the system latency, energy consumption, and incentive cost while achieving the target accuracy for the model. Under the uncertainty of the BFL environment, it is challenging for the MLMO to determine the optimal decisions. We propose to use the Deep Reinforcement Learning (DRL) to derive the optimal decisions for the MLMO.

Resource Allocation in Mobility-Aware Federated Learning Networks: A Deep Reinforcement Learning Approach

Abstract: Federated learning allows mobile devices, i.e., workers, to use their local data to collaboratively train a global model required by the model owner. Federated learning thus addresses the privacy issues of traditional machine learning. However, federated learning faces the energy constraints of the workers and the high network resource cost due to the fact that a number of global model transmissions may be required to achieve the target accuracy. To address the energy constraint, a power beacon can be used that recharges energy to the workers. However, the model owner may need to pay an energy cost to the power beacon for the energy recharge. To address the high network resource cost, the model owner can use a WiFi channel, called default channel, for the global model transmissions. However, communication interruptions may occur due to the instability of the default channel quality. For this, special channels such as LTE channels can be used, but this incurs channel cost. As such, the problem of the model owner is to decide amounts of energy recharged to the workers and to choose channels used to transmit its global model to the workers to maximize the number of global model transmissions while minimizing the energy and channel costs. This is challenging for the model owner under the uncertainty of the channel, energy and mobility states of the workers. In this paper, we thus propose to employ the Deep Q-Network (DQN) that enables the model owner to find the optimal decisions on the energy and the channels without any a priori network knowledge. Simulation results show that the proposed DQN always achieves better performance compared to the conventional algorithms.

Ownership preserving AI Market Places using Blockchain

Abstract: We present a blockchain based system that allows data owners, cloud vendors, and AI developers to collaboratively train machine learning models in a trustless AI marketplace. Data is a highly valued digital asset and central to deriving business insights. Our system enables data owners to retain ownership and privacy of their data, while still allowing AI developers to leverage the data for training. Similarly, AI developers can utilize compute resources from cloud vendors without loosing ownership or privacy of their trained models. Our system protocols are set up to incentivize all three entities - data owners, cloud vendors, and AI developers to truthfully record their actions on the distributed ledger, so that the blockchain system provides verifiable evidence of wrongdoing and dispute resolution. Our system is implemented on the Hyperledger Fabric and can provide a viable alternative to centralized AI systems that do not guarantee data or model privacy. We present experimental performance results that demonstrate the latency and throughput of its transactions under different network configurations where peers on the blockchain may be spread across different datacenters and geographies. Our results indicate that the proposed solution scales well to large number of data and model owners and can train up to 70 models per second on a 12-peer non optimized blockchain network and roughly 30 models per second in a 24 peer network.

Secure Architectures Implementing Trusted Coalitions for Blockchained Distributed Learning (TCLearn)

Abstract: Distributed learning across coalitions is becoming popular for multi-centric implementation of deep learning models. However, the level of trust between the members of a coalition can vary and requires different security architectures. Privacy of the training data has been largely described in distributed learning. In this paper, we present a scalable security architecture providing additional features such as validation on the sources quality, confidentiality on the model within a trusted coalition or confidentiality among untrusted partners inside the coalition. More specifically, we propose solutions that guarantee preservation not only of data privacy but also of data quality, enforce a trustworthy sequence of iterative learning, and that lead to equitable sharing of the learned model among the coalition’s members. We give an example of its deployment in the case of the distributed optimization of a deep learning convolutional neural network trained on medical images.

Taxonomy of FLchain design and use cases.

In summary, we list FLchain design issues and use cases in Table {{table:46b870da-7185-406f-b6c2-40c8cfcf5812}}  to summarize the technical aspects as well as the key contributions and limitations of each reference.


Adding parts to the setup so that a choice to measure either the particle or the wave nature of the light is made after the light in the interferometer has been detected.

Implementing a quantum key distribution protocol, such as BB84.

Delayed-choice quantum eraser for the undergraduate laboratory

Abstract: In a delayed-choice quantum eraser, interference fringes are obtained by erasing which-way information after the interfering particle has already been irreversibly detected. Following an introductory review of delayed-choice experiments and quantum erasure, we describe the experimental realization of an optical delayed-choice quantum eraser, suitable for advanced undergraduates, based on polarization-entangled pairs of single photons. In our experiment, the delay of the erasure is implemented using two different setups. The first setup employs an arrangement of mirrors to increase the optical path length of the photons carrying which-way information. In the second setup, we use fiber-optic cables to elongate the path of these photons after their passage through the polarization analyzer but prior to their arrival at the detector. We compare our results to data obtained in the absence of a delay and find excellent agreement. This shows that the timing of the erasure is irrelevant, as also predicted by quantum mechanics. The experiment can serve as a valuable pedagogical tool for conveying the fundamentals of quantum mechanics.

A Demonstration of Quantum Key Distribution with Entangled Photons for the Undergraduate Laboratory

Abstract: Now that fundamental quantum principles of indeterminacy and measurement have become the basis of new technologies that provide secrecy between two communicating parties, there is a need to provide teaching laboratories that illustrate how these technologies work. In this article we describe a laboratory exercise in which students perform quantum key distribution with single photons, and see how the secrecy of the communication is ensured by the principles of quantum superposition and state projection. We used a table-top apparatus, similar to those used in correlated-photon undergraduate laboratories, to implement the Bennett-Brassard-84 protocol with polarization-entangled photons. Our experiment shows how the communication between two parties is disrupted by an eavesdropper. We use a simple quartz plate to mimic how an eavesdropper intercepts, measures, and resends the photons used in the communication, and we analyze the state of the light to show how the eavesdropper changes it.

Description of individual experiments that can be part of the sequence of the single-photon experiments.

Each instructor filled out a survey that had several sections that aimed to gather information from multiple courses and experiments. The survey began asking for information about the courses (up to a maximum of two) in which the instructors used the single-photon experiments. This included the course name, the level and major of the intended students, and the option to link to, or upload, a course syllabus. For each course, the survey then asked the instructors which experiments they utilized, using the descriptions shown in Table {{table:1287c982-9c54-4366-8689-094d1768f9d7}} . Different instructors implement these experiments in different ways and may combine some of the experiments that we have described separately, but we tried to use the most common names and delineations between experiments. The survey then asked the instructors which actions the students performed, giving instructors the option to select different actions for each experiment. The last part of the survey asked about the importance of a variety of learning goals, letting instructors select not important, somewhat important, or very important for each goal for each experiment. When instructors marked that they used the single photon experiments in a second course, the questions about the experiments, actions, and goals were repeated for the second course. Almost all of the questions were closed response with some open-response options where instructors could add an experiment, action, or goal not included in the closed response options.


Image classification, segmentation, etc.

Auto-Encoding Variational Bayes

Abstract: How can we perform efficient inference and learning in directed probabilistic models, in the presence of continuous latent variables with intractable posterior distributions, and large datasets? We introduce a stochastic variational inference and learning algorithm that scales to large datasets and, under some mild differentiability conditions, even works in the intractable case. Our contributions are two-fold. First, we show that a reparameterization of the variational lower bound yields a lower bound estimator that can be straightforwardly optimized using standard stochastic gradient methods. Second, we show that for i.i.d. datasets with continuous latent variables per datapoint, posterior inference can be made especially efficient by fitting an approximate inference model (also called a recognition model) to the intractable posterior using the proposed lower bound estimator. Theoretical advantages are reflected in experimental results.

InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets

Abstract: This paper describes InfoGAN, an information-theoretic extension to the Generative Adversarial Network that is able to learn disentangled representations in a completely unsupervised manner. InfoGAN is a generative adversarial network that also maximizes the mutual information between a small subset of the latent variables and the observation. We derive a lower bound to the mutual information objective that can be optimized efficiently, and show that our training procedure can be interpreted as a variation of the Wake-Sleep algorithm. Specifically, InfoGAN successfully disentangles writing styles from digit shapes on the MNIST dataset, pose from lighting of 3D rendered images, and background digits from the central digit on the SVHN dataset. It also discovers visual concepts that include hair styles, presence/absence of eyeglasses, and emotions on the CelebA face dataset. Experiments show that InfoGAN learns interpretable representations that are competitive with representations learned by existing fully supervised methods.

Deep Convolutional Inverse Graphics Network

Abstract: This paper presents the Deep Convolution Inverse Graphics Network (DC-IGN), a model that learns an interpretable representation of images. This representation is disentangled with respect to transformations such as out-of-plane rotations and lighting variations. The DC-IGN model is composed of multiple layers of convolution and de-convolution operators and is trained using the Stochastic Gradient Variational Bayes (SGVB) algorithm. We propose a training procedure to encourage neurons in the graphics code layer to represent a specific transformation (e.g. pose or light). Given a single input image, our model can generate new images of the same object with variations in pose and lighting. We present qualitative and quantitative results of the model's efficacy at learning a 3D rendering engine.

Image-to-image translation for cross-domain disentanglement

Abstract: Deep image translation methods have recently shown excellent results, outputting high-quality images covering multiple modes of the data distribution. There has also been increased interest in disentangling the internal representations learned by deep methods to further improve their performance and achieve a finer control. In this paper, we bridge these two objectives and introduce the concept of cross-domain disentanglement. We aim to separate the internal representation into three parts. The shared part contains information for both domains. The exclusive parts, on the other hand, contain only factors of variation that are particular to each domain. We achieve this through bidirectional image translation based on Generative Adversarial Networks and cross-domain autoencoders, a novel network component. Our model offers multiple advantages. We can output diverse samples covering multiple modes of the distributions of both domains, perform domain-specific image transfer and interpolation, and cross-domain retrieval without the need of labeled data, only paired images. We compare our model to the state-of-the-art in multi-modal image translation and achieve better results for translation on challenging datasets as well as for cross-domain retrieval on realistic datasets.

Rare Event Detection using Disentangled Representation Learning

Abstract: This paper presents a novel method for rare event detection from an image pair with class-imbalanced datasets. A straightforward approach for event detection tasks is to train a detection network from a large-scale dataset in an end-to-end manner. However, in many applications such as building change detection on satellite images, few positive samples are available for the training. Moreover, scene image pairs contain many trivial events, such as in illumination changes or background motions. These many trivial events and the class imbalance problem lead to false alarms for rare event detection. In order to overcome these difficulties, we propose a novel method to learn disentangled representations from only low-cost negative samples. The proposed method disentangles different aspects in a pair of observations: variant and invariant factors that represent trivial events and image contents, respectively. The effectiveness of the proposed approach is verified by the quantitative evaluations on four change detection datasets, and the qualitative analysis shows that the proposed method can acquire the representations that disentangle rare events from trivial ones.

Unsupervised Learning of Disentangled Representations from Video

Abstract: We present a new model DrNET that learns disentangled image representations from video. Our approach leverages the temporal coherence of video and a novel adversarial loss to learn a representation that factorizes each frame into a stationary part and a temporally varying component. The disentangled representation can be used for a range of tasks. For example, applying a standard LSTM to the time-vary components enables prediction of future frames. We evaluate our approach on a range of synthetic and real videos, demonstrating the ability to coherently generate hundreds of steps into the future.

Improving Disentangled Text Representation Learning with Information-Theoretic Guidance

Abstract: Learning disentangled representations of natural language is essential for many NLP tasks, e.g., conditional text generation, style transfer, personalized dialogue systems, etc. Similar problems have been studied extensively for other forms of data, such as images and videos. However, the discrete nature of natural language makes the disentangling of textual representations more challenging (e.g., the manipulation over the data space cannot be easily achieved). Inspired by information theory, we propose a novel method that effectively manifests disentangled representations of text, without any supervision on semantics. A new mutual information upper bound is derived and leveraged to measure dependence between style and content. By minimizing this upper bound, the proposed method induces style and content embeddings into two independent low-dimensional spaces. Experiments on both conditional text generation and text-style transfer demonstrate the high quality of our disentangled representation in terms of content and style preservation.

Disentangling by Partitioning: A Representation Learning Framework for Multimodal Sensory Data

Abstract: Multimodal sensory data resembles the form of information perceived by humans for learning, and are easy to obtain in large quantities. Compared to unimodal data, synchronization of concepts between modalities in such data provides supervision for disentangling the underlying explanatory factors of each modality. Previous work leveraging multimodal data has mainly focused on retaining only the modality-invariant factors while discarding the rest. In this paper, we present a partitioned variational autoencoder (PVAE) and several training objectives to learn disentangled representations, which encode not only the shared factors, but also modality-dependent ones, into separate latent variables. Specifically, PVAE integrates a variational inference framework and a multimodal generative model that partitions the explanatory factors and conditions only on the relevant subset of them for generation. We evaluate our model on two parallel speech/image datasets, and demonstrate its ability to learn disentangled representations by qualitatively exploring within-modality and cross-modality conditional generation with semantics and styles specified by examples. For quantitative analysis, we evaluate the classification accuracy of automatically discovered semantic units. Our PVAE can achieve over 99% accuracy on both modalities.

Learning Factorized Multimodal Representations

Abstract: Learning multimodal representations is a fundamentally complex research problem due to the presence of multiple heterogeneous sources of information. Although the presence of multiple modalities provides additional valuable information, there are two key challenges to address when learning from multimodal data: 1) models must learn the complex intra-modal and cross-modal interactions for prediction and 2) models must be robust to unexpected missing or noisy modalities during testing. In this paper, we propose to optimize for a joint generative-discriminative objective across multimodal data and labels. We introduce a model that factorizes representations into two sets of independent factors: multimodal discriminative and modality-specific generative factors. Multimodal discriminative factors are shared across all modalities and contain joint multimodal features required for discriminative tasks such as sentiment prediction. Modality-specific generative factors are unique for each modality and contain the information required for generating data. Experimental results show that our model is able to learn meaningful multimodal representations that achieve state-of-the-art or competitive performance on six multimodal datasets. Our model demonstrates flexible generative capabilities by conditioning on independent factors and can reconstruct missing modalities without significantly impacting performance. Lastly, we interpret our factorized representations to understand the interactions that influence multimodal learning.

Towards Counterfactual Image Manipulation via CLIP

Abstract: Leveraging StyleGAN's expressivity and its disentangled latent codes, existing methods can achieve realistic editing of different visual attributes such as age and gender of facial images. An intriguing yet challenging problem arises: Can generative models achieve counterfactual editing against their learnt priors? Due to the lack of counterfactual samples in natural datasets, we investigate this problem in a text-driven manner with Contrastive-Language-Image-Pretraining (CLIP), which can offer rich semantic knowledge even for various counterfactual concepts. Different from in-domain manipulation, counterfactual manipulation requires more comprehensive exploitation of semantic knowledge encapsulated in CLIP as well as more delicate handling of editing directions for avoiding being stuck in local minimum or undesired editing. To this end, we design a novel contrastive loss that exploits predefined CLIP-space directions to guide the editing toward desired directions from different perspectives. In addition, we design a simple yet effective scheme that explicitly maps CLIP embeddings (of target text) to the latent space and fuses them with latent codes for effective latent code optimization and accurate editing. Extensive experiments show that our design achieves accurate and realistic editing while driving by target texts with various counterfactual concepts.

Predict, Prevent, and Evaluate: Disentangled Text-Driven Image Manipulation Empowered by Pre-Trained Vision-Language Model

Abstract: To achieve disentangled image manipulation, previous works depend heavily on manual annotation. Meanwhile, the available manipulations are limited to a pre-defined set the models were trained for. We propose a novel framework, i.e., Predict, Prevent, and Evaluate (PPE), for disentangled text-driven image manipulation that requires little manual annotation while being applicable to a wide variety of manipulations. Our method approaches the targets by deeply exploiting the power of the large-scale pre-trained vision-language model CLIP. Concretely, we firstly Predict the possibly entangled attributes for a given text command. Then, based on the predicted attributes, we introduce an entanglement loss to Prevent entanglements during training. Finally, we propose a new evaluation metric to Evaluate the disentangled image manipulation. We verify the effectiveness of our method on the challenging face editing task. Extensive experiments show that the proposed PPE framework achieves much better quantitative and qualitative results than the up-to-date StyleCLIP baseline.

Learning Disentangled Representations for Recommendation

Abstract: User behavior data in recommender systems are driven by the complex interactions of many latent factors behind the users' decision making processes. The factors are highly entangled, and may range from high-level ones that govern user intentions, to low-level ones that characterize a user's preference when executing an intention. Learning representations that uncover and disentangle these latent factors can bring enhanced robustness, interpretability, and controllability. However, learning such disentangled representations from user behavior is challenging, and remains largely neglected by the existing literature. In this paper, we present the MACRo-mIcro Disentangled Variational Auto-Encoder (MacridVAE) for learning disentangled representations from user behavior. Our approach achieves macro disentanglement by inferring the high-level concepts associated with user intentions (e.g., to buy a shirt or a cellphone), while capturing the preference of a user regarding the different concepts separately. A micro-disentanglement regularizer, stemming from an information-theoretic interpretation of VAEs, then forces each dimension of the representations to independently reflect an isolated low-level factor (e.g., the size or the color of a shirt). Empirical results show that our approach can achieve substantial improvement over the state-of-the-art baselines. We further demonstrate that the learned representations are interpretable and controllable, which can potentially lead to a new paradigm for recommendation where users are given fine-grained control over targeted aspects of the recommendation lists.

Interpretable Deep Graph Generation with Node-edge Co-disentanglement

Abstract: Disentangled representation learning has recently attracted a significant amount of attention, particularly in the field of image representation learning. However, learning the disentangled representations behind a graph remains largely unexplored, especially for the attributed graph with both node and edge features. Disentanglement learning for graph generation has substantial new challenges including 1) the lack of graph deconvolution operations to jointly decode node and edge attributes; and 2) the difficulty in enforcing the disentanglement among latent factors that respectively influence: i) only nodes, ii) only edges, and iii) joint patterns between them. To address these challenges, we propose a new disentanglement enhancement framework for deep generative models for attributed graphs. In particular, a novel variational objective is proposed to disentangle the above three types of latent factors, with novel architecture for node and edge deconvolutions. Qualitative and quantitative experiments on both synthetic and real-world datasets demonstrate the effectiveness of the proposed model and its extensions.

Representatives of disentangled representation learning applications

In this section, we discuss the broad applications of DRL for various downstream tasks. Firstly, we will discuss DRL applications of different modalities, e.g., image, video, natural language, and multi-modality, which almost cover all the application fields of deep learning. Besides, we introduce DRL applications in recommendation and graph learning. Table {{table:88cbe23e-7a8d-4b72-8b05-b2378d0a9a3f}}  is a brief summary.
Finally, we demonstrate the effectiveness of DRL in few-shot and out-of-distribution (OOD) problems.


P: P-17 is a mixed use skyscraper proposed for construction in Dubai... The design is for a 379 m tall building, comprising 78 floors. H: P-17 is designed to have 78 floors and be over 500 meters tall.

Learning from the Worst: Dynamically Generated Datasets to Improve Online Hate Detection

Abstract: We present a human-and-model-in-the-loop process for dynamically generating datasets and training better performing and more robust hate detection models. We provide a new dataset of ~40,000 entries, generated and labelled by trained annotators over four rounds of dynamic data creation. It includes ~15,000 challenging perturbations and each hateful entry has fine-grained labels for the type and target of hate. Hateful entries make up 54% of the dataset, which is substantially higher than comparable datasets. We show that model performance is substantially improved using this approach. Models trained on later rounds of data collection perform better on test sets and are harder for annotators to trick. They also perform better on HateCheck, a suite of functional tests for online hate detection. We provide the code, dataset and annotation guidelines for other researchers to use. Accepted at ACL 2021.

Adversarial NLI: A New Benchmark for Natural Language Understanding

Abstract: We introduce a new large-scale NLI benchmark dataset, collected via an iterative, adversarial human-and-model-in-the-loop procedure. We show that training models on this new dataset leads to state-of-the-art performance on a variety of popular NLI benchmarks, while posing a more difficult challenge with its new test set. Our analysis sheds light on the shortcomings of current state-of-the-art models, and shows that non-expert annotators are successful at finding their weaknesses. The data collection method can be applied in a never-ending learning scenario, becoming a moving target for NLU, rather than a static benchmark that will quickly saturate.

Description of considered datasets. For ANLI, each example comes with a premise (P) providing context and a hypothesis statement (H).

An advantage of the proposed generative method is that we can automatically extend human-in-the-loop adversarial generation methods like Adversarial NLI , which are costly and time-consuming to curate. Given this motivation, we focus on particularly challenging human-in-the-loop examples rather than cases that are already well solved by existing benchmarks. To study effectiveness of the proposed approach, we conducted experiments on the hate speech detection (DynaHate {{cite:234f837}}) and natural language inference (NLI). For DynaHate we use a RoBERTa classifier trained on tweets {{cite:d7d605b}}, {{cite:ed28017}}.
We test generalization across both model architectures and (non-adversarial) data domains using a BERT model {{cite:55b3f35}} trained on the HateXplain dataset {{cite:4fdf613}}. For NLI we use DeBERTa {{cite:e00f542}} trained on MNLI {{cite:b4d5882}}. We test generalization using BERT trained on the QNLI dataset {{cite:5b47c6d}}. Further details are provided in Table {{table:9ac271e7-cd38-4a90-9a5d-61e5fcb148ae}}  and Appendix .


3: Resource allocation (corresponds to HLS scheduling)

Modeling real hardware constraints in backend FPGA target

Enforces inter-table and intra-action dependencies for global optimization

Extended functionality of resources available in Menshen

Tofino compiler uses heuristics for resource allocation

Packet Transactions: High-Level Programming for Line-Rate Switches

Abstract: Many algorithms for congestion control, scheduling, network measurement, active queue management, and traffic engineering require custom processing of packets in the data plane of a network switch. To run at line rate, these data-plane algorithms must be implemented in hardware. With today's switch hardware, algorithms cannot be changed, nor new algorithms installed, after a switch has been built. This paper shows how to program data-plane algorithms in a high-level language and compile those programs into low-level microcode that can run on emerging programmable line-rate switching chips. The key challenge is that many data-plane algorithms create and modify algorithmic state. To achieve line-rate programmability for stateful algorithms, we introduce the notion of a packet transaction: a sequential packet-processing code block that is atomic and isolated from other such code blocks. We have developed this idea in Domino, a C-like imperative language to express data-plane algorithms. We show with many examples that Domino provides a convenient way to express sophisticated data-plane algorithms, and show that these algorithms can be run at line rate with modest estimated chip-area overhead.

Isolation Mechanisms for High-Speed Packet-Processing Pipelines

Abstract: Data-plane programmability is now mainstream. As we find more use cases, deployments need to be able to run multiple packet-processing modules in a single device. These are likely to be developed by independent teams, either within the same organization or from multiple organizations. Therefore, we need isolation mechanisms to ensure that modules on the same device do not interfere with each other. This paper presents Menshen, an extension of the Reconfigurable Match Tables (RMT) pipeline that enforces isolation between different packet-processing modules. Menshen is comprised of a set of lightweight hardware primitives and an extension to the open source P4-16 reference compiler that act in conjunction to meet this goal. We have prototyped Menshen on two FPGA platforms (NetFPGA and Corundum). We show that our design provides isolation, and allows new modules to be loaded without impacting the ones already running. Finally, we demonstrate that feasibility of implementing Menshen on ASICs by using the FreePDK45nm technology library and the Synopsys DC synthesis software, showing that our design meets timing at a 1GHz clock frequency and needs approximately 6% additional chip area. We have open sourced the code for Menshen's hardware and software at https://isolation.quest/.

CaT in relation to HLS and other related work.

To produce high quality implementations, CaT combines ideas from several prior projects (Table {{table:768939a6-29e8-4215-b4c2-42d0bf6e6dce}} ) into an end-to-end system for the first time.
The HLS workflow of CaT divides up the process of compiling a P4-16 program into three phases – their correspondence with traditional HLS steps {{cite:f88fe30}} and relation with prior work are highlighted in Table {{table:ad46ba77-8e0d-4067-a05c-7cf8812f25ff}} .
As shown, our resource transformation corresponds to the code transformation step in HLS, our resource synthesis corresponds to operation binding (or module selection {{cite:56ca8eb}}) in HLS, and our resource allocation corresponds to the scheduling step in HLS.
To further improve the effectiveness of HLS in our setting, we customize these phases to RMT pipelines. Specifically, our resource transformations focus on scarce resources (pipeline stages, gateways). Our resource synthesis procedure leverages the strict requirement that stateful updates must fit in a single stage to ease the task of pipelining, which is often challenging for HLS in general.
Our resource allocation phase uses the result of synthesis on action blocks, to pack multiple action blocks in each pipeline stage.
Thus, while these phases build on top of well-known prior work including HLS efforts (Column 3), there are significant differences and novelties in our work (Column 4). The last column lists complementary techniques that could be combined in our compiler.


ACDC: Altering Control Dependence Chains for Automated Patch Generation

Abstract: Once a failure is observed, the primary concern of the developer is to identify what caused it in order to repair the code that induced the incorrect behavior. Until a permanent repair is afforded, code repair patches are invaluable. The aim of this work is to devise an automated patch generation technique that proceeds as follows: Step1) It identifies a set of failure-causing control dependence chains that are minimal in terms of number and length. Step2) It identifies a set of predicates within the chains along with associated execution instances, such that negating the predicates at the given instances would exhibit correct behavior. Step3) For each candidate predicate, it creates a classifier that dictates when the predicate should be negated to yield correct program behavior. Step4) Prior to each candidate predicate, the faulty program is injected with a call to its corresponding classifier passing it the program state and getting a return value predictively indicating whether to negate the predicate or not. The role of the classifiers is to ensure that: 1) the predicates are not negated during passing runs; and 2) the predicates are negated at the appropriate instances within failing runs. We implemented our patch generation approach for the Java platform and evaluated our toolset using 148 defects from the Introclass and Siemens benchmarks. The toolset identified 56 full patches and another 46 partial patches, and the classification accuracy averaged 84%.

Causal Program Dependence Analysis

Abstract: We introduce Causal Program Dependence Analysis (CPDA), a dynamic dependence analysis that applies causal inference to model the strength of program dependence relations in a continuous space. CPDA observes the association between program elements by constructing and executing modified versions of a program. One advantage of CPDA is that this construction requires only light-weight parsing rather than sophisticated static analysis. The result is a collection of observations based on how often a change in the value produced by a mutated program element affects the behavior of other elements. From this set of observations, CPDA discovers a causal structure capturing the causal (i.e., dependence) relation between program elements. Qualitative evaluation finds that CPDA concisely expresses key dependence relationships between program elements. As an example application, we apply CPDA to the problem of fault localization. Using minimal test suites, our approach can rank twice as many faults compared to SBFL.

Estimation methods for papers relying upon an explicit graphical causal model.

RQ2: structure of the research community

Based upon our results (see section REF  and figure {{figure:b0241962-cc3d-45be-bc4e-ff0f8d803dbe}} ), we can say that the application of causal inference in software engineering is relatively young. The community of researchers applying statistical causal inference in software engineering seems also to be fragmented. We could only find one group of researchers (the group of prof. Podgurski and coauthors at Case Western Reserve University, Cleveland, OH, USA) that applied statistical causal inference over more than 10 years. As already noted in section , there are not so many cross-references. Only one paper {{cite:7ef6715}} provided an extensive state of the art (although focused on testing). The rest of the papers either did not mention related statistical causal inference work or clearly mentioned a lack of known related work. For example {{cite:ce50cf2}} states "To the best of our knowledge, this is the first time such a [propensity score matching] model is used in software development publications." although different matching methods have already been applied before (see tables {{table:48b2bc18-e14a-4eea-9755-88e842752b3d}}  and {{table:6891d495-488f-4ae3-867f-042966b0de5f}} ). This may be due both to the fact that statistical causal inference methods are relatively new and not so well known in software engineering, and to the fact that no surveys were available prior to this article.

The first group contains 24 papers that rely on explicitly adjusting for counfounders. These papers first define an explicit graphical causal model and apply the rules of the do-calculus (like the backdoor criterion) for identifying causal effects. Estimation is then done, either through classical statistical estimation methods (e.g., linear regression or random forest) or matching techniques. Table {{table:48b2bc18-e14a-4eea-9755-88e842752b3d}}  provides the details of the corresponding estimation methods and papers.


Double machine learning + Instrumental variables

Bayesian propensity score matching in automotive embedded software engineering

Abstract: Randomised field experiments, such as A/B testing, have long been the gold standard for evaluating the value that new software brings to customers. However, running randomised field experiments is not always desired, possible or even ethical in the development of automotive embedded software. In the face of such restrictions, we propose the use of the Bayesian propensity score matching technique for causal inference of observational studies in the automotive domain. In this paper, we present a method based on the Bayesian propensity score matching framework, applied in the unique setting of automotive software engineering. This method is used to generate balanced control and treatment groups from an observational online evaluation and estimate causal treatment effects from the software changes, even with limited samples in the treatment group. We exemplify the method with a proof-of-concept in the automotive domain. In the example, we have a larger control (Nc = 1100) fleet of cars using the current software and a small treatment fleet (Nt = 38), in which we introduce a new software variant. We demonstrate a scenario that shipping of a new software to all users is restricted, as a result, a fully randomised experiment could not be conducted. Therefore, we utilised the Bayesian propensity score matching method with 14 observed covariates as inputs. The results show more balanced groups, suitable for estimating causal treatment effects from the collected observational data. We describe the method in detail and share our configuration. Furthermore, we discuss how can such a method be used for online evaluation of new software utilising small groups of samples.

Bayesian causal inference in automotive software engineering and online evaluation

Abstract: Randomised field experiments, such as A/B testing, have long been the gold standard for evaluating software changes. In the automotive domain, running randomised field experiments is not always desired, possible, or even ethical. In the face of such limitations, we develop a framework BOAT (Bayesian causal modelling for ObvservAtional Testing), utilising observational studies in combination with Bayesian causal inference, in order to understand real-world impacts from complex automotive software updates and help software development organisations arrive at causal conclusions. In this study, we present three causal inference models in the Bayesian framework and their corresponding cases to address three commonly experienced challenges of software evaluation in the automotive domain. We develop the BOAT framework with our industry collaborator, and demonstrate the potential of causal inference by conducting empirical studies on a large fleet of vehicles. Moreover, we relate the causal assumption theories to their implications in practise, aiming to provide a comprehensive guide on how to apply the causal models in automotive software engineering. We apply Bayesian propensity score matching for producing balanced control and treatment groups when we do not have access to the entire user base, Bayesian regression discontinuity design for identifying covariate dependent treatment assignments and the local treatment effect, and Bayesian difference-in-differences for causal inference of treatment effect overtime and implicitly control unobserved confounding factors. Each one of the demonstrative case has its grounds in practise, and is a scenario experienced when randomisation is not feasible. With the BOAT framework, we enable online software evaluation in the automotive domain without the need of a fully randomised experiment.

Perfce: Performance Debugging on Databases with Chaos Engineering-Enhanced Causality Analysis

Abstract: Debugging performance anomalies in databases is challenging. Causal inference techniques enable qualitative and quantitative root cause analysis of performance downgrades. Nevertheless, causality analysis is challenging in practice, particularly due to limited observability. Recently, chaos engineering (CE) has been applied to test complex software systems. CE frameworks mutate chaos variables to inject catastrophic events (e.g., network slowdowns) to stress-test these software systems. The systems under chaos stress are then tested (e.g., via differential testing) to check if they retain normal functionality, such as returning correct SQL query outputs even under stress. To date, CE is mainly employed to aid software testing. This paper identifies the novel usage of CE in diagnosing performance anomalies in databases. Our framework, PERFCE, has two phases - offline and online. The offline phase learns statistical models of a database using both passive observations and proactive chaos experiments. The online phase diagnoses the root cause of performance anomalies from both qualitative and quantitative aspects on-the-fly. In evaluation, Perfce outperformed previous works on synthetic datasets and is highly accurate and moderately expensive when analyzing real-world (distributed) databases like MySQL and TiDB.

Estimation methods for papers not relying upon a graphical causal model.

The second group contains 7 papers {{cite:a30fcd8}}, {{cite:ce50cf2}}, {{cite:8be4c78}}, {{cite:fd3387f}}, {{cite:84a87f4}}, {{cite:ee63475}}, {{cite:705f58f}} that rely upon methods that implicitly deal with confounders and do not require an explicit graphical causal model. Table {{table:6891d495-488f-4ae3-867f-042966b0de5f}}  provides the details of the corresponding estimation methods and papers.


MRQA 2019 Shared Task: Evaluating Generalization in Reading Comprehension

Abstract: We present the results of the Machine Reading for Question Answering (MRQA) 2019 shared task on evaluating the generalization capabilities of reading comprehension systems. In this task, we adapted and unified 18 distinct question answering datasets into the same format. Among them, six datasets were made available for training, six datasets were made available for development, and the final six were hidden for final evaluation. Ten teams submitted systems, which explored various ideas including data sampling, multi-task learning, adversarial training and ensembling. The best system achieved an average F1 score of 72.5 on the 12 held-out datasets, 10.7 absolute points higher than our initial baseline based on BERT.

Know What You Don’t Know: Unanswerable Questions for SQuAD

Abstract: Extractive reading comprehension systems can often locate the correct answer to a question in a context document, but they also tend to make unreliable guesses on questions for which the correct answer is not stated in the context. Existing datasets either focus exclusively on answerable questions, or use automatically generated unanswerable questions that are easy to identify. To address these weaknesses, we present SQuADRUn, a new dataset that combines the existing Stanford Question Answering Dataset (SQuAD) with over 50,000 unanswerable questions written adversarially by crowdworkers to look similar to answerable ones. To do well on SQuADRUn, systems must not only answer questions when possible, but also determine when no answer is supported by the paragraph and abstain from answering. SQuADRUn is a challenging natural language understanding task for existing models: a strong neural system that gets 86% F1 on SQuAD achieves only 66% F1 on SQuADRUn. We release SQuADRUn to the community as the successor to SQuAD.

Adversarial Examples for Evaluating Reading Comprehension Systems

Abstract: Standard accuracy metrics indicate that reading comprehension systems are making rapid progress, but the extent to which these systems truly understand language remains unclear. To reward systems with real language understanding abilities, we propose an adversarial evaluation scheme for the Stanford Question Answering Dataset (SQuAD). Our method tests whether systems can answer questions about paragraphs that contain adversarially inserted sentences, which are automatically generated to distract computer systems without changing the correct answer or misleading humans. In this adversarial setting, the accuracy of sixteen published models drops from an average of $75\%$ F1 score to $36\%$; when the adversary is allowed to add ungrammatical sequences of words, average accuracy on four models decreases further to $7\%$. We hope our insights will motivate the development of new models that understand language more precisely.

Using contradictions improves question answering systems

Abstract: This work examines the use of contradiction in natural language inference (NLI) for question answering (QA). Typically, NLI systems help answer questions by determining if a potential answer is \emph{entailed} (supported) by some background context. But is it useful to also determine if an answer contradicts the context? We test this in two settings, multiple choice and extractive QA, and find that systems that incorporate contradiction can do slightly better than entailment-only systems on certain datasets. However, the best performances come from using contradiction, entailment, and QA model confidence scores together. This has implications for the deployment of QA systems in domains such as medicine and science where safety is an issue.

Extractive QA datasets used. Validation sets are used on the SQuAD2.0 and SQuAD adversarial datasets. MRQA 2019 dev sets are used for the other five datasets.

Tables {{table:94f2e8c4-1e68-4cff-9550-c2c13b99674a}}  (multiple choice) and {{table:22e664b7-dd67-4780-923e-ca5570dbaeee}}  (extractive QA) outline the datasets we used. Additional details such as train size and preprocessing steps are available in the references provided. When space doesn't allow CosmosQA is aliased to Cosmos, MCScript to MCS, MCScript-2.0 to MCS2, and MCTest to MCT. The only preprocessing step we performed was to filter out questions where no context passage is provided. Validation splits (as opposed to test splits) are used in the CosmosQA and QASC cases, since context passages or gold standard answers are not available for these datasets.


Generating Multi-label Discrete Patient Records using Generative Adversarial Networks

Abstract: Access to electronic health record (EHR) data has motivated computational advances in medical research. However, various concerns, particularly over privacy, can limit access to and collaborative use of EHR data. Sharing synthetic EHR data could mitigate risk. In this paper, we propose a new approach, medical Generative Adversarial Network (medGAN), to generate realistic synthetic patient records. Based on input real patient records, medGAN can generate high-dimensional discrete variables (e.g., binary and count features) via a combination of an autoencoder and generative adversarial networks. We also propose minibatch averaging to efficiently avoid mode collapse, and increase the learning efficiency with batch normalization and shortcut connections. To demonstrate feasibility, we showed that medGAN generates synthetic patient records that achieve comparable performance to real data on many experiments including distribution statistics, predictive modeling tasks and a medical expert review. We also empirically observe a limited privacy risk in both identity and attribute disclosure using medGAN.

Data Synthesis based on Generative Adversarial Networks

Abstract:
 Privacy is an important concern for our society where sharing data with partners or releasing data to the public is a frequent occurrence. Some of the techniques that are being used to achieve privacy are to remove identifiers, alter quasi-identifiers, and perturb values. Unfortunately, these approaches suffer from two limitations. First, it has been shown that private information can still be leaked if attackers possess some background knowledge or other information sources. Second, they do not take into account the adverse impact these methods will have on the utility of the released data. In this paper, we propose a method that meets both requirements. Our method, called
 table-GAN
 , uses generative adversarial networks (GANs) to synthesize fake tables that are statistically similar to the original table yet do not incur information leakage. We show that the machine learning models trained using our synthetic tables exhibit performance that is similar to that of models trained using the original table for unknown testing cases. We call this property
 model compatibility
 . We believe that anonymization/perturbation/synthesis methods without model compatibility are of little value. We used four real-world datasets from four different domains for our experiments and conducted indepth comparisons with state-of-the-art anonymization, perturbation, and generation techniques. Throughout our experiments, only our method consistently shows balance between privacy level and model compatibility.

Modeling Tabular data using Conditional GAN

Abstract: Modeling the probability distribution of rows in tabular data and generating realistic synthetic data is a non-trivial task. Tabular data usually contains a mix of discrete and continuous columns. Continuous columns may have multiple modes whereas discrete columns are sometimes imbalanced making the modeling difficult. Existing statistical and deep neural network models fail to properly model this type of data. We design TGAN, which uses a conditional generative adversarial network to address these challenges. To aid in a fair and thorough comparison, we design a benchmark with 7 simulated and 8 real datasets and several Bayesian network baselines. TGAN outperforms Bayesian methods on most of the real datasets whereas other deep learning methods could not.

CTAB-GAN: Effective Table Data Synthesizing

Abstract: While data sharing is crucial for knowledge development, privacy concerns and strict regulation (e.g., European General Data Protection Regulation (GDPR)) unfortunately limit its full effectiveness. Synthetic tabular data emerges as an alternative to enable data sharing while fulfilling regulatory and privacy constraints. The state-of-the-art tabular data synthesizers draw methodologies from generative Adversarial Networks (GAN) and address two main data types in the industry, i.e., continuous and categorical. In this paper, we develop CTAB-GAN, a novel conditional table GAN architecture that can effectively model diverse data types, including a mix of continuous and categorical variables. Moreover, we address data imbalance and long-tail issues, i.e., certain variables have drastic frequency differences across large values. To achieve those aims, we first introduce the information loss and classification loss to the conditional GAN. Secondly, we design a novel conditional vector, which efficiently encodes the mixed data type and skewed distribution of data variable. We extensively evaluate CTAB-GAN with the state of the art GANs that generate synthetic tables, in terms of data similarity and analysis utility. The results on five datasets show that the synthetic data of CTAB-GAN remarkably resembles the real data for all three types of variables and results into higher accuracy for five machine learning algorithms, by up to 17%.

Deep learning methods for tabular data synthesis

We focus on deep-learning approaches for tabular data synthesis and skip the discussion of classical methods such as Copulas {{cite:5f58139}}, {{cite:322d64e}} and Bayesian Networks {{cite:dd675ad}}. Table {{table:e7a7fbe2-bec6-4dd4-8f93-62ee2746d448}}  summarizes the recently developed deep learning methods for tabular data synthesis in terms of models, network architecture, and datasets. MedGAN {{cite:2318c73}} is designed for aggregated electronic health records (EHRs), which only have count and binary features. Since EHRs are high-dimensional and sparse {{cite:1fe711e}}, medGAN uses a pre-trained autoencoder to learn compact representations of the input data and thereby simplifies the GAN's task. MedGAN is improved by {{cite:1fe711e}}, where the standard GAN loss is replaced by Wasserstein loss with gradient penalty, and the new model is named medWGAN. However, different from AE-GAN, medGAN and medWGAN are limited in generalizing to real-world scenarios because they only consider count and binary features.


Scaling Local Self-Attention for Parameter Efficient Visual Backbones

Abstract: Self-attention has the promise of improving computer vision systems due to parameter-independent scaling of receptive fields and content-dependent interactions, in contrast to parameter-dependent scaling and content-independent interactions of convolutions. Self-attention models have recently been shown to have encouraging improvements on accuracy-parameter trade-offs compared to baseline convolutional models such as ResNet-50. In this work, we aim to develop self-attention models that can outperform not just the canonical baseline models, but even the high-performing convolutional models. We propose two extensions to self-attention that, in conjunction with a more efficient implementation of self-attention, improve the speed, memory usage, and accuracy of these models. We leverage these improvements to develop a new self-attention model family, HaloNets, which reach state-of-the-art accuracies on the parameter-limited setting of the ImageNet classification benchmark. In preliminary transfer learning experiments, we find that HaloNet models outperform much larger models and have better inference performance. On harder tasks such as object detection and instance segmentation, our simple local self-attention and convolutional hybrids show improvements over very strong baselines. These results mark another step in demonstrating the efficacy of self-attention models on settings traditionally dominated by convolutional models.

PVT v2: Improved Baselines with Pyramid Vision Transformer

Abstract: Transformer recently has presented encouraging progress in computer vision. In this work, we present new baselines by improving the original Pyramid Vision Transformer (PVT v1) by adding three designs, including (1) linear complexity attention layer, (2) overlapping patch embedding, and (3) convolutional feed-forward network. With these modifications, PVT v2 reduces the computational complexity of PVT v1 to linear and achieves significant improvements on fundamental vision tasks such as classification, detection, and segmentation. Notably, the proposed PVT v2 achieves comparable or better performances than recent works such as Swin Transformer. We hope this work will facilitate state-of-the-art Transformer researches in computer vision. Code is available at https://github.com/whai362/PVT.

Swin Transformer: Hierarchical Vision Transformer using Shifted Windows

Abstract: This paper presents a new vision Transformer, called Swin Transformer, that capably serves as a general-purpose backbone for computer vision. Challenges in adapting Transformer from language to vision arise from differences between the two domains, such as large variations in the scale of visual entities and the high resolution of pixels in images compared to words in text. To address these differences, we propose a hierarchical Transformer whose representation is computed with \textbf{S}hifted \textbf{win}dows. The shifted windowing scheme brings greater efficiency by limiting self-attention computation to non-overlapping local windows while also allowing for cross-window connection. This hierarchical architecture has the flexibility to model at various scales and has linear computational complexity with respect to image size. These qualities of Swin Transformer make it compatible with a broad range of vision tasks, including image classification (87.3 top-1 accuracy on ImageNet-1K) and dense prediction tasks such as object detection (58.7 box AP and 51.1 mask AP on COCO test-dev) and semantic segmentation (53.5 mIoU on ADE20K val). Its performance surpasses the previous state-of-the-art by a large margin of +2.7 box AP and +2.6 mask AP on COCO, and +3.2 mIoU on ADE20K, demonstrating the potential of Transformer-based models as vision backbones. The hierarchical design and the shifted window approach also prove beneficial for all-MLP architectures. The code and models are publicly available at~\url{https://github.com/microsoft/Swin-Transformer}.

A ConvNet for the 2020s

Abstract: The "Roaring 20s" of visual recognition began with the introduction of Vision Transformers (ViTs), which quickly superseded ConvNets as the state-of-the-art image classification model. A vanilla ViT, on the other hand, faces difficulties when applied to general computer vision tasks such as object detection and semantic segmentation. It is the hierarchical Transformers (e.g., Swin Transformers) that reintroduced several ConvNet priors, making Transformers practically viable as a generic vision backbone and demonstrating remarkable performance on a wide variety of vision tasks. However, the effectiveness of such hybrid approaches is still largely credited to the intrinsic superiority of Transformers, rather than the inherent inductive biases of convolutions. In this work, we reexamine the design spaces and test the limits of what a pure ConvNet can achieve. We gradually "modernize" a standard ResNet toward the design of a vision Transformer, and discover several key components that contribute to the performance difference along the way. The outcome of this exploration is a family of pure ConvNet models dubbed ConvNeXt. Constructed entirely from standard ConvNet modules, ConvNeXts compete favorably with Transformers in terms of accuracy and scalability, achieving 87.8% ImageNet top-1 accuracy and outperforming Swin Transformers on COCO detection and ADE20K segmentation, while maintaining the simplicity and efficiency of standard ConvNets.

InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutions

Abstract: Compared to the great progress of large-scale vision transformers (ViTs) in recent years, large-scale models based on convolutional neural networks (CNNs) are still in an early state. This work presents a new large-scale CNN-based foundation model, termed InternImage, which can obtain the gain from increasing parameters and training data like ViTs. Different from the recent CNNs that focus on large dense kernels, InternImage takes deformable convolution as the core operator, so that our model not only has the large effective receptive field required for downstream tasks such as detection and segmentation, but also has the adaptive spatial aggregation conditioned by input and task information. As a result, the proposed InternImage reduces the strict inductive bias of traditional CNNs and makes it possible to learn stronger and more robust patterns with large-scale parameters from massive data like ViTs. The effectiveness of our model is proven on challenging benchmarks including ImageNet, COCO, and ADE20K. It is worth mentioning that InternImage-H achieved a new record 65.4 mAP on COCO test-dev and 62.9 mIoU on ADE20K, outperforming current leading CNNs and ViTs. The code will be released at https://github.com/OpenGVLab/InternImage.

Comparison of different spatial token mixer designs.

STMs differ in the generation of the sampling point set and the aggregation weight,
as show in Table {{table:b9d095d0-1cd8-4b46-b9af-3e8016abff53}} .
In this paper, we consider four popular types of STM for comparison, , depth-wise convolution, dynamic convolution, local attention, and global attention.


Multimodal Quasi-AutoRegression: Forecasting the visual popularity of new fashion products

Abstract: Estimating the preferences of consumers is of utmost importance for the fashion industry as appropriately leveraging this information can be beneficial in terms of profit. Trend detection in fashion is a challenging task due to the fast pace of change in the fashion industry. Moreover, forecasting the visual popularity of new garment designs is even more demanding due to lack of historical data. To this end, we propose MuQAR, a Multimodal Quasi-AutoRegressive deep learning architecture that combines two modules: (1) a multi-modal multi-layer perceptron processing categorical, visual and textual features of the product and (2) a quasi-autoregressive neural network modelling the "target" time series of the product's attributes along with the "exogenous" time series of all other attributes. We utilize computer vision, image classification and image captioning, for automatically extracting visual features and textual descriptions from the images of new products. Product design in fashion is initially expressed visually and these features represent the products' unique characteristics without interfering with the creative process of its designers by requiring additional inputs (e.g manually written texts). We employ the product's target attributes time series as a proxy of temporal popularity patterns, mitigating the lack of historical data, while exogenous time series help capture trends among interrelated attributes. We perform an extensive ablation analysis on two large scale image fashion datasets, Mallzee and SHIFT15m to assess the adequacy of MuQAR and also use the Amazon Reviews: Home and Kitchen dataset to assess generalisability to other domains. A comparative study on the VISUELLE dataset, shows that MuQAR is capable of competing and surpassing the domain's current state of the art by 4.65% and 4.8% in terms of WAPE and MAE respectively.

Well Googled is Half Done: Multimodal Forecasting of New Fashion Product Sales with Image-based Google Trends

Abstract: New fashion product sales forecasting is a challenging problem that involves many business dynamics and cannot be solved by classical forecasting approaches. In this paper, we investigate the effectiveness of systematically probing exogenous knowledge in the form of Google Trends time series and combining it with multi-modal information related to a brand-new fashion item, in order to effectively forecast its sales despite the lack of past data. In particular, we propose a neural network-based approach, where an encoder learns a representation of the exogenous time series, while the decoder forecasts the sales based on the Google Trends encoding and the available visual and metadata information. Our model works in a non-autoregressive manner, avoiding the compounding effect of large first-step errors. As a second contribution, we present VISUELLE, a publicly available dataset for the task of new fashion product sales forecasting, containing multimodal information for 5577 real, new products sold between 2016-2019 from Nunalie, an Italian fast-fashion company. The dataset is equipped with images of products, metadata, related sales, and associated Google Trends. We use VISUELLE to compare our approach against state-of-the-art alternatives and several baselines, showing that our neural network-based approach is the most accurate in terms of both percentage and absolute error. It is worth noting that the addition of exogenous knowledge boosts the forecasting accuracy by 1.5% in terms of Weighted Absolute Percentage Error (WAPE), revealing the importance of exploiting informative external information. The code and dataset are both available at https://github.com/HumaticsLAB/GTM-Transformer.

POP: Mining POtential Performance of new fashion products via webly cross-modal query expansion

Abstract: We propose a data-centric pipeline able to generate exogenous observation data for the New Fashion Product Performance Forecasting (NFPPF) problem, i.e., predicting the performance of a brand-new clothing probe with no available past observations. Our pipeline manufactures the missing past starting from a single, available image of the clothing probe. It starts by expanding textual tags associated with the image, querying related fashionable or unfashionable images uploaded on the web at a specific time in the past. A binary classifier is robustly trained on these web images by confident learning, to learn what was fashionable in the past and how much the probe image conforms to this notion of fashionability. This compliance produces the POtential Performance (POP) time series, indicating how performing the probe could have been if it were available earlier. POP proves to be highly predictive for the probe's future performance, ameliorating the sales forecasts of all state-of-the-art models on the recent VISUELLE fast-fashion dataset. We also show that POP reflects the ground-truth popularity of new styles (ensembles of clothing items) on the Fashion Forward benchmark, demonstrating that our webly-learned signal is a truthful expression of popularity, accessible by everyone and generalizable to any time of analysis. Forecasting code, data and the POP time series are available at: https://github.com/HumaticsLAB/POP-Mining-POtential-Performance

New Product Demand Forecasting results on Visuelle in terms of Weighted Mean Absolute Percentage Error (WAPE) and Mean Absolute Error (MAE). The lower the better for both metrics.

A few issues that were not considered in the aforementioned papers, are the concept of trends and popularity, as well as a benchmark where the new product demand forecasting task could be tested. In {{cite:b738079}}, the authors tackle all of these problems, proposing a Transformer-based, multimodal architecture that can also process exogenous Google Trends signals. These signals bring the idea of the fashion expert judgment full circle, as now the model can reason based on the past performances of similar products in the dataset, as well as the online popularity of the different attributes that make up the product. Additionally, to avoid the error accumulation problem of autoregressive RNNs, the authors propose a generative approach where the whole demand series is forecasted in one step. Finally, the authors render their dataset, named Visuelle, public. This is an important aspect as it promotes further research on the methodological aspect of this challenging task, by relying on a dataset that is actually based on the real sales of a fast-fashion company. On Visuelle, the GTM-Transformer model of {{cite:b738079}} reports a wMAPE of 55,2%, compared to 59% of Cross-Attention Explainable RNN, as shown also in Table {{table:0e7332d2-04fb-4bb2-96fd-c81c14224284}} . In {{cite:f0aa019}} the authors introduce a data-centric pipeline able to generate a novel exogenous observation data, the POtential Performance (POP) signal, comparing directly the new product image with the images uploaded on the web in the past. The POP signal fed into the GTM-Transformer model represents the state-of-the-art on this task, with a wMAPE of 52,39%. In {{cite:73ea63d}} a novel Multimodal Quasi-AutoRegressive deep learning architecture (MuQAR) for the New Product Demand Forecasting task has been proposed. The model uses attributes and captions automatically extracted from the product image by the use of image classification and captioning methods. Attributes are used both to broaden the set of google trends and as input to the model together with the image captions. MuQAR model reports a wMAPE of 52,63% (Table {{table:0e7332d2-04fb-4bb2-96fd-c81c14224284}} ).


Dressing in Order: Recurrent Person Image Generation for Pose Transfer, Virtual Try-on and Outfit Editing

Abstract: We propose a flexible person generation framework called Dressing in Order (DiOr), which supports 2D pose transfer, virtual try-on, and several fashion editing tasks. The key to DiOr is a novel recurrent generation pipeline to sequentially put garments on a person, so that trying on the same garments in different orders will result in different looks. Our system can produce dressing effects not achievable by existing work, including different interactions of garments (e.g., wearing a top tucked into the bottom or over it), as well as layering of multiple garments of the same type (e.g., jacket over shirt over t-shirt). DiOr explicitly encodes the shape and texture of each garment, enabling these elements to be edited separately. Joint training on pose transfer and inpainting helps with detail preservation and coherence of generated garments. Extensive evaluations show that DiOr outperforms other recent methods like ADGAN in terms of output quality, and handles a wide range of editing functions for which there is no direct supervision.

Cross Attention Based Style Distribution for Controllable Person Image Synthesis

Abstract: Controllable person image synthesis task enables a wide range of applications through explicit control over body pose and appearance. In this paper, we propose a cross attention based style distribution module that computes between the source semantic styles and target pose for pose transfer. The module intentionally selects the style represented by each semantic and distributes them according to the target pose. The attention matrix in cross attention expresses the dynamic similarities between the target pose and the source styles for all semantics. Therefore, it can be utilized to route the color and texture from the source image, and is further constrained by the target parsing map to achieve a clearer objective. At the same time, to encode the source appearance accurately, the self attention among different semantic styles is also added. The effectiveness of our model is validated quantitatively and qualitatively on pose transfer and virtual try-on tasks.

Survey results for gauging the human preference of VGFlow over competing baselines. The percentage indicates the ratio of images which are voted to be better than the compared method.

We conduct a user study with 25 volunteers to further support the performance capabilities of VGFlow. Participants were shown 45 distinct result pairs sampled randomly from 8,570 test set results. Each pair consisted of one VGFlow result and the other sampled from the results of Dior/CASD baselines. These baselines were chosen to represent keypoint based flow and attention based methods respectively. As can be seen from Tab {{table:adfd1487-2f64-4148-8f64-d4ee856f1d45}} , VGFlow outperforms both the baselines in terms of preference by humans. We note that the majority of cases in which the users chose the baseline over VGFlow had minor segmentation issues in the invisible region, for which the humans are highly sensitive. These can be alleviated by incorporating the target segmentation mask in future.


Packet Transactions: High-Level Programming for Line-Rate Switches

Abstract: Many algorithms for congestion control, scheduling, network measurement, active queue management, and traffic engineering require custom processing of packets in the data plane of a network switch. To run at line rate, these data-plane algorithms must be implemented in hardware. With today's switch hardware, algorithms cannot be changed, nor new algorithms installed, after a switch has been built. This paper shows how to program data-plane algorithms in a high-level language and compile those programs into low-level microcode that can run on emerging programmable line-rate switching chips. The key challenge is that many data-plane algorithms create and modify algorithmic state. To achieve line-rate programmability for stateful algorithms, we introduce the notion of a packet transaction: a sequential packet-processing code block that is atomic and isolated from other such code blocks. We have developed this idea in Domino, a C-like imperative language to express data-plane algorithms. We show with many examples that Domino provides a convenient way to express sophisticated data-plane algorithms, and show that these algorithms can be run at line rate with modest estimated chip-area overhead.

Isolation Mechanisms for High-Speed Packet-Processing Pipelines

Abstract: Data-plane programmability is now mainstream. As we find more use cases, deployments need to be able to run multiple packet-processing modules in a single device. These are likely to be developed by independent teams, either within the same organization or from multiple organizations. Therefore, we need isolation mechanisms to ensure that modules on the same device do not interfere with each other. This paper presents Menshen, an extension of the Reconfigurable Match Tables (RMT) pipeline that enforces isolation between different packet-processing modules. Menshen is comprised of a set of lightweight hardware primitives and an extension to the open source P4-16 reference compiler that act in conjunction to meet this goal. We have prototyped Menshen on two FPGA platforms (NetFPGA and Corundum). We show that our design provides isolation, and allows new modules to be loaded without impacting the ones already running. Finally, we demonstrate that feasibility of implementing Menshen on ASICs by using the FreePDK45nm technology library and the Synopsys DC synthesis software, showing that our design meets timing at a 1GHz clock frequency and needs approximately 6% additional chip area. We have open sourced the code for Menshen's hardware and software at https://isolation.quest/.

To produce high quality implementations, CaT combines ideas from several prior projects (Table {{table:c7ad2737-e43a-4127-ae62-a702689cdf54}} ) into an end-to-end system for the first time.
The HLS workflow of CaT divides up the process of compiling a P4-16 program into three phases – their correspondence with traditional HLS steps {{cite:0eebb47}} and relation with prior work are highlighted in Table {{table:bc58cea7-5fcb-4054-a5fb-f0a023df40aa}} .
As shown, our resource transformation corresponds to the code transformation step in HLS, our resource synthesis corresponds to operation binding (or module selection {{cite:142df81}}) in HLS, and our resource allocation corresponds to the scheduling step in HLS.
To further improve the effectiveness of HLS in our setting, we customize these phases to RMT pipelines. Specifically, our resource transformations focus on scarce resources (pipeline stages, gateways). Our resource synthesis procedure leverages the strict requirement that stateful updates must fit in a single stage to ease the task of pipelining, which is often challenging for HLS in general.
Our resource allocation phase uses the result of synthesis on action blocks, to pack multiple action blocks in each pipeline stage.
Thus, while these phases build on top of well-known prior work including HLS efforts (Column 3), there are significant differences and novelties in our work (Column 4). The last column lists complementary techniques that could be combined in our compiler.