Show and Tell: A Neural Image Caption Generator

Abstract: Automatically describing the content of an image is a fundamental problem in artificial intelligence that connects computer vision and natural language processing. In this paper, we present a generative model based on a deep recurrent architecture that combines recent advances in computer vision and machine translation and that can be used to generate natural sentences describing an image. The model is trained to maximize the likelihood of the target description sentence given the training image. Experiments on several datasets show the accuracy of the model and the fluency of the language it learns solely from image descriptions. Our model is often quite accurate, which we verify both qualitatively and quantitatively. For instance, while the current state-of-the-art BLEU-1 score (the higher the better) on the Pascal dataset is 25, our approach yields 59, to be compared to human performance around 69. We also show BLEU-1 score improvements on Flickr30k, from 56 to 66, and on SBU, from 19 to 28. Lastly, on the newly released COCO dataset, we achieve a BLEU-4 of 27.7, which is the current state-of-the-art.

Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering

Abstract: Top-down visual attention mechanisms have been used extensively in image captioning and visual question answering (VQA) to enable deeper image understanding through fine-grained analysis and even multiple steps of reasoning. In this work, we propose a combined bottom-up and top-down attention mechanism that enables attention to be calculated at the level of objects and other salient image regions. This is the natural basis for attention to be considered. Within our approach, the bottom-up mechanism (based on Faster R-CNN) proposes image regions, each with an associated feature vector, while the top-down mechanism determines feature weightings. Applying this approach to image captioning, our results on the MSCOCO test server establish a new state-of-the-art for the task, achieving CIDEr / SPICE / BLEU-4 scores of 117.9, 21.5 and 36.9, respectively. Demonstrating the broad applicability of the method, applying the same approach to VQA we obtain first place in the 2017 VQA Challenge.

Knowing When to Look: Adaptive Attention via a Visual Sentinel for Image Captioning

Abstract: Attention-based neural encoder-decoder frameworks have been widely adopted for image captioning. Most methods force visual attention to be active for every generated word. However, the decoder likely requires little to no visual information from the image to predict non-visual words such as the and of. Other words that may seem visual can often be predicted reliably just from the language model e.g., sign after behind a red stop or phone following talking on a cell. In this paper, we propose a novel adaptive attention model with a visual sentinel. At each time step, our model decides whether to attend to the image (and if so, to which regions) or to the visual sentinel. The model decides whether to attend to the image and where, in order to extract meaningful information for sequential word generation. We test our method on the COCO image captioning 2015 challenge dataset and Flickr30K. Our approach sets the new state-of-the-art by a significant margin.

Self-critical Sequence Training for Image Captioning

Abstract: Recently it has been shown that policy-gradient methods for reinforcement learning can be utilized to train deep end-to-end systems directly on non-differentiable metrics for the task at hand. In this paper we consider the problem of optimizing image captioning systems using reinforcement learning, and show that by carefully optimizing our systems using the test metrics of the MSCOCO task, significant gains in performance can be realized. Our systems are built using a new optimization approach that we call self-critical sequence training (SCST). SCST is a form of the popular REINFORCE algorithm that, rather than estimating a "baseline" to normalize the rewards and reduce variance, utilizes the output of its own test-time inference algorithm to normalize the rewards it experiences. Using this approach, estimating the reward signal (as actor-critic methods must do) and estimating normalization (as REINFORCE algorithms typically do) is avoided, while at the same time harmonizing the model with respect to its test-time inference procedure. Empirically we find that directly optimizing the CIDEr metric with SCST and greedy decoding at test-time is highly effective. Our results on the MSCOCO evaluation sever establish a new state-of-the-art on the task, improving the best result in terms of CIDEr from 104.9 to 114.7.

Improving LSTM-based Video Description with Linguistic Knowledge Mined from Text

Abstract: This paper investigates how linguistic knowledge mined from large text corpora can aid the generation of natural language descriptions of videos. Specifically, we integrate both a neural language model and distributional semantics trained on large text corpora into a recent LSTM-based architecture for video description. We evaluate our approach on a collection of Youtube videos as well as two large movie description datasets showing significant improvements in grammaticality while modestly improving descriptive quality.

End-to-End Dense Video Captioning with Masked Transformer

Abstract: Dense video captioning aims to generate text descriptions for all events in an untrimmed video. This involves both detecting and describing events. Therefore, all previous methods on dense video captioning tackle this problem by building two models, i.e. an event proposal and a captioning model, for these two sub-problems. The models are either trained separately or in alternation. This prevents direct influence of the language description to the event proposal, which is important for generating accurate descriptions. To address this problem, we propose an end-to-end transformer model for dense video captioning. The encoder encodes the video into appropriate representations. The proposal decoder decodes from the encoding with different anchors to form video event proposals. The captioning decoder employs a masking network to restrict its attention to the proposal event over the encoding feature. This masking network converts the event proposal to a differentiable mask, which ensures the consistency between the proposal and captioning during training. In addition, our model employs a self-attention mechanism, which enables the use of efficient non-recurrent structure during encoding and leads to performance improvements. We demonstrate the effectiveness of this end-to-end model on ActivityNet Captions and YouCookII datasets, where we achieved 10.12 and 6.58 METEOR score, respectively.

A Reinforced Topic-Aware Convolutional Sequence-to-Sequence Model for Abstractive Text Summarization

Abstract: In this paper, we propose a deep learning approach to tackle the automatic summarization tasks by incorporating topic information into the convolutional sequence-to-sequence (ConvS2S) model and using self-critical sequence training (SCST) for optimization. Through jointly attending to topics and word-level alignment, our approach can improve coherence, diversity, and informativeness of generated summaries via a biased probability generation mechanism. On the other hand, reinforcement training, like SCST, directly optimizes the proposed model with respect to the non-differentiable metric ROUGE, which also avoids the exposure bias during inference. We carry out the experimental evaluation with state-of-the-art methods over the Gigaword, DUC-2004, and LCSTS datasets. The empirical results demonstrate the superiority of our proposed method in the abstractive summarization.

Adversarial Inference for Multi-Sentence Video Description

Abstract: While significant progress has been made in the image captioning task, video description is still in its infancy due to the complex nature of video data. Generating multi-sentence descriptions for long videos is even more challenging. Among the main issues are the fluency and coherence of the generated descriptions, and their relevance to the video. Recently, reinforcement and adversarial learning based methods have been explored to improve the image captioning models; however, both types of methods suffer from a number of issues, e.g. poor readability and high redundancy for RL and stability issues for GANs. In this work, we instead propose to apply adversarial techniques during inference, designing a discriminator which encourages better multi-sentence video description. In addition, we find that a multi-discriminator "hybrid" design, where each discriminator targets one aspect of a description, leads to the best results. Specifically, we decouple the discriminator to evaluate on three criteria: 1) visual relevance to the video, 2) language diversity and fluency, and 3) coherence across sentences. Our approach results in more accurate, diverse, and coherent multi-sentence video descriptions, as shown by automatic as well as human evaluation on the popular ActivityNet Captions dataset.

VideoBERT: A Joint Model for Video and Language Representation Learning

Abstract: Self-supervised learning has become increasingly important to leverage the abundance of unlabeled data available on platforms like YouTube. Whereas most existing approaches learn low-level representations, we propose a joint visual-linguistic model to learn high-level features without any explicit supervision. In particular, inspired by its recent success in language modeling, we build upon the BERT model to learn bidirectional joint distributions over sequences of visual and linguistic tokens, derived from vector quantization of video data and off-the-shelf speech recognition outputs, respectively. We use VideoBERT in numerous tasks, including action classification and video captioning. We show that it can be applied directly to open-vocabulary classification, and confirm that large amounts of training data and cross-modal information are critical to performance. Furthermore, we outperform the state-of-the-art on video captioning, and quantitative results verify that the model learns high-level semantic features.

ActBERT: Learning Global-Local Video-Text Representations

Abstract: In this paper, we introduce ActBERT for self-supervised learning of joint video-text representations from unlabeled data. First, we leverage global action information to catalyze the mutual interactions between linguistic texts and local regional objects. It uncovers global and local visual clues from paired video sequences and text descriptions for detailed visual and text relation modeling. Second, we introduce an ENtangled Transformer block (ENT) to encode three sources of information, i.e., global actions, local regional objects, and linguistic descriptions. Global-local correspondences are discovered via judicious clues extraction from contextual information. It enforces the joint videotext representation to be aware of fine-grained objects as well as global human intention. We validate the generalization capability of ActBERT on downstream video-and language tasks, i.e., text-video clip retrieval, video captioning, video question answering, action segmentation, and action step localization. ActBERT significantly outperforms the state-of-the-arts, demonstrating its superiority in video-text representation learning.

Less is More: ClipBERT for Video-and-Language Learning via Sparse Sampling

Abstract: The canonical approach to video-and-language learning (e.g., video question answering) dictates a neural model to learn from offline-extracted dense video features from vision models and text features from language models. These feature extractors are trained independently and usually on tasks different from the target domains, rendering these fixed features sub-optimal for downstream tasks. Moreover, due to the high computational overload of dense video features, it is often difficult (or infeasible) to plug feature extractors directly into existing approaches for easy finetuning. To provide a remedy to this dilemma, we propose a generic framework ClipBERT that enables affordable end-to-end learning for video-and-language tasks, by employing sparse sampling, where only a single or a few sparsely sampled short clips from a video are used at each training step. Experiments on text-to-video retrieval and video question answering on six datasets demonstrate that ClipBERT outperforms (or is on par with) existing methods that exploit full-length videos, suggesting that end-to-end learning with just a few sparsely sampled clips is often more accurate than using densely extracted offline features from full-length videos, proving the proverbial less-is-more principle. Videos in the datasets are from considerably different domains and lengths, ranging from 3-second generic domain GIF videos to 180-second YouTube human activity videos, showing the generalization ability of our approach. Comprehensive ablation studies and thorough analyses are provided to dissect what factors lead to this success. Our code is publicly available at https://github.com/jayleicn/ClipBERT

UniViLM: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generation

Abstract: With the recent success of the pre-training technique for NLP and image-linguistic tasks, some video-linguistic pre-training works are gradually developed to improve video-text related downstream tasks. However, most of the existing multimodal models are pre-trained for understanding tasks, leading to a pretrain-finetune discrepancy for generation tasks. This paper proposes UniVL: a Unified Video and Language pre-training model for both multimodal understanding and generation. It comprises four components, including two single-modal encoders, a cross encoder, and a decoder with the Transformer backbone. Five objectives, including video-text joint, conditioned masked language model (CMLM), conditioned masked frame model (CMFM), video-text alignment, and language reconstruction, are designed to train each of the components. We further develop two pre-training strategies, stage by stage pre-training (StagedP) and enhanced video representation (EnhancedV), to make the training process of the UniVL more effective. The pre-train is carried out on a sizeable instructional video dataset HowTo100M. Experimental results demonstrate that the UniVL can learn strong video-text representation and achieves state-of-the-art results on five downstream tasks.

Informative Visual Storytelling with Cross-modal Rules

Abstract: Existing methods in the Visual Storytelling field often suffer from the problem of generating general descriptions, while the image contains a lot of meaningful contents remaining unnoticed. The failure of informative story generation can be concluded to the model's incompetence of capturing enough meaningful concepts. The categories of these concepts include entities, attributes, actions, and events, which are in some cases crucial to grounded storytelling. To solve this problem, we propose a method to mine the cross-modal rules to help the model infer these informative concepts given certain visual input. We first build the multimodal transactions by concatenating the CNN activations and the word indices. Then we use the association rule mining algorithm to mine the cross-modal rules, which will be used for the concept inference. With the help of the cross-modal rules, the generated stories are more grounded and informative. Besides, our proposed method holds the advantages of interpretation, expandability, and transferability, indicating potential for wider application. Finally, we leverage these concepts in our encoder-decoder framework with the attention mechanism. We conduct several experiments on the VIsual StoryTelling~(VIST) dataset, the results of which demonstrate the effectiveness of our approach in terms of both automatic metrics and human evaluation. Additional experiments are also conducted showing that our mined cross-modal rules as additional knowledge helps the model gain better performance when trained on a small dataset.

Knowledge-Enriched Attention Network With Group-Wise Semantic for Visual Storytelling

Abstract: As a technically challenging topic, visual storytelling aims at generating an imaginary and coherent story with narrative multi-sentences from a group of relevant images. Existing methods often generate direct and rigid descriptions of apparent image-based contents, because they are not capable of exploring implicit information beyond images. Hence, these schemes could not capture consistent dependencies from holistic representation, impairing the generation of reasonable and fluent stories. To address these problems, a novel knowledge-enriched attention network with group-wise semantic model is proposed. Three main novel components are designed and supported by substantial experiments to reveal practical advantages. First, a knowledge-enriched attention network is designed to extract implicit concepts from external knowledge system, and these concepts are followed by a cascade cross-modal attention mechanism to characterize imaginative and concrete representations. Second, a group-wise semantic module with second-order pooling is developed to explore the globally consistent guidance. Third, a unified one-stage story generation model with encoder-decoder structure is proposed to simultaneously train and infer the knowledge-enriched attention network, group-wise semantic module and multi-modal story generation decoder in an end-to-end fashion. Substantial experiments on the visual storytelling datasets with both objective and subjective evaluation metrics demonstrate the superior performance of the proposed scheme as compared with other state-of-the-art methods. The source code of this work can be found in https://mic.tongji.edu.cn.

From Image to Text Generation

The image-based text generation aims at explaining or summarizing the visual concept of the given image, which mainly consists of three parts: image caption, video caption, and visual storytelling. The purpose of image captioning is to generate summaries from an image. Based on image captions, video caption aims to generate the summary of a series of images. Visual storytelling not only identifies the correlation between objects in a single picture but also gives the logical relationship between consecutive sequential images. It should be noted that the language generation component of VQA {{cite:b275a27}}, {{cite:82dfccf}}, {{cite:f147a64}} model is relatively similar to that of image caption. There exists the main distinction that current VQA systems {{cite:d72ae3b}}, {{cite:02e847a}}, {{cite:bcc4a46}}, {{cite:7e01388}}, {{cite:6f491d1}}, {{cite:83f7e31}} are focused on reasoning process and mainly designed to choose answers from a given candidate answer set, which is not quite related to the natural language generation. Several popular methods for each subtask are shown in Table {{table:08aa379e-aba0-4c2a-aecb-69143045cfc5}} .


Causal Autoregressive Flows

Abstract: Two apparently unrelated fields -- normalizing flows and causality -- have recently received considerable attention in the machine learning community. In this work, we highlight an intrinsic correspondence between a simple family of autoregressive normalizing flows and identifiable causal models. We exploit the fact that autoregressive flow architectures define an ordering over variables, analogous to a causal ordering, to show that they are well-suited to performing a range of causal inference tasks, ranging from causal discovery to making interventional and counterfactual predictions. First, we show that causal models derived from both affine and additive autoregressive flows with fixed orderings over variables are identifiable, i.e. the true direction of causal influence can be recovered. This provides a generalization of the additive noise model well-known in causal discovery. Second, we derive a bivariate measure of causal direction based on likelihood ratios, leveraging the fact that flow models can estimate normalized log-densities of data. Third, we demonstrate that flows naturally allow for direct evaluation of both interventional and counterfactual queries, the latter case being possible due to the invertible nature of flows. Finally, throughout a series of experiments on synthetic and real data, the proposed method is shown to outperform current approaches for causal discovery as well as making accurate interventional and counterfactual predictions.

Deep Structural Causal Models for Tractable Counterfactual Inference

Abstract: We formulate a general framework for building structural causal models (SCMs) with deep learning components. The proposed approach employs normalising flows and variational inference to enable tractable inference of exogenous noise variables - a crucial step for counterfactual inference that is missing from existing deep causal learning methods. Our framework is validated on a synthetic dataset built on MNIST as well as on a real-world medical dataset of brain MRI scans. Our experimental results indicate that we can successfully train deep SCMs that are capable of all three levels of Pearl's ladder of causation: association, intervention, and counterfactuals, giving rise to a powerful new approach for answering causal questions in imaging applications and beyond. The code for all our experiments is available at https://github.com/biomedia-mira/deepscm.

Counterfactual (Non-)identifiability of Learned Structural Causal Models

Abstract: Recent advances in probabilistic generative modeling have motivated learning Structural Causal Models (SCM) from observational datasets using deep conditional generative models, also known as Deep Structural Causal Models (DSCM). If successful, DSCMs can be utilized for causal estimation tasks, e.g., for answering counterfactual queries. In this work, we warn practitioners about non-identifiability of counterfactual inference from observational data, even in the absence of unobserved confounding and assuming known causal structure. We prove counterfactual identifiability of monotonic generation mechanisms with single dimensional exogenous variables. For general generation mechanisms with multi-dimensional exogenous variables, we provide an impossibility result for counterfactual identifiability, motivating the need for parametric assumptions. As a practical approach, we propose a method for estimating worst-case errors of learned DSCMs' counterfactual predictions. The size of this error can be an essential metric for deciding whether or not DSCMs are a viable approach for counterfactual inference in a specific problem setting. In evaluation, our method confirms negligible counterfactual errors for an identifiable SCM from prior work, and also provides informative error bounds on counterfactual errors for a non-identifiable synthetic SCM.

Diffusion Models for Causal Discovery via Topological Ordering

Abstract: Discovering causal relations from observational data becomes possible with additional assumptions such as considering the functional relations to be constrained as nonlinear with additive noise (ANM). Even with strong assumptions, causal discovery involves an expensive search problem over the space of directed acyclic graphs (DAGs). \emph{Topological ordering} approaches reduce the optimisation space of causal discovery by searching over a permutation rather than graph space. For ANMs, the \emph{Hessian} of the data log-likelihood can be used for finding leaf nodes in a causal graph, allowing its topological ordering. However, existing computational methods for obtaining the Hessian still do not scale as the number of variables and the number of samples increase. Therefore, inspired by recent innovations in diffusion probabilistic models (DPMs), we propose \emph{DiffAN}\footnote{Implementation is available at \url{https://github.com/vios-s/DiffAN} .}, a topological ordering algorithm that leverages DPMs for learning a Hessian function. We introduce theory for updating the learned Hessian without re-training the neural network, and we show that computing with a subset of samples gives an accurate approximation of the ordering, which allows scaling to datasets with more samples and variables. We show empirically that our method scales exceptionally well to datasets with up to $500$ nodes and up to $10^5$ samples while still performing on par over small datasets with state-of-the-art causal discovery methods. Implementation is available at https://github.com/vios-s/DiffAN .

Interventional and Counterfactual Inference with Diffusion Models

Abstract: We consider the problem of answering observational, interventional, and counterfactual queries in a causally sufficient setting where only observational data and the causal graph are available. Utilizing the recent developments in diffusion models, we introduce diffusion-based causal models (DCM) to learn causal mechanisms, that generate unique latent encodings. These encodings enable us to directly sample under interventions and perform abduction for counterfactuals. Diffusion models are a natural fit here, since they can encode each node to a latent representation that acts as a proxy for exogenous noise. Our empirical evaluations demonstrate significant improvements over existing state-of-the-art methods for answering causal queries. Furthermore, we provide theoretical results that offer a methodology for analyzing counterfactual estimation in general encoder-decoder models, which could be useful in settings beyond our proposed approach.

Estimating the Effects of Continuous-valued Interventions using Generative Adversarial Networks

Abstract: While much attention has been given to the problem of estimating the effect of discrete interventions from observational data, relatively little work has been done in the setting of continuous-valued interventions, such as treatments associated with a dosage parameter. In this paper, we tackle this problem by building on a modification of the generative adversarial networks (GANs) framework. Our model, SCIGAN, is flexible and capable of simultaneously estimating counterfactual outcomes for several different continuous interventions. The key idea is to use a significantly modified GAN model to learn to generate counterfactual outcomes, which can then be used to learn an inference model, using standard supervised methods, capable of estimating these counterfactuals for a new sample. To address the challenges presented by shifting to continuous interventions, we propose a novel architecture for our discriminator - we build a hierarchical discriminator that leverages the structure of the continuous intervention setting. Moreover, we provide theoretical results to support our use of the GAN framework and of the hierarchical discriminator. In the experiments section, we introduce a new semi-synthetic data simulation for use in the continuous intervention setting and demonstrate improvements over the existing benchmark models.

Summary of counterfactual inference leveraging DGMs.

Counterfactual Inference Leveraging DGMs

Counterfactual inference, positioned at the third level of Pearl's causal hierarchy, addresses questions like, "What would have happened to the patient if she had received a different treatment?".
However, this domain presents challenges due to the absence of data for counterfactual scenarios.
While there exist effective non-DGM methods for counterfactual inference {{cite:7bb11aa}}, {{cite:7f05b89}}, this section emphasizes the role of DGMs in enhancing counterfactual inference.
A summary of our discussion can be found in Table {{table:c5f93d66-fe85-47dc-ae18-818dfce42c6b}} .


Imperfect unbalanced detectors, detector dead time

Low operating temperature, limited signal-to-noise ratio, higher probability of false detection events

A Fast and Compact Quantum Random Number Generator

Abstract: We present the realization of a physical quantum random number generator based on the process of splitting a beam of photons on a beam splitter, a quantum mechanical source of true randomness. By utilizing either a beam splitter or a polarizing beam splitter, single photon detectors and high speed electronics the presented devices are capable of generating a binary random signal with an autocorrelation time of 11.8 ns and a continuous stream of random numbers at a rate of 1 Mbit/s. The randomness of the generated signals and numbers is shown by running a series of tests upon data samples. The devices described in this paper are built into compact housings and are simple to operate.

Detecting Single Infrared Photons with 93% System Efficiency

Abstract: Single-photon detectors (SPDs) at near infrared wavelengths with high system detection efficiency (> 90%), low dark count rate (< 1 counts per second, cps), low timing jitter (< 100 ps), and short reset time (< 100 ns) would enable landmark experiments in a variety of fields. Although some of the existing approaches to single-photon detection fulfill one or two of the above specifications, to date no detector has met all of the specifications simultaneously. Here we report on a fiber-coupled single-photon-detection system employing superconducting nanowire single photon detectors (SNSPDs) that closely approaches the ideal performance of SPDs. Our detector system has a system detection efficiency (SDE), including optical coupling losses, greater than 90% in the wavelength range \lambda = 1520-1610 nm; device dark count rate (measured with the device shielded from room-temperature blackbody radiation) of ~ 0.01 cps; timing jitter of ~ 150 ps FWHM; and reset time of 40 ns.

Fast nondeterministic random bit generator based on weakly correlated physical events

Abstract: Nondeterministic random bits are needed in many scientific fields. Unfortunately today’s computers are very limited in ability to produce them. We present here a method for extraction of nondeterministic random bits from random physics processes and one practical realization of a physical generator based on it. Even if processes are weakly correlateed the method is shown to deliver increasingly good randomness in the limit of slow sampling. A sample of approximately 109 bits produced by the physical generator prototype is subjected to a series of well-known statistical tests showing no weaknesses.

Summary of proposals for QRNGs

Several parameters of the quantum states of light feature inherent randomness, and therefore, optical sources have been widely adopted to design QRNGs, see Table {{table:c3f7dac5-b6a2-4be9-b58a-16c669e2cf4f}} .
These QRNGs are based on the premise that the quantum level of an optical field can be explained in terms of photons.
Generating and detecting uncorrelated single photons is, therefore, the goal of various technologies, including single-photon detectors {{cite:c3074bb}}, {{cite:c191d67}}, {{cite:983d252}}, silicon detectors {{cite:9a30981}}, superconducting nanowire single-photon detectors {{cite:3868b4a}}, {{cite:ec7e78b}}.
Nonetheless, some of the shortcomings of single-photon detectors are their limited capabilities to count the photons, infeasibility due to the high cost, and long dead-time {{cite:3e7d3db}}.

Non-optical Randomness Sources for QRNGs

Although optical randomness sources have been widely adopted in practice, proposals for non-optical QRNGs have been introduced, for instance, radioactive decay-based and noise-based QRNGsFor details on other classes of non-optical QRNGs, see {{cite:3e7d3db}}. (see Table {{table:c3f7dac5-b6a2-4be9-b58a-16c669e2cf4f}} ).
The former family is interesting due to the similarities between that and the optical QRNGs, whereas noise-based QRNGs are of great importance as commercial products have been developed in compliance with this principle, see, e.g., {{cite:92100f6}}.


Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering

Abstract: In this work, we are interested in generalizing convolutional neural networks (CNNs) from low-dimensional regular grids, where image, video and speech are represented, to high-dimensional irregular domains, such as social networks, brain connectomes or words' embedding, represented by graphs. We present a formulation of CNNs in the context of spectral graph theory, which provides the necessary mathematical background and efficient numerical schemes to design fast localized convolutional filters on graphs. Importantly, the proposed technique offers the same linear computational complexity and constant learning complexity as classical CNNs, while being universal to any graph structure. Experiments on MNIST and 20NEWS demonstrate the ability of this novel deep learning system to learn local, stationary, and compositional features on graphs.

Semi-Supervised Classification with Graph Convolutional Networks

Abstract: We present a scalable approach for semi-supervised learning on graph-structured data that is based on an efficient variant of convolutional neural networks which operate directly on graphs. We motivate the choice of our convolutional architecture via a localized first-order approximation of spectral graph convolutions. Our model scales linearly in the number of graph edges and learns hidden layer representations that encode both local graph structure and features of nodes. In a number of experiments on citation networks and on a knowledge graph dataset we demonstrate that our approach outperforms related methods by a significant margin.

Gated Graph Sequence Neural Networks

Abstract: Graph-structured data appears frequently in domains including chemistry, natural language semantics, social networks, and knowledge bases. In this work, we study feature learning techniques for graph-structured inputs. Our starting point is previous work on Graph Neural Networks (Scarselli et al., 2009), which we modify to use gated recurrent units and modern optimization techniques and then extend to output sequences. The result is a flexible and broadly useful class of neural network models that has favorable inductive biases relative to purely sequence-based models (e.g., LSTMs) when the problem is graph-structured. We demonstrate the capabilities on some simple AI (bAbI) and graph algorithm learning tasks. We then show it achieves state-of-the-art performance on a problem from program verification, in which subgraphs need to be matched to abstract data structures.

Inductive Representation Learning on Large Graphs

Abstract: Low-dimensional embeddings of nodes in large graphs have proved extremely useful in a variety of prediction tasks, from content recommendation to identifying protein functions. However, most existing approaches require that all nodes in the graph are present during training of the embeddings; these previous approaches are inherently transductive and do not naturally generalize to unseen nodes. Here we present GraphSAGE, a general, inductive framework that leverages node feature information (e.g., text attributes) to efficiently generate node embeddings for previously unseen data. Instead of training individual embeddings for each node, we learn a function that generates embeddings by sampling and aggregating features from a node's local neighborhood. Our algorithm outperforms strong baselines on three inductive node-classification benchmarks: we classify the category of unseen nodes in evolving information graphs based on citation and Reddit post data, and we show that our algorithm generalizes to completely unseen graphs using a multi-graph dataset of protein-protein interactions.

Large-Scale Learnable Graph Convolutional Networks

Abstract: Convolutional neural networks (CNNs) have achieved great success on grid-like data such as images, but face tremendous challenges in learning from more generic data such as graphs. In CNNs, the trainable local filters enable the automatic extraction of high-level features. The computation with filters requires a fixed number of ordered units in the receptive fields. However, the number of neighboring units is neither fixed nor are they ordered in generic graphs, thereby hindering the applications of convolutional operations. Here, we address these challenges by proposing the learnable graph convolutional layer (LGCL). LGCL automatically selects a fixed number of neighboring nodes for each feature based on value ranking in order to transform graph data into grid-like structures in 1-D format, thereby enabling the use of regular convolutional operations on generic graphs. To enable model training on large-scale graphs, we propose a sub-graph training method to reduce the excessive memory and computational resource requirements suffered by prior methods on graph convolutions. Our experimental results on node classification tasks in both transductive and inductive learning settings demonstrate that our methods can achieve consistently better performance on the Cora, Citeseer, Pubmed citation network, and protein-protein interaction network datasets. Our results also indicate that the proposed methods using sub-graph training strategy are more efficient as compared to prior approaches.

SplineCNN: Fast Geometric Deep Learning with Continuous B-Spline Kernels

Abstract: We present Spline-based Convolutional Neural Networks (SplineCNNs), a variant of deep neural networks for irregular structured and geometric input, e.g., graphs or meshes. Our main contribution is a novel convolution operator based on B-splines, that makes the computation time independent from the kernel size due to the local support property of the B-spline basis functions. As a result, we obtain a generalization of the traditional CNN convolution operator by using continuous kernel functions parametrized by a fixed number of trainable weights. In contrast to related approaches that filter in the spectral domain, the proposed method aggregates features purely in the spatial domain. In addition, SplineCNN allows entire end-to-end training of deep architectures, using only the geometric structure as input, instead of handcrafted feature descriptors. For validation, we apply our method on tasks from the fields of image graph classification, shape correspondence and graph node classification, and show that it outperforms or pars state-of-the-art approaches while being significantly faster and having favorable properties like domain-independence. Our source code is available on GitHub1.

Graph Attention Networks

Abstract: We present graph attention networks (GATs), novel neural network architectures that operate on graph-structured data, leveraging masked self-attentional layers to address the shortcomings of prior methods based on graph convolutions or their approximations. By stacking layers in which nodes are able to attend over their neighborhoods' features, we enable (implicitly) specifying different weights to different nodes in a neighborhood, without requiring any kind of costly matrix operation (such as inversion) or depending on knowing the graph structure upfront. In this way, we address several key challenges of spectral-based graph neural networks simultaneously, and make our model readily applicable to inductive as well as transductive problems. Our GAT models have achieved or matched state-of-the-art results across four established transductive and inductive graph benchmarks: the Cora, Citeseer and Pubmed citation network datasets, as well as a protein-protein interaction dataset (wherein test graphs remain unseen during training).

Variational Graph Auto-Encoders

Abstract: We introduce the variational graph auto-encoder (VGAE), a framework for unsupervised learning on graph-structured data based on the variational auto-encoder (VAE). This model makes use of latent variables and is capable of learning interpretable latent representations for undirected graphs. We demonstrate this model using a graph convolutional network (GCN) encoder and a simple inner product decoder. Our model achieves competitive results on a link prediction task in citation networks. In contrast to most existing models for unsupervised learning on graph-structured data and link prediction, our model can naturally incorporate node features, which significantly improves predictive performance on a number of benchmark datasets.

Spatio-temporal Graph Convolutional Neural Network: A Deep Learning Framework for Traffic Forecasting

Abstract: Timely accurate traffic forecast is crucial for urban traffic control and guidance. Due to the high nonlinearity and complexity of traffic flow, traditional methods cannot satisfy the requirements of mid-and-long term prediction tasks and often neglect spatial and temporal dependencies. In this paper, we propose a novel deep learning framework, Spatio-Temporal Graph Convolutional Networks (STGCN), to tackle the time series prediction problem in traffic domain. Instead of applying regular convolutional and recurrent units, we formulate the problem on graphs and build the model with complete convolutional structures, which enable much faster training speed with fewer parameters. Experiments show that our model STGCN effectively captures comprehensive spatio-temporal correlations through modeling multi-scale traffic networks and consistently outperforms state-of-the-art baselines on various real-world traffic datasets.

Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition

Abstract: Dynamics of human body skeletons convey significant information for human action recognition. Conventional approaches for modeling skeletons usually rely on hand-crafted parts or traversal rules, thus resulting in limited expressive power and difficulties of generalization. In this work, we propose a novel model of dynamic skeletons called Spatial-Temporal Graph Convolutional Networks (ST-GCN), which moves beyond the limitations of previous methods by automatically learning both the spatial and temporal patterns from data. This formulation not only leads to greater expressive power but also stronger generalization capability. On two large datasets, Kinetics and NTU-RGBD, it achieves substantial improvements over mainstream methods.

Structural-RNN: Deep Learning on Spatio-Temporal Graphs

Abstract: Deep Recurrent Neural Network architectures, though remarkably capable at modeling sequences, lack an intuitive high-level spatio-temporal structure. That is while many problems in computer vision inherently have an underlying high-level structure and can benefit from it. Spatio-temporal graphs are a popular tool for imposing such high-level intuitions in the formulation of real world problems. In this paper, we propose an approach for combining the power of high-level spatio-temporal graphs and sequence learning success of Recurrent Neural Networks~(RNNs). We develop a scalable method for casting an arbitrary spatio-temporal graph as a rich RNN mixture that is feedforward, fully differentiable, and jointly trainable. The proposed method is generic and principled as it can be used for transforming any spatio-temporal graph through employing a certain set of well defined steps. The evaluations of the proposed approach on a diverse set of problems, ranging from modeling human motion to object interactions, shows improvement over the state-of-the-art with a large margin. We expect this method to empower new approaches to problem formulation through high-level spatio-temporal graphs and Recurrent Neural Networks.

A Summary of Open-source Implementations

Benchmarks & Open-source Implementations

Of the datasets listed in Table {{table:3cf22704-e1a9-4d79-b9c1-1dac3fd590ef}} , Cora, Pubmed, Citeseer, and PPI are the most frequently used datasets. They are often tested to compare the performance of graph convolution networks in node classification tasks. In Table {{table:06be4d80-e187-45b0-92cb-a0fbfffed445}} , we report the benchmark performance of these four datasets, all of which use standard data splits.
Open-source implementations facilitate the work of baseline experiments in deep learning research. Due to the vast number of hyper-parameters, it is difficult to achieve the same results as reported in the literature without using published codes. In Table {{table:ccf91cfd-6596-4889-8692-c950cef7a322}} , we provide the hyperlinks of open-source implementations of the graph neural network models reviewed in Section 4-5. Noticeably, Fey et al. {{cite:aea63db}} published a geometric learning library in PyTorch named PyTorch Geometric https://github.com/rusty1s/pytorch_geometric, which implements several graph neural networks including ChebNet {{cite:9024374}}, 1stChebNet {{cite:746e1e9}}, GraphSage {{cite:955d7ce}}, MPNNs {{cite:0eac96b}}, GAT {{cite:7b30319}} and SplineCNN {{cite:aea63db}}. Most recently, the Deep Graph Library (DGL) https://www.dgl.ai/ is released which provides a fast implementation of many graph neural networks with a set of functions on top of popular deep learning platforms such as PyTorch and MXNet.


VAE branch is added to regualrize decoder. The winning contribution of BRATS2018 challenge.

Captures the global context of MR slices.

VAE was used for regularization to address over-fitting

A knowledge transfer method based on Bayesian generative models for training with small labeled data sets.

Based on the belief function theory, where the architecture is composed of a light UNet (LUNet) and an evidential neural network (ENN).

Modality-adaptive method in which convolutional restricted Boltzmann machines are used to model the tumor shape.

3D MRI brain tumor segmentation using autoencoder regularization

Abstract: Automated segmentation of brain tumors from 3D magnetic resonance images (MRIs) is necessary for the diagnosis, monitoring, and treatment planning of the disease. Manual delineation practices require anatomical knowledge, are expensive, time consuming and can be inaccurate due to human error. Here, we describe a semantic segmentation network for tumor subregion segmentation from 3D MRIs based on encoder-decoder architecture. Due to a limited training dataset size, a variational auto-encoder branch is added to reconstruct the input image itself in order to regularize the shared decoder and impose additional constraints on its layers. The current approach won 1st place in the BraTS 2018 challenge.

Unsupervised Detection of Lesions in Brain MRI using constrained adversarial auto-encoders

Abstract: Lesion detection in brain Magnetic Resonance Images (MRI) remains a challenging task. State-of-the-art approaches are mostly based on supervised learning making use of large annotated datasets. Human beings, on the other hand, even non-experts, can detect most abnormal lesions after seeing a handful of healthy brain images. Replicating this capability of using prior information on the appearance of healthy brain structure to detect lesions can help computers achieve human level abnormality detection, specifically reducing the need for numerous labeled examples and bettering generalization of previously unseen lesions. To this end, we study detection of lesion regions in an unsupervised manner by learning data distribution of brain MRI of healthy subjects using auto-encoder based methods. We hypothesize that one of the main limitations of the current models is the lack of consistency in latent representation. We propose a simple yet effective constraint that helps mapping of an image bearing lesion close to its corresponding healthy image in the latent space. We use the Human Connectome Project dataset to learn distribution of healthy-appearing brain MRI and report improved detection, in terms of AUC, of the lesions in the BRATS challenge dataset.

Deep Autoencoding Models for Unsupervised Anomaly Segmentation in Brain MR Images

Abstract: Reliably modeling normality and differentiating abnormal appearances from normal cases is a very appealing approach for detecting pathologies in medical images. A plethora of such unsupervised anomaly detection approaches has been made in the medical domain, based on statistical methods, content-based retrieval, clustering and recently also deep learning. Previous approaches towards deep unsupervised anomaly detection model patches of normal anatomy with variants of Autoencoders or GANs, and detect anomalies either as outliers in the learned feature space or from large reconstruction errors. In contrast to these patch-based approaches, we show that deep spatial autoencoding models can be efficiently used to capture normal anatomical variability of entire 2D brain MR images. A variety of experiments on real MR data containing MS lesions corroborates our hypothesis that we can detect and even delineate anomalies in brain MR images by simply comparing input images to their reconstruction. Results show that constraints on the latent space and adversarial training can further improve the segmentation performance over standard deep representation learning.

A Two-Stage Cascade Model with Variational Autoencoders and Attention Gates for MRI Brain Tumor Segmentation

Abstract: Automatic MRI brain tumor segmentation is of vital importance for the disease diagnosis, monitoring, and treatment planning. In this paper, we propose a two-stage encoder-decoder based model for brain tumor subregional segmentation. Variational autoencoder regularization is utilized in both stages to prevent the overfitting issue. The second-stage network adopts attention gates and is trained additionally using an expanded dataset formed by the first-stage outputs. On the BraTS 2020 validation dataset, the proposed method achieves the mean Dice score of 0.9041, 0.8350, and 0.7958, and Hausdorff distance (95%) of 4.953, 6.299, and 23.608 for the whole tumor, tumor core, and enhancing tumor, respectively. The corresponding results on the BraTS 2020 testing dataset are 0.8729, 0.8357, and 0.8205 for Dice score, and 11.4288, 19.9690, and 15.6711 for Hausdorff distance. The code is publicly available at https://github.com/shu-hai/two-stage-VAE-Attention-gate-BraTS2020.

Bayesian Generative Models for Knowledge Transfer in MRI Semantic Segmentation Problems

Abstract: Automatic segmentation methods based on deep learning have recently demonstrated state-of-the-art performance, outperforming the ordinary methods. Nevertheless, these methods are inapplicable for small datasets, which are very common in medical problems. To this end, we propose a knowledge transfer method between diseases via the Generative Bayesian Prior network. Our approach is compared to a pre-train approach and random initialization and obtains the best results in terms of Dice Similarity Coefficient metric for the small subsets of the Brain Tumor Segmentation 2018 database (BRATS2018).

Belief function-based semi-supervised learning for brain tumor segmentation

Abstract: Precise segmentation of a lesion area is important for optimizing its treatment. Deep learning makes it possible to detect and segment a lesion field using annotated data. However, obtaining precisely annotated data is very challenging in the medical domain. Moreover, labeling uncertainty and imprecision make segmentation results unreliable. In this paper, we address the uncertain boundary problem by a new evidential neural network with an information fusion strategy, and the scarcity of annotated data by semi-supervised learning. Experimental results show that our proposal has better performance than state-of-the-art methods.

A Modality-Adaptive Method for Segmenting Brain Tumors and Organs-at-Risk in Radiation Therapy Planning

Abstract: In this paper we present a method for simultaneously segmenting brain tumors and an extensive set of organs-at-risk for radiation therapy planning of glioblastomas. The method combines a contrast-adaptive generative model for whole-brain segmentation with a new spatial regularization model of tumor shape using convolutional restricted Boltzmann machines. We demonstrate experimentally that the method is able to adapt to image acquisitions that differ substantially from any available training data, ensuring its applicability across treatment sites; that its tumor segmentation accuracy is comparable to that of the current state of the art; and that it captures most organs-at-risk sufficiently well for radiation therapy planning purposes. The proposed method may be a valuable step towards automating the delineation of brain tumors and organs-at-risk in glioblastoma patients undergoing radiation therapy.

PSIGAN: Joint probabilistic segmentation and image distribution matching for unpaired cross-modality adaptation based MRI segmentation

Abstract: We developed a new joint probabilistic segmentation and image distribution matching generative adversarial network (PSIGAN) for unsupervised domain adaptation (UDA) and multi-organ segmentation from magnetic resonance (MRI) images. Our UDA approach models the co-dependency between images and their segmentation as a joint probability distribution using a new structure discriminator. The structure discriminator computes structure of interest focused adversarial loss by combining the generated pseudo MRI with probabilistic segmentations produced by a simultaneously trained segmentation sub-network. The segmentation sub-network is trained using the pseudo MRI produced by the generator sub-network. This leads to a cyclical optimization of both the generator and segmentation sub-networks that are jointly trained as part of an end-to-end network. Extensive experiments and comparisons against multiple state-of-the-art methods were done on four different MRI sequences totalling 257 scans for generating multi-organ and tumor segmentation. The experiments included, (a) 20 T1-weighted (T1w) in-phase mdixon and (b) 20 T2-weighted (T2w) abdominal MRI for segmenting liver, spleen, left and right kidneys, (c) 162 T2-weighted fat suppressed head and neck MRI (T2wFS) for parotid gland segmentation, and (d) 75 T2w MRI for lung tumor segmentation. Our method achieved an overall average DSC of 0.87 on T1w and 0.90 on T2w for the abdominal organs, 0.82 on T2wFS for the parotid glands, and 0.77 on T2w MRI for lung tumors.

Applications of traditional methods combined with deep learning in tumor segmentation

The research area covered in this section entails probabilistic deep learning and the applications of deep/statistical hybrid methods in medical image segmentation, summarised in Table {{table:4990f104-7769-4677-a548-e24ba4224733}} . Probabilistic deep learning has been discussed under two approaches: deep probabilistic models and probabilistic neural networks. The limitations of the current state-of-the-art can be addressed by unifying traditional models with newer deep learning approaches. In summary, it could be argued that the future path for medical decisions with computer-aided systems lies in the merging of traditional statistics with deep learning.


Task completion ratio and model accuracy/Offloading redundancy, task assignment, beam selection

Model accuracy and energy consumption/Number of local iterations

Sustainable AIGC service provisioning with mobility management

Provision AIGC services via a network of UAVs

MEET: Mobility-Enhanced Edge inTelligence for Smart and Green 6G Networks

Abstract: Edge intelligence is an emerging paradigm for real-time training and inference at the wireless edge, thus enabling mission-critical applications. Accordingly, base stations (BSs) and edge servers (ESs) need to be densely deployed, leading to huge deployment and operation costs, in particular energy costs. In this article, we propose a new framework called Mobility-Enhanced Edge inTelligence (MEET), which exploits the sensing, communication, computing, and self-powering capabilities of intelligent connected vehicles for smart and green 6G networks. Specifically, the operators can incorporate infrastructural vehicles as movable BSs or ESs, and schedule them in a more flexible way to align with communication and computation traffic fluctuations. Meanwhile, the remaining compute resources of opportunistic vehicles are exploited for edge training and inference, where mobility can further enhance edge intelligence by bringing more compute resources, communication opportunities, and diverse data. In this way, the deployment and operation costs are spread over the vastly available vehicles so that edge intelligence is realized cost-effectively and sustainably. Furthermore, these vehicles can be either powered by renewable energy to reduce carbon emissions or charged more flexibly during off-peak hours to cut electricity bills.

UAVs as a Service: Boosting Edge Intelligence for Air-Ground Integrated Networks

Abstract: The air-ground integrated network is a key component of future sixth generation (6G) networks to support seamless and near-instant super-connectivity. There is a pressing need to intelligently provision various services in 6G networks, which however is challenging. To meet this need, in this article, we propose a novel architecture called UaaS (UAVs as a Service) for the air-ground integrated network, featuring UAV as a key enabler to boost edge intelligence with the help of machine learning (ML) techniques. We envision that the proposed UaaS architecture could intelligently provision wireless communication service, edge computing service, and edge caching service by a network of UAVs, making full use of UAVs' flexible deployment and diverse ML techniques. We also conduct a case study where UAVs participate in the model training of distributed ML among multiple terrestrial users, whose result shows that the model training is efficient with a negligible energy consumption of UAVs, compared to the flight energy consumption. Finally, we discuss the challenges and open research issues in the UaaS.

Summary of scenarios, problems, benefits/challenges, and mathematical tools for mobility management.

As summarized in Table {{table:5257075d-dc02-4f11-a049-5970b3eb6ca6}} , here are several KPIs for mobility management in AIGC networks:


Mitotic Figure Recognition: Agreement among Pathologists and Computerized Detector

Abstract: Despite the prognostic importance of mitotic count as one of the components of the Bloom – Richardson grade [3], several studies ([2, 9, 10]) have found that pathologists’ agreement on the mitotic grade is fairly modest. Collecting a set of more than 4,200 candidate mitotic figures, we evaluate pathologists' agreement on individual figures, and train a computerized system for mitosis detection, comparing its performance to the classifications of three pathologists. The system’s and the pathologists’ classifications are based on evaluation of digital micrographs of hematoxylin and eosin stained breast tissue. On figures where the majority of pathologists agree on a classification, we compare the performance of the trained system to that of the individual pathologists. We find that the level of agreement of the pathologists ranges from slight to moderate, with strong biases, and that the system performs competitively in rating the ground truth set. This study is a step towards automatic mitosis count to accelerate a pathologist's work and improve reproducibility.

The importance of stain normalization in colorectal tissue classification with convolutional networks

Abstract: The development of reliable imaging biomarkers for the analysis of colorectal cancer (CRC) in hematoxylin and eosin (H&E) stained histopathology images requires an accurate and reproducible classification of the main tissue components in the image. In this paper, we propose a system for CRC tissue classification based on convolutional networks (ConvNets). We investigate the importance of stain normalization in tissue classification of CRC tissue samples in H&E-stained images. Furthermore, we report the performance of ConvNets on a cohort of rectal cancer samples and on an independent publicly available dataset of colorectal H&E images.

Segmentation and classification of colon glands with deep convolutional neural networks and total variation regularization

Abstract: Segmentation of histopathology sections is a necessary preprocessing step for digital pathology. Due to the large variability of biological tissue, machine learning techniques have shown superior performance over conventional image processing methods. Here we present our deep neural network-based approach for segmentation and classification of glands in tissue of benign and malignant colorectal cancer, which was developed to participate in the GlaS@MICCAI2015 colon gland segmentation challenge. We use two distinct deep convolutional neural networks (CNN) for pixel-wise classification of Hematoxylin-Eosin stained images. While the first classifier separates glands from background, the second classifier identifies gland-separating structures. In a subsequent step, a figure-ground segmentation based on weighted total variation produces the final segmentation result by regularizing the CNN predictions. We present both quantitative and qualitative segmentation results on the recently released and publicly available Warwick-QU colon adenocarcinoma dataset associated with the GlaS@MICCAI2015 challenge and compare our approach to the simultaneously developed other approaches that participated in the same challenge. On two test sets, we demonstrate our segmentation performance and show that we achieve a tissue classification accuracy of 98% and 95%, making use of the inherent capability of our system to distinguish between benign and malignant tissue. Our results show that deep learning approaches can yield highly accurate and reproducible results for biomedical image analysis, with the potential to significantly improve the quality and speed of medical diagnoses.

Gland Instance Segmentation Using Deep Multichannel Neural Networks

Abstract: Objective: A new image instance segmentation method is proposed to segment individual glands (instances) in colon histology images. This process is challenging since the glands not only need to be segmented from a complex background, they must also be individually identified. Methods: We leverage the idea of image-to-image prediction in recent deep learning by designing an algorithm that automatically exploits and fuses complex multichannel information - regional, location, and boundary cues - in gland histology images. Our proposed algorithm, a deep multichannel framework, alleviates heavy feature design due to the use of convolutional neural networks and is able to meet multifarious requirements by altering channels. Results: Compared with methods reported in the 2015 MICCAI Gland Segmentation Challenge and other currently prevalent instance segmentation methods, we observe state-of-the-art results based on the evaluation metrics. Conclusion: The proposed deep multichannel algorithm is an effective method for gland instance segmentation. Significance: The generalization ability of our model not only enable the algorithm to solve gland instance segmentation problems, but the channel is also alternative that can be replaced for a specific task.

Sparse Autoencoder for Unsupervised Nucleus Detection and Representation in Histopathology Images

Abstract: Histopathology images are crucial to the study of complex diseases such as cancer. The histologic characteristics of nuclei play a key role in disease diagnosis, prognosis and analysis. In this work, we propose a sparse Convolutional Autoencoder (CAE) for fully unsupervised, simultaneous nucleus detection and feature extraction in histopathology tissue images. Our CAE detects and encodes nuclei in image patches in tissue images into sparse feature maps that encode both the location and appearance of nuclei. Our CAE is the first unsupervised detection network for computer vision applications. The pretrained nucleus detection and feature extraction modules in our CAE can be fine-tuned for supervised learning in an end-to-end fashion. We evaluate our method on four datasets and reduce the errors of state-of-the-art methods up to 42%. We are able to achieve comparable performance with only 5% of the fully-supervised annotation cost.

Classification of Histopathological Biopsy Images Using Ensemble of Deep Learning Networks

Abstract: Breast cancer is one of the leading causes of death across the world in women. Early diagnosis of this type of cancer is critical for treatment and patient care. Computer-aided detection (CAD) systems using convolutional neural networks (CNN) could assist in the classification of abnormalities. In this study, we proposed an ensemble deep learning-based approach for automatic binary classification of breast histology images. The proposed ensemble model adapts three pre-trained CNNs, namely VGG19, MobileNet, and DenseNet. The ensemble model is used for the feature representation and extraction steps. The extracted features are then fed into a multi-layer perceptron classifier to carry out the classification task. Various pre-processing and CNN tuning techniques such as stain-normalization, data augmentation, hyperparameter tuning, and fine-tuning are used to train the model. The proposed method is validated on four publicly available benchmark datasets, i.e., ICIAR, BreakHis, PatchCamelyon, and Bioimaging. The proposed multi-model ensemble method obtains better predictions than single classifiers and machine learning algorithms with accuracies of 98.13%, 95.00%, 94.64% and 83.10% for BreakHis, ICIAR, PatchCamelyon and Bioimaging datasets, respectively.

Texture CNN for Histopathological Image Classification

Abstract: Biopsies are the gold standard for breast cancer diagnosis. This task can be improved by the use of Computer Aided Diagnosis (CAD) systems, reducing the time of diagnosis and reducing the inter and intra-observer variability. The advances in computing have brought this type of system closer to reality. However, datasets of Histopathological Images (HI) from biopsies are quite small and unbalanced what makes difficult to use modern machine learning techniques such as deep learning. In this paper we propose a compact architecture based on texture filters that has fewer parameters than traditional deep models but is able to capture the difference between malignant and benign tissues with relative accuracy. The experimental results on the BreakHis dataset have show that the proposed texture CNN achieves almost 90% of accuracy for classifying benign and malignant tissues.

Double Transfer Learning for Breast Cancer Histopathologic Image Classification

Abstract: This work proposes a classification approach for breast cancer histopathologic images (HI) that uses transfer learning to extract features from HI using an Inception-v3 CNN pre-trained with ImageNet dataset. We also use transfer learning on training a support vector machine (SVM) classifier on a tissue labeled colorectal cancer dataset aiming to filter the patches from a breast cancer HI and remove the irrelevant ones. We show that removing irrelevant patches before training a second SVM classifier, improves the accuracy for classifying malign and benign tumors on breast cancer images. We are able to improve the classification accuracy in 3.7% using the feature extraction transfer learning and an additional 0.7% using the irrelevant patch elimination. The proposed approach outperforms the state-of-the-art in three out of the four magnification factors of the breast cancer dataset.

Data Augmentation for Histopathological Images Based on Gaussian-Laplacian Pyramid Blending

Abstract: Data imbalance is a major problem that affects several machine learning (ML) algorithms. Such a problem is troublesome because most of the ML algorithms attempt to optimize a loss function that does not take into account the data imbalance. Accordingly, the ML algorithm simply generates a trivial model that is biased toward predicting the most frequent class in the training data. In the case of histopathologic images (HIs), both low-level and high-level data augmentation (DA) techniques still present performance issues when applied in the presence of inter-patient variability; whence the model tends to learn color representations, which is related to the staining process. In this paper, we propose a novel approach capable of not only augmenting HI dataset but also distributing the inter-patient variability by means of image blending using the Gaussian-Laplacian pyramid. The proposed approach consists of finding the Gaussian pyramids of two images of different patients and finding the Laplacian pyramids thereof. Afterwards, the left-half side and the right-half side of different HIs are joined in each level of the Laplacian pyramid, and from the joint pyramids, the original image is reconstructed. This composition combines the stain variation of two patients, avoiding that color differences mislead the learning process. Experimental results on the BreakHis dataset have shown promising gains vis-a-vis the majority of DA techniques presented in the literature.

Histopathological Classification of Breast Cancer Images Using a Multi-Scale Input and Multi-Feature Network

Abstract: Diagnosis of pathologies using histopathological images can be time-consuming when many images with different magnification levels need to be analyzed. State-of-the-art computer vision and machine learning methods can help automate the diagnostic pathology workflow and thus reduce the analysis time. Automated systems can also be more efficient and accurate, and can increase the objectivity of diagnosis by reducing operator variability. We propose a multi-scale input and multi-feature network (MSI-MFNet) model, which can learn the overall structures and texture features of different scale tissues by fusing multi-resolution hierarchical feature maps from the network’s dense connectivity structure. The MSI-MFNet predicts the probability of a disease on the patch and image levels. We evaluated the performance of our proposed model on two public benchmark datasets. Furthermore, through ablation studies of the model, we found that multi-scale input and multi-feature maps play an important role in improving the performance of the model. Our proposed model outperformed the existing state-of-the-art models by demonstrating better accuracy, sensitivity, and specificity.

Summary of publications using DL methods in HI analysis.

DL methods are gaining the attention of the scientific community due to recent achievements to solve complex machine learning problems on large datasets. A convolutional neural network (CNN) is able to learn in a single optimization process, both a representation and a decision boundary. However, CNNs usually require a huge amount of data for adequate training in order to avoid overfitting problems, but most of the HI datasets have only a few patients and hundreds of images. Data augmentation {{cite:7735566}} {{cite:2fd1b79}} and transfer learning {{cite:c53a543}} are two possible approaches to circumvent the lack of data in HI datasets. For instance, ImageNet, which has more than 14 million images, is one of the most common datasets used for training CNNs for object recognition. Data augmentation generates new HIs from existing ones by using affine transformations or morphological operations. Another common way of data augmentation is patching HIs, which consists in producing the effect of selecting pieces of a HI with the same structure but that belong to different classes. On the other hand, the transfer learning method reuses CNNs previously trained in large datasets, which usually belongs to a different domain from the target problem. The pre-trained CNNs can be used in two ways: to extract features from HIs and use these features with shallow classifiers, as already described in Sections  and ; to fine-tune such CNNs on an HI dataset, which means that filters learned on a large dataset will be adapted to the HI dataset. Recently, DL methods have been employed in HI analysis. Table {{table:69a95953-e416-4af2-82e5-21d673b98a64}}  summarizes the works reviewed in this section in terms of network architecture, tissue or organ from where the HI was obtained and the publication year.

In the years preceding the wide adoption of DL methods, several works had focused on identifying nuclei in HIs, which are important structures to cancer diagnosis. Therefore, that lead to the exploitation of different segmentation approaches as reviewed in Section . Some works used the concept of semantic features, based on the e.g. counting of nuclei, its relation to the stroma, the distance between nuclei. Stain normalization is also a recurrent topic that has appeared in several works across the years covered by this review. Such an image processing method, which reduces the color and intensity variations present in stained images, has been widely used even in conjunction with DL methods.
Feature extraction methods were the focus of interest of researchers between 2008 and 2016. Morphometric feature and textural features such as GLCM, LBP and their variants have been the most frequent features used in HI analysis, either alone or in combination with other feature types. It is important to note that the shallow classifiers require a feature extraction method. Again, the adoption of DL methods, which are able to learn representation and decision boundaries in a single optimization process, is probably the main cause of declining interest in feature extraction methods from 2016. Furthermore, pre-trained CNNs can also be used as feature extractors for HIs. Several works removed the fully connected layers of pre-trained CNNs and used the output of the last convolutional layer as feature vectors to feed shallow classifiers.
Comparing Tables {{table:5ec7fb98-52c0-4d56-a2a2-ce4aa71b95ca}} , {{table:520941b5-9a93-4a23-a28c-7a8ad7b06038}}  and {{table:69a95953-e416-4af2-82e5-21d673b98a64}}  we can say that DL approaches are becoming prevalent over shallow approaches in the last five years. Although studies are still necessary for understanding how these networks learn data representation, especially with respect to HIs.


Automated Hate Speech Detection and the Problem of Offensive Language

Abstract: A key challenge for automatic hate-speech detection on social media is the separation of hate speech from other instances of offensive language. Lexical detection methods tend to have low precision because they classify all messages containing particular terms as hate speech and previous work using supervised learning has failed to distinguish between the two categories. We used a crowd-sourced hate speech lexicon to collect tweets containing hate speech keywords. We use crowd-sourcing to label a sample of these tweets into three categories: those containing hate speech, only offensive language, and those with neither. We train a multi-class classifier to distinguish between these different categories. Close analysis of the predictions and the errors shows when we can reliably separate hate speech from other offensive language and when this differentiation is more difficult. We find that racist and homophobic tweets are more likely to be classified as hate speech but that sexist tweets are generally classified as offensive. Tweets without explicit hate keywords are also more difficult to classify.

Character-level Convolutional Networks for Text Classification

Abstract: This article offers an empirical exploration on the use of character-level convolutional networks (ConvNets) for text classification. We constructed several large-scale datasets to show that character-level convolutional networks could achieve state-of-the-art or competitive results. Comparisons are offered against traditional models such as bag of words, n-grams and their TFIDF variants, and deep learning models such as word-based ConvNets and recurrent neural networks.

Dataset statistics.

Results. Table {{table:9d72400b-e8b8-4129-8af8-86cd7a73ccf9}}  and {{table:1efc3184-2f42-4fc3-9d25-03a09dbdef37}}  presents the evaluation results. We observe that after fine-tuning, the two models behave normally on all downstream tasks. However, the backdoors only remain on 1 or 2 tasks. Both models have high ASRs on SST-2 and NeuBA can successfully attack HSOL,
but they fail to attack AG's News and Lingspam simultaneously. We attribute this to the dataset size and text length. As shown in Table {{table:bbc77722-65d3-44f5-aeb5-ab49107a81a4}} , SST-2 and HSOL are the smallest in size and average text length, while AG's News is large in size and Lingspam has the longest texts. Therefore, our experiments show that fine-tuning on large datasets or testing on long texts will greatly affect ASR, which reveal severe shortcomings that are neglected in previous researches. For stealthiness, NeuBA is better than POR on both PPL and grammar error. Both models preserve the semantics well and get high validity scores.
Compare Table {{table:1efc3184-2f42-4fc3-9d25-03a09dbdef37}}  with Table {{table:9d72400b-e8b8-4129-8af8-86cd7a73ccf9}} , we can find that increasing the number of triggers in each sentence benefits POR on HSOL while hurting NeuBA on both SST-2 and HSOL.
And even with more triggers, these two methods still fail to attack AG's News and Lingspam.

We list the statistics of datasets used in experiments in Table {{table:bbc77722-65d3-44f5-aeb5-ab49107a81a4}} .
All these datasets are available in OpenBackdoor.


False Information Dissemination, Jamming

Intrusion Detection of UAVs Based on the Deep Belief Network Optimized by PSO

Abstract: With the rapid development of information technology, the problem of the network security of unmanned aerial vehicles (UAVs) has become increasingly prominent. In order to solve the intrusion detection problem of massive, high-dimensional, and nonlinear data, this paper proposes an intrusion detection method based on the deep belief network (DBN) optimized by particle swarm optimization (PSO). First, a classification model based on the DBN is constructed, and the PSO algorithm is then used to optimize the number of hidden layer nodes of the DBN, to obtain the optimal DBN structure. The simulations are conducted on a benchmark intrusion dataset, and the results show that the accuracy of the DBN-PSO algorithm reaches 92.44%, which is higher than those of the support vector machine (SVM), artificial neural network (ANN), deep neural network (DNN), and Adaboost. It can be seen from comparative experiments that the optimization effect of PSO is better than those of the genetic algorithm, simulated annealing algorithm, and Bayesian optimization algorithm. The method of PSO-DBN provides an effective solution to the problem of intrusion detection of UAV networks.

Unsupervised Intrusion Detection System for Unmanned Aerial Vehicle with Less Labeling Effort

Abstract: Along with the importance of safety, an IDS has become a significant task in the real world. Prior studies proposed various intrusion detection models for the UAV. Past rule-based approaches provided a concrete baseline IDS model, and the machine learning-based method achieved a precise intrusion detection performance on the UAV with supervised learning models. However, previous methods have room for improvement to be implemented in the real world. Prior methods required a large labeling effort on the dataset, and the model could not identify attacks that were not trained before. To jump over these hurdles, we propose an IDS with unsupervised learning. As unsupervised learning does not require labeling, our model let the practitioner not to label every type of attack from the flight data. Moreover, the model can identify an abnormal status of the UAV regardless of the type of attack. We trained an autoencoder with the benign flight data only and checked the model provides a different reconstruction loss at the benign flight and the flight under attack. We discovered that the model produces much higher reconstruction loss with the flight under attack than the benign flight; thus, this reconstruction loss can be utilized to recognize an intrusion to the UAV. With consideration of the computation overhead and the detection performance in the wild, we expect our model can be a concrete and practical baseline IDS on the UAV.

Prior researches of the IDS on UAVs

Researchers have proposed various IDS approaches on the UAV. Prior works can be categorized into two streams as shown in Table {{table:a1422d36-a1b8-4930-a48c-14414288874e}} : a rule-based approach and the machine learning-based approach.


Using social network analysis to prevent money laundering

Abstract: This research explores the opportunities for the application of network analytic techniques to prevent money laundering. We worked on real world data by analyzing the central database of a factoring company, mainly operating in Italy, over a period of 19 months. This database contained the financial operations linked to the factoring business, together with other useful information about the company clients. We propose a new approach to sort and map relational data and present predictive models, based on network metrics, to assess risk profiles of clients involved in the factoring business. We find that risk profiles can be predicted by using social network metrics. In our dataset, the most dangerous social actors deal with bigger or more frequent financial operations; they are more peripheral in the transactions network; they mediate transactions across different economic sectors and operate in riskier countries or Italian regions. Finally, to spot potential clusters of criminals, we propose a visual analysis of the tacit links existing among different companies who share the same owner or representative. Our findings show the importance of using a network-based approach when looking for suspicious financial operations and potential criminals.

Scalable Graph Learning for Anti-Money Laundering: A First Look

Abstract: Organized crime inflicts human suffering on a genocidal scale: the Mexican drug cartels have murdered 150,000 people since 2006, upwards of 700,000 people per year are "exported" in a human trafficking industry enslaving an estimated 40 million people. These nefarious industries rely on sophisticated money laundering schemes to operate. Despite tremendous resources dedicated to anti-money laundering (AML) only a tiny fraction of illicit activity is prevented. The research community can help. In this brief paper, we map the structural and behavioral dynamics driving the technical challenge. We review AML methods, current and emergent. We provide a first look at scalable graph convolutional neural networks for forensic analysis of financial data, which is massive, dense, and dynamic. We report preliminary experimental results using a large synthetic graph (1M nodes, 9M edges) generated by a data simulator we created called AMLSim. We consider opportunities for high performance efficiency, in terms of computation and memory, and we share results from a simple graph compression experiment. Our results support our working hypothesis that graph deep learning for AML bears great promise in the fight against criminal financial activity.

Uncovering Insurance Fraud Conspiracy with Network Learning

Abstract: Fraudulent claim detection is one of the greatest challenges the insurance industry faces. Alibaba's return-freight insurance, providing return-shipping postage compensations over product return on the e-commerce platform, receives thousands of potentially fraudulent claims everyday. Such deliberate abuse of the insurance policy could lead to heavy financial losses. In order to detect and prevent fraudulent insurance claims, we developed a novel data-driven procedure to identify groups of organized fraudsters, one of the major contributions to financial losses, by learning network information. In this paper, we introduce a device-sharing network among claimants, followed by developing an automated solution for fraud detection based on graph learning algorithms, to separate fraudsters from regular customers and uncover groups of organized fraudsters. This solution applied at Alibaba achieves more than 80% precision while covering 44% more suspicious accounts compared with a previously deployed rule-based classifier after human expert investigations. Our approach can easily and effectively generalizes to other types of insurance.

Typical graph approach used in transaction security.

Among the strategies of transaction security protection, graph-based methods model behaviors and attributes of transactions simultaneously, which attract more attention recently. Typical papers and corresponding graph approaches are listed in Table {{table:1109914f-6fa8-4f98-b7b0-662f5dcddb7a}} .


Convolutional Neural Networks for Sentence Classification

Abstract: We report on a series of experiments with convolutional neural networks (CNN) trained on top of pre-trained word vectors for sentence-level classification tasks. We show that a simple CNN with little hyperparameter tuning and static vectors achieves excellent results on multiple benchmarks. Learning task-specific vectors through fine-tuning offers further gains in performance. We additionally propose a simple modification to the architecture to allow for the use of both task-specific and static vectors. The CNN models discussed herein improve upon the state of the art on 4 out of 7 tasks, which include sentiment analysis and question classification.

Summary of data and models used in our experiments.

Table {{table:0a5327d8-8cbc-4390-9bf3-a2da68ad2ef1}}  summarizes the data and models used in our experiments. We compare our proposed model with the following black-box non-targeted attacks:


Markline Automotive Information Platform

Firms' known connections, products, intermediate inputs

Firms' sales, intermediate expenses by sector, network density (for calibration)

Firms' sales (time series), industrial sector, network sectoral structure

Predicting Hidden Links in Supply Networks

Abstract: Manufacturing companies often lack visibility of the procurement interdependencies between the suppliers within their supply network. However, knowledge of these interdependencies is useful to plan for potential operational disruptions. In this paper, we develop the Supply Network Link Predictor (SNLP) method to infer supplier interdependencies using the manufacturer’s incomplete knowledge of the network. SNLP uses topological data to extract relational features from the known network to train a classifier for predicting potential links. Using a test case from the automotive industry, four features are extracted: (i) number of existing supplier links, (ii) overlaps between supplier product portfolios, (iii) product outsourcing associations, and (iv) likelihood of buyers purchasing from two suppliers together. Naive Bayes and Logistic Regression are then employed to predict whether these features can help predict interdependencies between two suppliers. Our results show that these features can indeed be used to predict interdependencies in the network and that predictive accuracy is maximised by (i) and (iii). The findings give rise to the exciting possibility of using data analytics for improving supply chain visibility. We then proceed to discuss to what extent such approaches can be adopted and their limitations, highlighting next steps for future work in this area.

Reconstructing firm-level interactions: the Dutch input-output network

Abstract: Recent crises have shown that the knowledge of the structure of input-output networks at the firm level is crucial when studying economic resilience from the microscopic point of view of firms that rewire their connections under supply and demand shocks. Unfortunately, empirical inter-firm network data are rarely accessible and protected by confidentiality. The available methods of network reconstruction from partial information, which have been devised for financial exposures, are inadequate for inter-firm relationships because they treat all pairs of nodes as potentially interacting, thereby overestimating the rewiring capabilities of the system. Here we use two big data sets of transactions in the Netherlands to represent a large portion of the Dutch inter-firm network and document the properties of one of the few analysed networks of this kind. We, then, introduce a generalized maximum-entropy reconstruction method that preserves the production function of each firm in the data, i.e. the input and output flows of each node for each product type. We confirm that the new method becomes increasingly more reliable as a finer product resolution is considered and can therefore be used as a generative model of inter-firm networks with fine production constraints. The likelihood of the model, being related to the entropy, proxies the rewiring capability of the system for a fixed input-output configuration.

Revealing production networks from firm growth dynamics

Abstract: We study the correlation structure of firm growth rates. We show that most firms are correlated because of their exposure to a common factor but that firms linked through the supply chain exhibit a stronger correlation on average than firms that are not. Removing this common factor significantly reduces the average correlation between two firms with no relationship in the supply chain while maintaining a significant correlation between two firms that are linked. We then investigate if this observation can be used to reconstruct the topology of a supply chain network using Gaussian Markov Models.

Overview of the papers that reconstruct the supply network topology.

We start by reviewing studies that reconstruct the network using link prediction, and then those that do so using network inference methods. Table {{table:e42121a5-3ea1-4b57-9a12-86e497f753b2}}  provides an overall summary of the methods and their differences.


Learning to Create Better Ads: Generation and Ranking Approaches for Ad Creative Refinement

Abstract: In the online advertising industry, the process of designing an ad creative (i.e., ad text and image) requires manual labor. Typically, each advertiser launches multiple creatives via online A/B tests to infer effective creatives for the target audience, that are then refined further in an iterative fashion. Due to the manual nature of this process, it is time-consuming to learn, refine, and deploy the modified creatives. Since major ad platforms typically run A/B tests for multiple advertisers in parallel, we explore the possibility of collaboratively learning ad creative refinement via A/B tests of multiple advertisers. In particular, given an input ad creative, we study approaches to refine the given ad text and image by: (i) generating new ad text, (ii) recommending keyphrases for new ad text, and (iii) recommending image tags (objects in image) to select new ad image. Based on A/B tests conducted by multiple advertisers, we form pairwise examples of inferior and superior ad creatives, and use such pairs to train models for the above tasks. For generating new ad text, we demonstrate the efficacy of an encoder-decoder architecture with copy mechanism, which allows some words from the (inferior) input text to be copied to the output while incorporating new words associated with higher click-through-rate. For the keyphrase and image tag recommendation task, we demonstrate the efficacy of a deep relevance matching model, as well as the relative robustness of ranking approaches compared to ad text generation in cold-start scenarios with unseen advertisers. We also share broadly applicable insights from our experiments using data from the Yahoo Gemini ad platform.

Algorithmic Copywriting: Automated Generation of Health-Related Advertisements to Improve their Performance

Abstract: Search advertising, a popular method for online marketing, has been employed to improve health by eliciting positive behavioral change. However, writing effective advertisements requires expertise and experimentation, which may not be available to health authorities wishing to elicit such changes, especially when dealing with public health crises such as epidemic outbreaks. Here we develop a framework, comprised of two neural networks models, that automatically generate ads. First, it employs a generator model, which create ads from web pages. It then employs a translation model, which transcribes ads to improve performance. We trained the networks using 114K health-related ads shown on Microsoft Advertising. We measure ads performance using the click-through rates (CTR). Our experiments show that the generated advertisements received approximately the same CTR as human-authored ads. The marginal contribution of the generator model was, on average, 28\% lower than that of human-authored ads, while the translator model received, on average, 32\% more clicks than human-authored ads. Our analysis shows that the translator model produces ads reflecting higher values of psychological attributes associated with a user action, including higher valance and arousal, and more calls-to-actions. In contrast, levels of these attributes in ads produced by the generator model are similar to those of human-authored ads. Our results demonstrate the ability to automatically generate useful advertisements for the health domain. We believe that our work offers health authorities an improved ability to nudge people towards healthier behaviors while saving the time and cost needed to build effective advertising campaigns.

Query-Variant Advertisement Text Generation with Association Knowledge

Abstract: Online advertising is an important revenue source for many IT companies. In the search advertising scenario, advertisement text that meets the need of the search query would be more attractive to the user. However, the manual creation of query-variant advertisement texts for massive items is expensive. Traditional text generation methods tend to focus on the general searching needs with high frequency while ignoring the diverse personalized searching needs with low frequency. In this paper, we propose the query-variant advertisement text generation task that aims to generate candidate advertisement texts for different web search queries with various needs based on queries and item keywords. To solve the problem of ignoring low-frequency needs, we propose a dynamic association mechanism to expand the receptive field based on external knowledge, which can obtain associated words to be added to the input. These associated words can serve as bridges to transfer the ability of the model from the familiar high-frequency words to the unfamiliar low-frequency words. With association, the model can make use of various personalized needs in queries and generate query-variant advertisement texts. Both automatic and human evaluations show that our model can generate more attractive advertisement text than baselines.

DeepGen: Diverse Search Ad Generation and Real-Time Customization

Abstract: We present DeepGen, a system deployed at web scale for automatically creating sponsored search advertisements (ads) for BingAds customers. We leverage state-of-the-art natural language generation (NLG) models to generate fluent ads from advertiser's web pages in an abstractive fashion and solve practical issues such as factuality and inference speed. In addition, our system creates a customized ad in real-time in response to the user's search query, therefore highlighting different aspects of the same product based on what the user is looking for. To achieve this, our system generates a diverse choice of smaller pieces of the ad ahead of time and, at query time, selects the most relevant ones to be stitched into a complete ad. We improve generation diversity by training a controllable NLG model to generate multiple ads for the same web page highlighting different selling points. Our system design further improves diversity horizontally by first running an ensemble of generation models trained with different objectives and then using a diversity sampling algorithm to pick a diverse subset of generation results for online selection. Experimental results show the effectiveness of our proposed system design. Our system is currently deployed in production, serving ${\sim}4\%$ of global ads served in Bing.

CREATER: CTR-driven Advertising Text Generation with Controlled Pre-Training and Contrastive Fine-Tuning

Abstract: This paper focuses on automatically generating the text of an ad, and the goal is that the generated text can capture user interest for achieving higher click-through rate (CTR). We propose CREATER, a CTR-driven advertising text generation approach, to generate ad texts based on high-quality user reviews. To incorporate CTR objective, our model learns from online A/B test data with contrastive learning, which encourages the model to generate ad texts that obtain higher CTR. To alleviate the low-resource issue, we design a customized self-supervised objective reducing the gap between pre-training and fine-tuning. Experiments on industrial datasets show that CREATER significantly outperforms current approaches. It has been deployed online in a leading advertising platform and brings uplift on core online metrics.

A summary of existing research on ad text generation. xACL () presents whether the paper belongs to the ACL community, or some other research community (no ).

ATG has experienced remarkable growth in recent years, garnering significant attention as a valuable application of natural language processing (NLP).
However, the automated evaluation of models presents substantial challenges.
These challenges are primarily due to the absence of a shared benchmark dataset that can benefit the entire research community, resulting in individual validation using non-public data and impeding comprehensive comparisons among different methods {{cite:0240104}}.
Table {{table:0c9ea5a6-bb3f-4d2d-bc9f-3f5445016237}}  summarizes existing studies in ATG.
This table demonstrates that ATG field has primarily been driven by advertising-related companies.
It is worth noting that there is no consensus on the input-output format as each company has independently verified their ATG systems using its datasets, which is a significant impediment to the generalization of technology.
In addition, while ATG research was initially dominated by data mining-related research areas such as KDD and CIKM, it has recently begun to attract attention in the ACL community.

In search ads, there are numerous variations in input/output settings based on individual company specifications, with the potential for further changes in the future (see Table {{table:0c9ea5a6-bb3f-4d2d-bc9f-3f5445016237}} ).
To foster the generalization of ATG technology in an academic research context, we aimed to develop a task not tied to a specific application (e.g., Google, Microsoft Bing, and Yahoo search engines) but focused on universal core problems shared across these applications.
Although this study's primary focus is search ads, our redesigned task setting is adaptable to tackling specific challenges in other advertising domains, including display ads and various merchandise advertisements.
For example, in the case of display ads, the user signal {{formula:842eb0c5-5685-46f0-870f-121638a6d19e}}  could be the user's purchase history.