Clustering and Locality-Sensitive Hashing

Star-Transformer

Abstract: Although Transformer has achieved great successes on many NLP tasks, its heavy structure with fully-connected attention connections leads to dependencies on large training data. In this paper, we present Star-Transformer, a lightweight alternative by careful sparsification. To reduce model complexity, we replace the fully-connected structure with a star-shaped topology, in which every two non-adjacent nodes are connected through a shared relay node. Thus, complexity is reduced from quadratic to linear, while preserving the capacity to capture both local composition and long-range dependency. The experiments on four tasks (22 datasets) show that Star-Transformer achieved significant improvements against the standard Transformer for the modestly sized datasets.

Generating Long Sequences with Sparse Transformers

Abstract: Transformers are powerful sequence models, but require time and memory that grows quadratically with the sequence length. In this paper we introduce sparse factorizations of the attention matrix which reduce this to $O(n \sqrt{n})$. We also introduce a) a variation on architecture and initialization to train deeper networks, b) the recomputation of attention matrices to save memory, and c) fast attention kernels for training. We call networks with these changes Sparse Transformers, and show they can model sequences tens of thousands of timesteps long using hundreds of layers. We use the same architecture to model images, audio, and text from raw bytes, setting a new state of the art for density modeling of Enwik8, CIFAR-10, and ImageNet-64. We generate unconditional samples that demonstrate global coherence and great diversity, and show it is possible in principle to use self-attention to model sequences of length one million or more.

Transformer on a Diet

Abstract: Transformer has been widely used thanks to its ability to capture sequence information in an efficient way. However, recent developments, such as BERT and GPT-2, deliver only heavy architectures with a focus on effectiveness. In this paper, we explore three carefully-designed light Transformer architectures to figure out whether the Transformer with less computations could produce competitive results. Experimental results on language model benchmark datasets hint that such trade-off is promising, and the light Transformer reduces 70% parameters at best, while obtains competitive perplexity compared to standard Transformer. The source code is publicly available.

Blockwise Self-Attention for Long Document Understanding

Abstract: We present BlockBERT, a lightweight and efficient BERT model for better modeling long-distance dependencies. Our model extends BERT by introducing sparse block structures into the attention matrix to reduce both memory consumption and training/inference time, which also enables attention heads to capture either short- or long-range contextual information. We conduct experiments on language model pre-training and several benchmark question answering datasets with various paragraph lengths. BlockBERT uses 18.7-36.1% less memory and 12.0-25.1% less time to learn the model. During testing, BlockBERT saves 27.8% inference time, while having comparable and sometimes better prediction accuracy, compared to an advanced BERT-based model, RoBERTa.

Longformer: The Long-Document Transformer

Abstract: Transformer-based models are unable to process long sequences due to their self-attention operation, which scales quadratically with the sequence length. To address this limitation, we introduce the Longformer with an attention mechanism that scales linearly with sequence length, making it easy to process documents of thousands of tokens or longer. Longformer's attention mechanism is a drop-in replacement for the standard self-attention and combines a local windowed attention with a task motivated global attention. Following prior work on long-sequence transformers, we evaluate Longformer on character-level language modeling and achieve state-of-the-art results on text8 and enwik8. In contrast to most prior work, we also pretrain Longformer and finetune it on a variety of downstream tasks. Our pretrained Longformer consistently outperforms RoBERTa on long document tasks and sets new state-of-the-art results on WikiHop and TriviaQA. We finally introduce the Longformer-Encoder-Decoder (LED), a Longformer variant for supporting long document generative sequence-to-sequence tasks, and demonstrate its effectiveness on the arXiv summarization dataset.

Sparse Sinkhorn Attention

Abstract: We propose Sparse Sinkhorn Attention, a new efficient and sparse method for learning to attend. Our method is based on differentiable sorting of internal representations. Concretely, we introduce a meta sorting network that learns to generate latent permutations over sequences. Given sorted sequences, we are then able to compute quasi-global attention with only local windows, improving the memory efficiency of the attention module. To this end, we propose new algorithmic innovations such as Causal Sinkhorn Balancing and SortCut, a dynamic sequence truncation method for tailoring Sinkhorn Attention for encoding and/or decoding purposes. Via extensive experiments on algorithmic seq2seq sorting, language modeling, pixel-wise image generation, document classification and natural language inference, we demonstrate that our memory efficient Sinkhorn Attention method is competitive with vanilla attention and consistently outperforms recently proposed efficient Transformer models such as Sparse Transformers.

SparseBERT: Rethinking the Importance Analysis in Self-attention

Abstract: Transformer-based models are popularly used in natural language processing (NLP). Its core component, self-attention, has aroused widespread interest. To understand the self-attention mechanism, a direct method is to visualize the attention map of a pre-trained model. Based on the patterns observed, a series of efficient Transformers with different sparse attention masks have been proposed. From a theoretical perspective, universal approximability of Transformer-based models is also recently proved. However, the above understanding and analysis of self-attention is based on a pre-trained model. To rethink the importance analysis in self-attention, we study the significance of different positions in attention matrix during pre-training. A surprising result is that diagonal elements in the attention map are the least important compared with other attention positions. We provide a proof showing that these diagonal elements can indeed be removed without deteriorating model performance. Furthermore, we propose a Differentiable Attention Mask (DAM) algorithm, which further guides the design of the SparseBERT. Extensive experiments verify our interesting findings and illustrate the effect of the proposed algorithm.

Adaptively Sparse Transformers

Abstract: Attention mechanisms have become ubiquitous in NLP. Recent architectures, notably the Transformer, learn powerful context-aware word representations through layered, multi-headed attention. The multiple heads learn diverse types of word relationships. However, with standard softmax attention, all attention heads are dense, assigning a non-zero weight to all context words. In this work, we introduce the adaptively sparse Transformer, wherein attention heads have flexible, context-dependent sparsity patterns. This sparsity is accomplished by replacing softmax with $\alpha$-entmax: a differentiable generalization of softmax that allows low-scoring words to receive precisely zero weight. Moreover, we derive a method to automatically learn the $\alpha$ parameter -- which controls the shape and sparsity of $\alpha$-entmax -- allowing attention heads to choose between focused or spread-out behavior. Our adaptively sparse Transformer improves interpretability and head diversity when compared to softmax Transformers on machine translation datasets. Findings of the quantitative and qualitative analysis of our approach include that heads in different layers learn different sparsity preferences and tend to be more diverse in their attention distributions than softmax Transformers. Furthermore, at no cost in accuracy, sparsity in attention heads helps to uncover different head specializations.

Reformer: The Efficient Transformer

Abstract: Large Transformer models routinely achieve state-of-the-art results on a number of tasks but training these models can be prohibitively costly, especially on long sequences. We introduce two techniques to improve the efficiency of Transformers. For one, we replace dot-product attention by one that uses locality-sensitive hashing, changing its complexity from O($L^2$) to O($L\log L$), where $L$ is the length of the sequence. Furthermore, we use reversible residual layers instead of the standard residuals, which allows storing activations only once in the training process instead of $N$ times, where $N$ is the number of layers. The resulting model, the Reformer, performs on par with Transformer models while being much more memory-efficient and much faster on long sequences.

Efficient Content-Based Sparse Attention with Routing Transformers

Abstract: Self-attention has recently been adopted for a wide range of sequence modeling problems. Despite its effectiveness, self-attention suffers from quadratic compute and memory requirements with respect to sequence length. Successful approaches to reduce this complexity focused on attending to local sliding windows or a small set of locations independent of content. Our work proposes to learn dynamic sparse attention patterns that avoid allocating computation and memory to attend to content unrelated to the query of interest. This work builds upon two lines of research: it combines the modeling flexibility of prior work on content-based sparse attention with the efficiency gains from approaches based on local, temporal sparse attention. Our model, the Routing Transformer, endows self-attention with a sparse routing module based on online k-means while reducing the overall complexity of attention to $O\left(n^{1.5}d\right)$ from $O\left(n^2d\right)$ for sequence length $n$ and hidden dimension $d$. We show that our model outperforms comparable sparse attention models on language modeling on Wikitext-103 (15.8 vs 18.3 perplexity) as well as on image generation on ImageNet-64 (3.43 vs 3.44 bits/dim) while using fewer self-attention layers. Additionally, we set a new state-of-the-art on the newly released PG-19 data-set, obtaining a test perplexity of 33.2 with a 22 layer Routing Transformer model trained on sequences of length 8192.

Synthesizer: Rethinking Self-Attention in Transformer Models

Abstract: The dot product self-attention is known to be central and indispensable to state-of-the-art Transformer models. But is it really required? This paper investigates the true importance and contribution of the dot product-based self-attention mechanism on the performance of Transformer models. Via extensive experiments, we find that (1) random alignment matrices surprisingly perform quite competitively and (2) learning attention weights from token-token (query-key) interactions is useful but not that important after all. To this end, we propose \textsc{Synthesizer}, a model that learns synthetic attention weights without token-token interactions. In our experiments, we first show that simple Synthesizers achieve highly competitive performance when compared against vanilla Transformer models across a range of tasks, including machine translation, language modeling, text generation and GLUE/SuperGLUE benchmarks. When composed with dot product attention, we find that Synthesizers consistently outperform Transformers. Moreover, we conduct additional comparisons of Synthesizers against Dynamic Convolutions, showing that simple Random Synthesizer is not only $60\%$ faster but also improves perplexity by a relative $3.5\%$. Finally, we show that simple factorized Synthesizers can outperform Linformers on encoding only tasks.

Nystr\"omformer: A Nystr\"om-Based Algorithm for Approximating Self-Attention

Abstract: Transformers have emerged as a powerful tool for a broad range of natural language processing tasks. A key component that drives the impressive performance of Transformers is the self-attention mechanism that encodes the influence or dependence of other tokens on each specific token. While beneficial, the quadratic complexity of self-attention on the input sequence length has limited its application to longer sequences -- a topic being actively studied in the community. To address this limitation, we propose Nystr\"{o}mformer -- a model that exhibits favorable scalability as a function of sequence length. Our idea is based on adapting the Nystr\"{o}m method to approximate standard self-attention with $O(n)$ complexity. The scalability of Nystr\"{o}mformer enables application to longer sequences with thousands of tokens. We perform evaluations on multiple downstream tasks on the GLUE benchmark and IMDB reviews with standard sequence length, and find that our Nystr\"{o}mformer performs comparably, or in a few cases, even slightly better, than standard self-attention. On longer sequence tasks in the Long Range Arena (LRA) benchmark, Nystr\"{o}mformer performs favorably relative to other efficient self-attention methods. Our code is available at https://github.com/mlpen/Nystromformer.

Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention

Abstract: Transformers achieve remarkable performance in several tasks but due to their quadratic complexity, with respect to the input's length, they are prohibitively slow for very long sequences. To address this limitation, we express the self-attention as a linear dot-product of kernel feature maps and make use of the associativity property of matrix products to reduce the complexity from $\mathcal{O}\left(N^2\right)$ to $\mathcal{O}\left(N\right)$, where $N$ is the sequence length. We show that this formulation permits an iterative implementation that dramatically accelerates autoregressive transformers and reveals their relationship to recurrent neural networks. Our linear transformers achieve similar performance to vanilla transformers and they are up to 4000x faster on autoregressive prediction of very long sequences.

Rethinking Attention with Performers

Abstract: We introduce Performers, Transformer architectures which can estimate regular (softmax) full-rank-attention Transformers with provable accuracy, but using only linear (as opposed to quadratic) space and time complexity, without relying on any priors such as sparsity or low-rankness. To approximate softmax attention-kernels, Performers use a novel Fast Attention Via positive Orthogonal Random features approach (FAVOR+), which may be of independent interest for scalable kernel methods. FAVOR+ can be also used to efficiently model kernelizable attention mechanisms beyond softmax. This representational power is crucial to accurately compare softmax with other kernels for the first time on large-scale tasks, beyond the reach of regular Transformers, and investigate optimal attention-kernels. Performers are linear architectures fully compatible with regular Transformers and with strong theoretical guarantees: unbiased or nearly-unbiased estimation of the attention matrix, uniform convergence and low estimation variance. We tested Performers on a rich set of tasks stretching from pixel-prediction through text models to protein sequence modeling. We demonstrate competitive results with other examined efficient sparse and dense attention methods, showcasing effectiveness of the novel attention-learning paradigm leveraged by Performers.

Fast Transformers with Clustered Attention

Abstract: Transformers have been proven a successful model for a variety of tasks in sequence modeling. However, computing the attention matrix, which is their key component, has quadratic complexity with respect to the sequence length, thus making them prohibitively expensive for large sequences. To address this, we propose clustered attention, which instead of computing the attention for every query, groups queries into clusters and computes attention just for the centroids. To further improve this approximation, we use the computed clusters to identify the keys with the highest attention per query and compute the exact key/query dot products. This results in a model with linear complexity with respect to the sequence length for a fixed number of clusters. We evaluate our approach on two automatic speech recognition datasets and show that our model consistently outperforms vanilla transformers for a given computational budget. Finally, we demonstrate that our model can approximate arbitrarily complex attention distributions with a minimal number of clusters by approximating a pretrained BERT model on GLUE and SQuAD benchmarks with only 25 clusters and no loss in performance.

Transformer-XL: Attentive Language Models beyond a Fixed-Length Context

Abstract: Transformers have a potential of learning longer-term dependency, but are limited by a fixed-length context in the setting of language modeling. We propose a novel neural architecture Transformer-XL that enables learning dependency beyond a fixed length without disrupting temporal coherence. It consists of a segment-level recurrence mechanism and a novel positional encoding scheme. Our method not only enables capturing longer-term dependency, but also resolves the context fragmentation problem. As a result, Transformer-XL learns dependency that is 80% longer than RNNs and 450% longer than vanilla Transformers, achieves better performance on both short and long sequences, and is up to 1,800+ times faster than vanilla Transformers during evaluation. Notably, we improve the state-of-the-art results of bpc/perplexity to 0.99 on enwiki8, 1.08 on text8, 18.3 on WikiText-103, 21.8 on One Billion Word, and 54.5 on Penn Treebank (without finetuning). When trained only on WikiText-103, Transformer-XL manages to generate reasonably coherent, novel text articles with thousands of tokens. Our code, pretrained models, and hyperparameters are available in both Tensorflow and PyTorch.

Compressive Transformers for Long-Range Sequence Modelling

Abstract: We present the Compressive Transformer, an attentive sequence model which compresses past memories for long-range sequence learning. We find the Compressive Transformer obtains state-of-the-art language modelling results in the WikiText-103 and Enwik8 benchmarks, achieving 17.1 ppl and 0.97 bpc respectively. We also find it can model high-frequency speech effectively and can be used as a memory mechanism for RL, demonstrated on an object matching task. To promote the domain of long-range sequence learning, we propose a new open-vocabulary language modelling benchmark derived from books, PG-19.

Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processing

Abstract: With the success of language pretraining, it is highly desirable to develop more efficient architectures of good scalability that can exploit the abundant unlabeled data at a lower cost. To improve the efficiency, we examine the much-overlooked redundancy in maintaining a full-length token-level presentation, especially for tasks that only require a single-vector presentation of the sequence. With this intuition, we propose Funnel-Transformer which gradually compresses the sequence of hidden states to a shorter one and hence reduces the computation cost. More importantly, by re-investing the saved FLOPs from length reduction in constructing a deeper or wider model, we further improve the model capacity. In addition, to perform token-level predictions as required by common pretraining objectives, Funnel-Transformer is able to recover a deep representation for each token from the reduced hidden sequence via a decoder. Empirically, with comparable or fewer FLOPs, Funnel-Transformer outperforms the standard Transformer on a wide variety of sequence-level prediction tasks, including text classification, language understanding, and reading comprehension. The code and pretrained checkpoints are available at this https URL.

Summary of the specialized methods and their associated models.

The remainder of this survey is organized as follows. Section  introduces the Transformer's architecture and the origin of the quadratic complexity. Section  investigates the popular general methods that have been applied to Transformers to reduce the computations and memory footprint. Section  explores the recent lower-complexity Transformers. Section  explains the limitations of the different approaches and the current evaluation methodology, Section  provides a discussion on the broader impact of lighter and faster Transformers, and Section  points out potential future research directions. Finally, Section  concludes this survey. Practitioners and researchers can find detailed practical guidelines regarding the general and specialized methods in appendix, as well as a summary of the specialized methods (see Table {{table:eaf8fab4-ea6d-4306-b71d-5e243b3420d7}} ) and a discussion about some of the most popular attention-free alternatives.


CEDR: Contextualized Embeddings for Document Ranking

Abstract: Although considerable attention has been given to neural ranking architectures recently, far less attention has been paid to the term representations that are used as input to these models. In this work, we investigate how two pretrained contextualized language models (ELMo and BERT) can be utilized for ad-hoc document ranking. Through experiments on TREC benchmarks, we find that several existing neural ranking architectures can benefit from the additional context provided by contextualized language models. Furthermore, we propose a joint approach that incorporates BERT's classification vector into existing neural models and show that it outperforms state-of-the-art ad-hoc ranking baselines. We call this joint approach CEDR (Contextualized Embeddings for Document Ranking). We also address practical challenges in using these models for ranking, including the maximum input length imposed by BERT and runtime performance impacts of contextualized language models.

ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT

Abstract: Recent progress in Natural Language Understanding (NLU) is driving fast-paced advances in Information Retrieval (IR), largely owed to fine-tuning deep language models (LMs) for document ranking. While remarkably effective, the ranking models based on these LMs increase computational cost by orders of magnitude over prior approaches, particularly as they must feed each query-document pair through a massive neural network to compute a single relevance score. To tackle this, we present ColBERT, a novel ranking model that adapts deep LMs (in particular, BERT) for efficient retrieval. ColBERT introduces a late interaction architecture that independently encodes the query and the document using BERT and then employs a cheap yet powerful interaction step that models their fine-grained similarity. By delaying and yet retaining this fine-granular interaction, ColBERT can leverage the expressiveness of deep LMs while simultaneously gaining the ability to pre-compute document representations offline, considerably speeding up query processing. Beyond reducing the cost of re-ranking the documents retrieved by a traditional model, ColBERT's pruning-friendly interaction mechanism enables leveraging vector-similarity indexes for end-to-end retrieval directly from a large document collection. We extensively evaluate ColBERT using two recent passage search datasets. Results show that ColBERT's effectiveness is competitive with existing BERT-based models (and outperforms every non-BERT baseline), while executing two orders-of-magnitude faster and requiring four orders-of-magnitude fewer FLOPs per query.

BERT-based document ranking algorithms.

We categorize BERT-based NRMs according to BERT encoder type and BERT representation usage.
First, following Humeau et al.{{cite:d6f802b}}, we classify BERT-based NRMs into `cross-encoder' and `bi-encoder' models. Cross-encoder models perform a full self-attention over the entire query-document pair, whereas bi-encoder models perform two independent self-attentions over the query and the document.
The BERT layers of cross-encoder NRMs can model the interaction between a query and a document, and the resulting BERT representations contain contextualized embedding. On the other hand, the BERT self-attention layers of bi-encoder NRMs cannot model the interaction. Therefore, a document is mapped to a fixed BERT representation regardless of the choice of query. This makes it possible for bi-encoder models to pre-compute document representations offline, significantly reducing the computational load per query at the time of inference. In general, cross-encoder models tend to perform better than bi-encoder models, but at the expense of a higher computational cost.
Second, we classify BERT-based NRMs models into `CLS', `query/document', and `CLS,query/document',
according to which BERT output representations are used for the relevance score calculation. The known BERT-based NRMs use one of the three representation choices.
For example, monoBERT uses only CLS representation for calculating the relevance score, whereas ColBERT uses both CLS and query/document representations.
Table {{table:3e484280-4b24-4bc7-b639-dedc3305b731}}  summarizes four of the well-known algorithms with their categorization according to BERT encoder type and BERT representation usage.

In our experiments, monoBERT {{cite:53ea9ae}} is chosen as the cross-encoder teacher, while either TwinBERT {{cite:ba2f1df}} or ColBERT {{cite:03e93a3}} is chosen as the bi-encoder teacher. As shown in the third column of Table {{table:3e484280-4b24-4bc7-b639-dedc3305b731}} , different models use different parts of BERT representations for estimating the relevance score. TRMD forces the student's BERT representations to become similar to the respective teacher's BERT representations.
An example is shown in Figure {{figure:22733279-f41a-4752-a6bd-8875ab7c1b7f}}  where monoBERT and ColBERT are used as the two teachers. In this case, the bi-encoder student learns both of monoBERT's CLS representation and ColBERT's CLS and query/document representations.
As will be shown in the experiment section, the results on Robust04 and Clueweb09 datasets show that TRMD can significantly improve the performance of the existing bi-encoder models: TwinBERT and ColBERT.


BERT-based document ranking algorithms.

We categorize BERT-based NRMs according to BERT encoder type and BERT representation usage.
First, following Humeau et al.{{cite:d00a397}}, we classify BERT-based NRMs into `cross-encoder' and `bi-encoder' models. Cross-encoder models perform a full self-attention over the entire query-document pair, whereas bi-encoder models perform two independent self-attentions over the query and the document.
The BERT layers of cross-encoder NRMs can model the interaction between a query and a document, and the resulting BERT representations contain contextualized embedding. On the other hand, the BERT self-attention layers of bi-encoder NRMs cannot model the interaction. Therefore, a document is mapped to a fixed BERT representation regardless of the choice of a query. This makes it possible for bi-encoder models to pre-compute document representations offline, significantly reducing the computational load per query at the time of inference. In general, cross-encoder models tend to perform better than bi-encoder models but at the expense of a higher computational cost.
Second, we classify BERT-based NRMs models into `CLS', `query/document', and `CLS,query/document',
according to which BERT output representations are used for the relevance score calculation. The known BERT-based NRMs use one of the three representation choices.
For example, monoBERT uses only CLS representation for calculating the relevance score, whereas ColBERT uses both CLS and query/document representations.
Table {{table:0d99ab83-d5f3-4427-a003-ef35eea4b87d}}  summarizes four of the well-known algorithms with their categorization according to BERT encoder type and BERT representation usage.

In our experiments, monoBERT {{cite:a32cdae}} is chosen as the cross-encoder teacher, while either TwinBERT {{cite:8e5ec57}} or ColBERT {{cite:fb1fecf}} is chosen as the bi-encoder teacher. As shown in the third column of Table {{table:0d99ab83-d5f3-4427-a003-ef35eea4b87d}} , different models use different parts of BERT representations for estimating the relevance score. TRMD forces the student's BERT representations to become similar to the respective teacher's BERT representations.
An example is shown in Figure {{figure:9780a408-3270-4f11-b7ab-a90833957cad}}  where monoBERT and ColBERT are used as the two teachers. In this case, the bi-encoder student learns both of monoBERT's CLS representation and ColBERT's CLS and query/document representations.
As will be shown in the experiment section, the results on Robust04 and Clueweb09 datasets show that TRMD can significantly improve the performance of the existing bi-encoder models: TwinBERT and ColBERT.


Abstract: Recently, neural networks purely based on attention were shown to address image understanding tasks such as image classification. However, these visual transformers are pre-trained with hundreds of millions of images using an expensive infrastructure, thereby limiting their adoption. In this work, we produce a competitive convolution-free transformer by training on Imagenet only. We train them on a single computer in less than 3 days. Our reference vision transformer (86M parameters) achieves top-1 accuracy of 83.1% (single-crop evaluation) on ImageNet with no external data. More importantly, we introduce a teacher-student strategy specific to transformers. It relies on a distillation token ensuring that the student learns from the teacher through attention. We show the interest of this token-based distillation, especially when using a convnet as a teacher. This leads us to report results competitive with convnets for both Imagenet (where we obtain up to 85.2% accuracy) and when transferring to other tasks. We share our code and models.

When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentations

Abstract: Vision Transformers (ViTs) and MLPs signal further efforts on replacing hand-wired features or inductive biases with general-purpose neural architectures. Existing works empower the models by massive data, such as large-scale pre-training and/or repeated strong data augmentations, and still report optimization-related problems (e.g., sensitivity to initialization and learning rates). Hence, this paper investigates ViTs and MLP-Mixers from the lens of loss geometry, intending to improve the models' data efficiency at training and generalization at inference. Visualization and Hessian reveal extremely sharp local minima of converged models. By promoting smoothness with a recently proposed sharpness-aware optimizer, we substantially improve the accuracy and robustness of ViTs and MLP-Mixers on various tasks spanning supervised, adversarial, contrastive, and transfer learning (e.g., +5.3\% and +11.0\% top-1 accuracy on ImageNet for ViT-B/16 and Mixer-B/16, respectively, with the simple Inception-style preprocessing). We show that the improved smoothness attributes to sparser active neurons in the first few layers. The resultant ViTs outperform ResNets of similar size and throughput when trained from scratch on ImageNet without large-scale pre-training or strong data augmentations. Model checkpoints are available at \url{https://github.com/google-research/vision_transformer}.

Swin Transformer: Hierarchical Vision Transformer using Shifted Windows

Abstract: This paper presents a new vision Transformer, called Swin Transformer, that capably serves as a general-purpose backbone for computer vision. Challenges in adapting Transformer from language to vision arise from differences between the two domains, such as large variations in the scale of visual entities and the high resolution of pixels in images compared to words in text. To address these differences, we propose a hierarchical Transformer whose representation is computed with \textbf{S}hifted \textbf{win}dows. The shifted windowing scheme brings greater efficiency by limiting self-attention computation to non-overlapping local windows while also allowing for cross-window connection. This hierarchical architecture has the flexibility to model at various scales and has linear computational complexity with respect to image size. These qualities of Swin Transformer make it compatible with a broad range of vision tasks, including image classification (87.3 top-1 accuracy on ImageNet-1K) and dense prediction tasks such as object detection (58.7 box AP and 51.1 mask AP on COCO test-dev) and semantic segmentation (53.5 mIoU on ADE20K val). Its performance surpasses the previous state-of-the-art by a large margin of +2.7 box AP and +2.6 mask AP on COCO, and +3.2 mIoU on ADE20K, demonstrating the potential of Transformer-based models as vision backbones. The hierarchical design and the shifted window approach also prove beneficial for all-MLP architectures. The code and models are publicly available at~\url{https://github.com/microsoft/Swin-Transformer}.

This paper presents a comprehensive study on the robustness of ViTs against adversarial perturbations.
Our results indicate that ViTs are more robust than CNNs on the considered adversarial attacks and certified robustness settings.
We show that the features learned by ViTs contain less low-level information, contributing to improved robustness against adversarial perturbations that often contain high-frequency components. Also,
introducing convolutional blocks in ViTs can facilitate learning low-level features but has a negative effect on adversarial robustness and makes the models more sensitive to high-frequency perturbations.
Moreover, both the sanity performance and the (certified and empirical) adversarial robustness are improved in the modern CNN designs that leverage techniques from ViTs to imitate the global attention behavior.
We also demonstrate adversarial training for ViT.
Our work provides a deep understanding of the intrinsic robustness of ViTs and can be used to inform the design of robust vision models based on the transformer structure. Rima Alaifari, Giovanni S Alberti, and Tandri Gauksson.
Adef: an iterative algorithm to construct adversarial deformations.
arXiv preprint arXiv:1804.07729, 2018.
Faisal Alamri, Sinan Kalkan, and Nicolas Pugeault.
Transformer-encoder detector module: Using context to improve
robustness to adversarial attacks on object detection.
arXiv preprint arXiv:2011.06978, 2020.
Ahmed Aldahdooh, Wassim Hamidouche, and Olivier Deforges.
Reveal of vision transformers robustness against adversarial attacks.
arXiv preprint arXiv:2106.03734, 2021.
Anish Athalye, Nicholas Carlini, and David Wagner.
Obfuscated gradients give a false sense of security: Circumventing
defenses to adversarial examples.
International Coference on International Conference on Machine
Learning, 2018.
Yutong Bai, Jieru Mei, Alan L Yuille, and Cihang Xie.
Are transformers more robust than cnns?
Advances in Neural Information Processing Systems,
34:26831–26843, 2021.
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla
Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell,
et al.
Language models are few-shot learners.
arXiv preprint arXiv:2005.14165, 2020.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander
Kirillov, and Sergey Zagoruyko.
End-to-end object detection with transformers.
In European Conference on Computer Vision, pp. 213–229.
Springer, 2020.
Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and
Ilya Sutskever.
Generative pretraining from pixels.
In International Conference on Machine Learning, pp. 1691–1703. PMLR, 2020.
Xiangning Chen, Cho-Jui Hsieh, and Boqing Gong.
When vision transformers outperform resnets without pretraining or
strong data augmentations.
arXiv preprint arXiv:2106.01548, 2021.
Jeremy Cohen, Elan Rosenfeld, and Zico Kolter.
Certified adversarial robustness via randomized smoothing.
In International Conference on Machine Learning, pp. 1310–1320. PMLR, 2019.
Francesco Croce and Matthias Hein.
Reliable evaluation of adversarial robustness with an ensemble of
diverse parameter-free attacks.
In International Conference on Machine Learning, pp. 2206–2216. PMLR, 2020a.
Francesco Croce and Matthias Hein.
Minimally distorted adversarial examples with a fast adaptive
boundary attack.
In International Conference on Machine Learning, pp. 2196–2205. PMLR, 2020b.
Francesco Croce, Maksym Andriushchenko, and Matthias Hein.
Provable robustness of relu networks via maximization of linear
regions.
In the 22nd International Conference on Artificial Intelligence
and Statistics, pp. 2057–2066. PMLR, 2019.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.
Imagenet: A large-scale hierarchical image database.
In 2009 IEEE conference on computer vision and pattern
recognition, pp. 248–255. Ieee, 2009.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.
Bert: Pre-training of deep bidirectional transformers for language
understanding.
arXiv preprint arXiv:1810.04805, 2018.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn,
Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg
Heigold, Sylvain Gelly, et al.
An image is worth 16x16 words: Transformers for image recognition at
scale.
arXiv preprint arXiv:2010.11929, 2020.
Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur.
Sharpness-aware minimization for efficiently improving
generalization.
arXiv preprint arXiv:2010.01412, 2020.
Siddhant Garg and Goutham Ramakrishnan.
Bae: Bert-based adversarial examples for text classification.
arXiv preprint arXiv:2004.01970, 2020.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy.
Explaining and harnessing adversarial examples.
arXiv preprint arXiv:1412.6572, 2014.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.
Identity mappings in deep residual networks.
In European conference on computer vision, pp. 630–645.
Springer, 2016.
Dan Hendrycks, Kimin Lee, and Mantas Mazeika.
Using pre-training can improve model robustness and uncertainty.
In International Conference on Machine Learning, pp. 2712–2721. PMLR, 2019.
Charles Herrmann, Kyle Sargent, Lu Jiang, Ramin Zabih, Huiwen Chang, Ce Liu,
Dilip Krishnan, and Deqing Sun.
Pyramid adversarial training improves vit performance.
In Proceedings of the IEEE/CVF Conference on Computer Vision
and Pattern Recognition, pp. 13419–13429, 2022.
Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang,
Tobias Weyand, Marco Andreetto, and Hartwig Adam.
Mobilenets: Efficient convolutional neural networks for mobile vision
applications.
arXiv preprint arXiv:1704.04861, 2017.
Yu-Lun Hsieh, Minhao Cheng, Da-Cheng Juan, Wei Wei, Wen-Lian Hsu, and Cho-Jui
Hsieh.
On the robustness of self-attentive models.
In ACL, pp. 1520–1529, 2019.
Jie Hu, Li Shen, and Gang Sun.
Squeeze-and-excitation networks.
In Proceedings of the IEEE conference on computer vision and
pattern recognition, pp. 7132–7141, 2018.
Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon
Tran, and Aleksander Madry.
Adversarial examples are not bugs, they are features.
Advances in neural information processing systems, 32, 2019.
Kishaan Jeeveswaran., Senthilkumar Kathiresan., Arnav Varma., Omar Magdy.,
Bahram Zonooz., and Elahe Arani.
A comprehensive study of vision transformers on dense prediction
tasks.
In Proceedings of the 17th International Joint Conference on
Computer Vision, Imaging and Computer Graphics Theory and Applications -
Volume 4: VISAPP,, pp. 213–223. INSTICC, SciTePress, 2022.
ISBN 978-989-758-555-5.
doi: 10.5220/0010917800003124.
Di Jin, Zhijing Jin, Joey Tianyi Zhou, and Peter Szolovits.
Is bert really robust? a strong baseline for natural language attack
on text classification and entailment.
In Proceedings of the AAAI conference on artificial
intelligence, volume 34, pp. 8018–8025, 2020.
Alex Krizhevsky, Geoffrey Hinton, et al.
Learning multiple layers of features from tiny images.
2009.
Dmitry Krotov and John Hopfield.
Dense associative memory is robust to adversarial inputs.
Neural computation, 30(12):3151–3167,
2018.
Dmitry Krotov and John J Hopfield.
Dense associative memory for pattern recognition.
Advances in Neural Information Processing Systems, 2016.
Alexey Kurakin, Ian J. Goodfellow, and Samy Bengio.
Adversarial machine learning at scale.
In 5th International Conference on Learning Representations,
ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track
Proceedings. OpenReview.net, 2017.
URL https://openreview.net/forum?id=BJm4T4Kgx.
Linyang Li, Ruotian Ma, Qipeng Guo, Xiangyang Xue, and Xipeng Qiu.
Bert-attack: Adversarial attack against bert using bert.
arXiv preprint arXiv:2004.09984, 2020.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer
Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.
Roberta: A robustly optimized bert pretraining approach.
arXiv preprint arXiv:1907.11692, 2019.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and
Baining Guo.
Swin transformer: Hierarchical vision transformer using shifted
windows.
arXiv preprint arXiv:2103.14030, 2021.
Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell,
and Saining Xie.
A convnet for the 2020s.
arXiv preprint arXiv:2201.03545, 2022.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and
Adrian Vladu.
Towards deep learning models resistant to adversarial attacks.
arXiv preprint arXiv:1706.06083, 2017.
Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri,
Yixuan Li, Ashwin Bharambe, and Laurens Van Der Maaten.
Exploring the limits of weakly supervised pretraining.
In Proceedings of the European Conference on Computer Vision
(ECCV), pp. 181–196, 2018.
Kaleel Mahmood, Rigel Mahmood, and Marten Van Dijk.
On the robustness of vision transformers to adversarial examples.
arXiv preprint arXiv:2104.02610, 2021.
Xiaofeng Mao, Gege Qi, Yuefeng Chen, Xiaodan Li, Ranjie Duan, Shaokai Ye, Yuan
He, and Hui Xue.
Towards robust vision transformer.
In Proceedings of the IEEE/CVF Conference on Computer Vision
and Pattern Recognition, pp. 12042–12051, 2022.
Muzammal Naseer, Kanchana Ranasinghe, Salman Khan, Munawar Hayat, Fahad Shahbaz
Khan, and Ming-Hsuan Yang.
Intriguing properties of vision transformers.
arXiv preprint arXiv:2105.10497, 2021a.
Muzammal Naseer, Kanchana Ranasinghe, Salman Khan, Fahad Shahbaz Khan, and
Fatih Porikli.
On improving adversarial transferability of vision transformers.
arXiv preprint arXiv:2106.04169, 2021b.
Tianyu Pang, Xiao Yang, Yinpeng Dong, Hang Su, and Jun Zhu.
Bag of tricks for adversarial training.
arXiv preprint arXiv:2010.00467, 2020.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory
Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al.
Pytorch: An imperative style, high-performance deep learning library.
arXiv preprint arXiv:1912.01703, 2019.
Sayak Paul and Pin-Yu Chen.
Vision transformers are robust learners.
arXiv preprint arXiv:2105.07581, 2021.
Yao Qin, Chiyuan Zhang, Ting Chen, Balaji Lakshminarayanan, Alex Beutel, and
Xuezhi Wang.
Understanding and improving robustness of vision transformers through
patch-based negative augmentation.
arXiv preprint arXiv:2110.07858, 2021.
Hubert Ramsauer, Bernhard Schäfl, Johannes Lehner, Philipp Seidl, Michael
Widrich, Lukas Gruber, Markus Holzleitner, Milena Pavlović, Geir Kjetil
Sandve, Victor Greiff, et al.
Hopfield networks is all you need.
arXiv preprint arXiv:2008.02217, 2020.
Jonas Rauber, Roland Zimmermann, Matthias Bethge, and Wieland Brendel.
Foolbox native: Fast adversarial attacks to benchmark the robustness
of machine learning models in pytorch, tensorflow, and jax.
Journal of Open Source Software, 5(53):2607, 2020.
doi: 10.21105/joss.02607.
URL https://doi.org/10.21105/joss.02607.
Hadi Salman, Mingjie Sun, Greg Yang, Ashish Kapoor, and J Zico Kolter.
Denoised smoothing: A provable defense for pretrained classifiers.
arXiv preprint arXiv:2003.01908, 2020.
Hadi Salman, Saachi Jain, Eric Wong, and Aleksander Mądry.
Certified patch robustness via smoothed vision transformers.
arXiv preprint arXiv:2110.07719, 2021.
Zhouxing Shi and Minlie Huang.
Robustness to modification with shared words in paraphrase
identification.
In Proceedings of the 2020 Conference on Empirical Methods in
Natural Language Processing: Findings, pp. 164–171, 2020.
Zhouxing Shi, Huan Zhang, Kai-Wei Chang, Minlie Huang, and Cho-Jui Hsieh.
Robustness verification for transformers.
arXiv preprint arXiv:2002.06622, 2020.
Karen Simonyan and Andrew Zisserman.
Very deep convolutional networks for large-scale image recognition.
arXiv preprint arXiv:1409.1556, 2014.
Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Xuyi Chen, Han Zhang, Xin Tian,
Danxiang Zhu, Hao Tian, and Hua Wu.
Ernie: Enhanced representation through knowledge integration.
arXiv preprint arXiv:1904.09223, 2019.
Shiyu Tang, Ruihao Gong, Yan Wang, Aishan Liu, Jiakai Wang, Xinyun Chen,
Fengwei Yu, Xianglong Liu, Dawn Song, Alan Yuille, et al.
Robustart: Benchmarking robustness on architecture design and
training techniques.
arXiv preprint arXiv:2109.05211, 2021.
Bart Thomee, David A Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni,
Douglas Poland, Damian Borth, and Li-Jia Li.
The new data and new challenges in multimedia research.
arXiv preprint arXiv:1503.01817, 1(8), 2015.
Ilya Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai,
Thomas Unterthiner, Jessica Yung, Daniel Keysers, Jakob Uszkoreit, Mario
Lucic, et al.
Mlp-mixer: An all-mlp architecture for vision.
arXiv preprint arXiv:2105.01601, 2021.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre
Sablayrolles, and Hervé Jégou.
Training data-efficient image transformers & distillation through
attention.
In International Conference on Machine Learning, pp. 10347–10357. PMLR, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones,
Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin.
Attention is all you need.
arXiv preprint arXiv:1706.03762, 2017.
Boxin Wang, Shuohang Wang, Y. Cheng, Zhe Gan, R. Jia, Bo Li, and Jing jing Liu.
Infobert: Improving robustness of language models from an information
theoretic perspective.
ArXiv, abs/2010.02329, 2020a.
Haohan Wang, Xindi Wu, Zeyi Huang, and Eric P Xing.
High-frequency component helps explain the generalization of
convolutional neural networks.
In Proceedings of the IEEE/CVF Conference on Computer Vision
and Pattern Recognition, pp. 8684–8694, 2020b.
Ross Wightman.
Pytorch image models.
https://github.com/rwightman/pytorch-image-models, 2019.
Eric Wong, Leslie Rice, and J Zico Kolter.
Fast is better than free: Revisiting adversarial training.
arXiv preprint arXiv:2001.03994, 2020.
Kaidi Xu, Zhouxing Shi, Huan Zhang, Yihan Wang, Kai-Wei Chang, Minlie Huang,
Bhavya Kailkhura, Xue Lin, and Cho-Jui Hsieh.
Automatic perturbation analysis for scalable certified robustness and
beyond.
Advances in Neural Information Processing Systems, 33, 2020.
I Zeki Yalniz, Hervé Jégou, Kan Chen, Manohar Paluri, and Dhruv
Mahajan.
Billion-scale semi-supervised learning for image classification.
arXiv preprint arXiv:1905.00546, 2019.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov,
and Quoc V Le.
Xlnet: Generalized autoregressive pretraining for language
understanding.
arXiv preprint arXiv:1906.08237, 2019.
Mao Ye, Chengyue Gong, and Qiang Liu.
Safer: A structure-free approach for certified robustness to
adversarial word substitutions.
arXiv preprint arXiv:2005.14424, 2020.
Fan Yin, Quanyu Long, Tao Meng, and Kai-Wei Chang.
On the robustness of language encoders against grammatical errors.
arXiv preprint arXiv:2005.05683, 2020.
Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Francis EH Tay, Jiashi
Feng, and Shuicheng Yan.
Tokens-to-token vit: Training vision transformers from scratch on
imagenet.
arXiv preprint arXiv:2101.11986, 2021.
Sergey Zagoruyko and Nikos Komodakis.
Wide residual networks.
arXiv preprint arXiv:1605.07146, 2016.
Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and
Michael Jordan.
Theoretically principled trade-off between robustness and accuracy.
In International Conference on Machine Learning, pp. 7472–7482. PMLR, 2019.
Kai Zhang, Wangmeng Zuo, Yunjin Chen, Deyu Meng, and Lei Zhang.
Beyond a gaussian denoiser: Residual learning of deep cnn for image
denoising.
IEEE transactions on image processing, 26(7):3142–3155, 2017.
Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun.
Shufflenet: An extremely efficient convolutional neural network for
mobile devices.
In Proceedings of the IEEE conference on computer vision and
pattern recognition, pp. 6848–6856, 2018.
Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai.
Deformable detr: Deformable transformers for end-to-end object
detection.
arXiv preprint arXiv:2010.04159, 2020.

Supplemental Material
In this supplemental material, we provide more analysis and results in our experiments.

The Source of Adversarial Robustness
In this section we examine the source of the adversarial robustness revealed in our experiments.
The improved robustness of ViT is not caused by insufficient attack optimization.
We first demonstrate that the better robustness of ViTs in white-box attacks is not caused by the difficult optimization in ViT by plotting the loss landscape with sufficient attack steps.
Cross entropy loss versus varying PGD attack steps for ViT-S/16 and RestNet18. The dashed lines corresponds to larger attach radius of {{formula:fdb800d6-452a-4177-8731-14427d4ddef7}}  and the full lines to smaller attack radius of {{formula:0d3050bd-64a8-4dee-a88e-883a6ce0ef99}} .
{{figure:acb3a8e3-0288-41e3-9025-71def06862c5}}Robust accuracy versus varying PGD attack steps. The attack radii used for evaluation are shown in subtitles.
{{figure:e77592f7-9c2d-4bbf-a647-27397545f759}}Figure {{figure:acb3a8e3-0288-41e3-9025-71def06862c5}}  shows the cross entropy loss versus varing PGD attack steps for ViT-S/16 and ResNet18. Figure {{figure:e77592f7-9c2d-4bbf-a647-27397545f759}}  shows the robust accuracy versus varing PGD attack steps. As shown in the figures, ViT's loss curves converge at a much lower value than RestNet18, suggesting that the improved robustness of ViT is not caused by insufficient attack optimization.
Adversarial accuracy of the target models against PGD attack with different attack radii (“eps”) and attack steps (“steps”). When the attack radius and attack steps are increased, the adversarial accuracy of the target model decreases to zero. Darker blocks stand for more robust models against PGD attack.
{{figure:28247535-b5de-4acc-bcb9-4b4c0f623839}}Figure {{figure:28247535-b5de-4acc-bcb9-4b4c0f623839}}  shows the robust accuracy of more target models against PGD attack with different attack radii (“eps”) and attack steps (“steps”). Vision transformers have darker blocks than CNNs', which stands for their superior adversarial robustness against PGD attack.

A Hopfield Network Perspective
The equivalence between the attention mechanism in transformers to the modern Hopfield network {{cite:932c767}} was recently shown in {{cite:91a38ce}}. Furthermore, on simple Hopfield network (one layer of attention-like network) and dataset (MNIST), improved adversarial robustness was shown in {{cite:3fb7dec}}. Therefore, the connection of attention in transformers to the Hopfield network can be used to explain the improved adversarial robustness for ViTs.

Experiments on SOTA ViT Structures
In this section, we supplement the experimental results of recently proposed SOTA ViTs.
Swin-Trasnformer {{cite:724fd9d}}
computes the representations with shifted windows scheme which brings greater efficiency by limiting self-attention computation to non-overlapping local windows while also allowing for cross-window connection.

DeiT {{cite:8819029}}
further improves the ViTs' performance using data augmentation or distillation from CNN teachers with an additional distillation token.

SAM-ViT {{cite:ee4bd56}}
uses sharpness-aware minimization {{cite:83deadb}} to train ViTs from scratch on ImageNet without large-scale pretraining or strong data augmentations.
Table {{table:23da1812-b7a3-47f8-9013-10010b787aa9}}  summarizes the information of models investigated in our experiments. The window size of the swin transformers in Table {{table:23da1812-b7a3-47f8-9013-10010b787aa9}}  is 7. The pre-trained weights of these models are available in timm package.
SOTA ViT models investigated in our experiments.

[1pt]
Model
Layers
Hidden size
Heads
Params
Deit-T/16 {{cite:8819029}}
12
192
3
6M
Deit-S/16 {{cite:8819029}}
12
384
6
22M
Deit-B/16 {{cite:8819029}}
12
768
12
87M
Dist-Deit-T/16 {{cite:8819029}}
12
192
3
6M
Dist-Deit-S/16 {{cite:8819029}}
12
384
6
22M
Dist-Deit-B/16 {{cite:8819029}}
12
768
12
87M
ViT-SAM-B/16 {{cite:ee4bd56}}
12
768
12
87M
ViT-SAM-B/32 {{cite:ee4bd56}}
12
768
12
88M
Swin-T/4 {{cite:724fd9d}}
(2,2,6,2)
96
(3,6,12,24)
28M
Swin-S/4 {{cite:724fd9d}}
(2,2,18,2)
96
(3,6,12,24)
50M
Swin-B/4 {{cite:724fd9d}}
(2,2,18,2)
128
(4,8,16,32)
88M
Swin-L/4 {{cite:724fd9d}}
(2,2,18,2)
192
(6,12,24,48)
197M
[1pt]
{{table:cf27ea8d-8b3d-4e6b-99bd-626c7c8482ba}}{{table:23da1812-b7a3-47f8-9013-10010b787aa9}}Robust accuracy (%) of ViTs described in Table {{table:23da1812-b7a3-47f8-9013-10010b787aa9}}  against 40-step PGD attack with different attack radii, and also the clean accuracy (“Clean”). A model is considered to be more robust if the robust accuracy is higher.
[1pt]
Model
Clean
0.001
0.003
0.005
0.01
Deit-T/16
72.3
36.8
8.3
2.6
0.3
Deit-S/16
77.7
48.9
17.6
7.1
1.1
Deit-B/16
81.3
46.6
14.3
6.0
0.9
Dist-Deit-T/16
74.4
40.6
5.7
0.7
0.2
Dist-Deit-S/16
79.3
52.4
15.1
4.3
0.3
Dist-Deit-B/16
81.8
55.6
17.7
4.5
0.4
ViT-SAM-B/16
76.7
63.4
37.0
20.1
3.8
ViT-SAM-B/32
63.8
53.2
32.3
19.7
3.1
Swin-T/4
78.8
33.5
6.0
1.2
0.1
Swin-S/4
81.8
40.0
12.4
3.2
0.2
Swin-B/4
82.3
38.8
11.1
4.1
0.3
Swin-L/4
84.2
38.7
11.1
2.9
0.4
[1pt]
{{table:2dea679f-a01f-459a-82ea-5c194db9ab85}}Robust accuracy (%) of ViTs described in Table {{table:23da1812-b7a3-47f8-9013-10010b787aa9}}  against AutoAttack with different attack radii, and also the clean accuracy (“Clean”). A model is considered to be more robust if the robust accuracy is higher.
[1pt]
Model
Clean
0.001
0.003
0.005
0.01
Deit-T/16
72.3
23.4
0.5
0.0
0.0
Deit-S/16
77.7
30.2
1.2
0.0
0.0
Deit-B/16
81.3
20.4
0.3
0.1
0.0
Dist-Deit-T/16
74.4
31.1
0.8
0.1
0.0
Dist-Deit-S/16
79.3
43.1
3.7
0.2
0.0
Dist-Deit-B/16
81.8
42.7
3.4
0.2
0.0
ViT-SAM-B/16
76.7
59.8
26.0
8.4
0.1
ViT-SAM-B/32
63.8
48.9
23.6
9.7
0.8
Swin-T/4
78.8
6.8
0.1
0.0
0.0
Swin-S/4
81.8
7.9
0.1
0.0
0.0
Swin-B/4
82.3
2.4
0.1
0.0
0.0
Swin-L/4
84.2
4.3
0.1
0.0
0.0
[1pt]
{{table:63d7264d-e05e-4812-8a84-c2348623c097}}Table {{table:2dea679f-a01f-459a-82ea-5c194db9ab85}}  shows the clean and robust accuracy of ViTs in Table {{table:23da1812-b7a3-47f8-9013-10010b787aa9}}  against 40-step PGD attack with different radii. And results for AutoAttack are shown in Table {{table:63d7264d-e05e-4812-8a84-c2348623c097}} . Swin-transformers introduce shifted windows scheme that limit self-attention computation to non-overlapping local windows, which harms the robustness as Tokens-to-Token scheme according to the above results.

Experiments on Cifar-10
We choose the ImageNet as the benchmark because ViTs can hardly converge when training directly on small datasets like Cifar. Therefore, we finetune the ViTs instead. As shown in Table {{table:67a2e2fc-d37a-486f-991a-17356ac9fe88}} , ViT-B/4 performs higher robust accuracy than WideResNet, which is consistent with the trend on ImageNet.
Robust accuracy of ViT-B/4 and WideResNet against PGD-10 attack with different attack radii.
[1pt]
Model
0.001
0.003
0.01
0.03
ViT-B/4
0.9202
0.6242
0.0994
0.0103
WideResNet
0.7744
0.5923
0.0854
0.0000
[1pt]
{{table:67a2e2fc-d37a-486f-991a-17356ac9fe88}}
Robustness Against Adversarial Deformation
Besides additively perturbing the correctly classified image, ADef {{cite:2c82c2d}} iteratively applies small deformations to the clean data. We show the robust accuracy against such perturbations in Table {{table:8b74c04a-b8eb-4e5e-810b-3299d2d0be06}} , which is in accordance to the results of PGD and AutoAttack.
Robust accuracy (%) against AFef under the default setting described in {{cite:2c82c2d}}.
Model
ViT-S/16
VGG16
DenseNet
MobileNet
ResNet18
Robust Accuracy
12.4
10.8
11.1
11.7
11.8
{{table:8b74c04a-b8eb-4e5e-810b-3299d2d0be06}}
Transfer Attack Results
Transfer attack results using more attack radii are provided in Figure  {{figure:fe9b5322-0b3d-4a02-b2db-328d7d6b2dba}}
ASR of transfer attack using FGSM with different attack radii. The rows stand for the surrogate models used to generated adversarial examples in the white-box attack approach. The columns stand for the target models. Darker rows correlate to the source models that generate more transferable adversarial examples. While darker columns mean that the target models are more vulnerable to the transfer attack.“Res50-ssl” and “Res50-swsl” are in short of “ResNeXt-32x4d-ssl” and “ResNet50-swsl” respectively.
{{figure:fe9b5322-0b3d-4a02-b2db-328d7d6b2dba}}
Adversarial Training
Settings
We also conduct a preliminary experiment on adversarial training for ViT.
For this experiment we use CIFAR-10 {{cite:be5e7aa}} with {{formula:09f4791f-a7f6-4d48-bd00-90cc4a17fbcc}}  and the ViT-B/16 model.
Since originally this ViT was pre-trained on ImageNet with image size {{formula:18d13232-c2cc-4549-b9f8-2f7a5d50b230}}  and patch size {{formula:cb295da7-accf-45a2-824b-1e071d00793a}}  while image size on CIFAR-10 is {{formula:b742008a-bf77-47c1-bcab-17729e55f100}} , we downsample the weights for patch embeddings and resize patches to {{formula:d77dca25-2e62-4422-9b85-9101a8901574}} , so that there are still {{formula:e266b0cd-1008-4f9b-a4c2-409021c2973a}}  patches and we name the new model as ViT-B/4.
Though ViT originally enlarged input images on CIFAR-10 for natural fine-tuning and evaluation, we keep the input size as {{formula:e6f92b47-5999-47f7-8632-04baf364f2a6}}  to make the attack radius comparable.
For training, we use PGD-7 (PGD with 7 iterations) {{cite:8eafd59}} and TRADES {{cite:1ceec9f}} methods respectively, with no additional data during adversarial training.
We compare ViT with two CNNs, ResNet18 {{cite:b51d59a}} and WideResNet-34-10 {{cite:58faff0}}.
To save training cost, we train each model for 20 epochs only, although some prior works used around hundreds of epochs {{cite:8eafd59}}, {{cite:0ac11bd}} and are very costly for large models.
We use a batch size of 128, an initial learning rate of 0.1, an SGD optimizer with momentum 0.9, and the learning rate decays after 15 epochs and 18 epochs respectively with a rate of 0.1.
While we use a weight decay of {{formula:2754b4c2-3d57-40d6-933d-b66dc545f44e}}  for CNNs as suggested by {{cite:0ac11bd}} that {{formula:92732ee1-f002-41d0-b676-ae9dbf901679}}  is better than {{formula:e1924783-190c-4931-832c-44b73479daec}} , we still use {{formula:bd63ddf9-6504-448f-bbf8-8b55dbe8f94f}}  for ViT as we find {{formula:29e834b9-d810-4dc1-a24e-4a21799b7227}}  causes an underfitting for ViT.
We evaluate the models with PGD-10 (PGD with 10 iterations) and AutoAttack respectively.
Results of adversarial training for different models using PGD-7 (7-step PGD attack) and TRADES respectively on CIFAR-10. ViT-B/4 is a variant of ViT-B/16 where we downsample the patch embedding kernel from {{formula:a22b346e-faab-44d4-9dca-554ed0633cba}}  to {{formula:83df7d83-ce57-498e-9c98-0747cd11894d}}  to accommodate the smaller image size on CIFAR-10. We report the clean accuracy (%) and robust accuracy (%) evaluated with PGD-10 and AutoAttack respectively. Each model is trained using only 20 epochs to reduce the cost.

[1pt]
Model
Method
Clean
PGD-10
AutoAttack
2*PreActResNet18
PGD-7
77.3
48.9
44.4

TRADES
77.6
49.4
44.9
2*WideResNet-34-10
PGD-7
80.3
52.2
48.4

TRADES
81.6
53.4
49.3
2*ViT-B/4
PGD-7
85.9
51.7
47.6

TRADES
85.0
53.9
49.2
[1pt]
{{table:41276a31-f7e0-48d8-bf9b-863b59e2a54d}}{{table:e85d06fb-7885-4683-86a9-7c7d065e0ef0}}
Results
We show the results in Table {{table:e85d06fb-7885-4683-86a9-7c7d065e0ef0}} .
The ViT model achieves higher robust accuracy compared to ResNet18, and comparable robust accuracy compared to WideResNet-34-10, while ViT achieves much better clean accuracy compared to the other two models.
Here ViT does not advance the robust accuracy after adversarial training compared to large CNNs such as WideResNet-34-10.
We conjecture that ViT may need larger training data or longer training epochs to further improve its robust training performance, inspired by the fact that on natural training ViT is not able to perform well either without large-scale pre-training.
And although T2T-ViT improved the performance of natural training when trained from scratch, our previous results in Table {{table:febd43eb-3fee-4635-9f8e-874a79e28d48}}  and Table {{table:d4a250db-9bd4-4134-b563-76395b90b724}}  show that the T2T-ViT structure may be inherently less robust.
We have also tried {{cite:d1da35a}} which was proposed to mitigate the overfitting of FGSM to conduct fast adversarial training with FGSM, but we find that it can still cause catastrophic overfitting for ViT such that the test accuracy on PGD attacks remains almost 0.
We conjecture that this fast training method may be not suitable for pre-trained models or require further adjustments.
Our experiments in this section demonstrate that the adversarial training framework with PGD or TRADES is applicable for transformers on vision tasks, and we provide baseline results and insights for future exploration and improvement.


Semi-Supervised Classification with Graph Convolutional Networks

Abstract: We present a scalable approach for semi-supervised learning on graph-structured data that is based on an efficient variant of convolutional neural networks which operate directly on graphs. We motivate the choice of our convolutional architecture via a localized first-order approximation of spectral graph convolutions. Our model scales linearly in the number of graph edges and learns hidden layer representations that encode both local graph structure and features of nodes. In a number of experiments on citation networks and on a knowledge graph dataset we demonstrate that our approach outperforms related methods by a significant margin.

Inductive Representation Learning on Large Graphs

Abstract: Low-dimensional embeddings of nodes in large graphs have proved extremely useful in a variety of prediction tasks, from content recommendation to identifying protein functions. However, most existing approaches require that all nodes in the graph are present during training of the embeddings; these previous approaches are inherently transductive and do not naturally generalize to unseen nodes. Here we present GraphSAGE, a general, inductive framework that leverages node feature information (e.g., text attributes) to efficiently generate node embeddings for previously unseen data. Instead of training individual embeddings for each node, we learn a function that generates embeddings by sampling and aggregating features from a node's local neighborhood. Our algorithm outperforms strong baselines on three inductive node-classification benchmarks: we classify the category of unseen nodes in evolving information graphs based on citation and Reddit post data, and we show that our algorithm generalizes to completely unseen graphs using a multi-graph dataset of protein-protein interactions.

FastGCN: Fast Learning with Graph Convolutional Networks via Importance Sampling

Abstract: The graph convolutional networks (GCN) recently proposed by Kipf and Welling are an effective graph model for semi-supervised learning. This model, however, was originally designed to be learned with the presence of both training and test data. Moreover, the recursive neighborhood expansion across layers poses time and memory challenges for training with large, dense graphs. To relax the requirement of simultaneous availability of test data, we interpret graph convolutions as integral transforms of embedding functions under probability measures. Such an interpretation allows for the use of Monte Carlo approaches to consistently estimate the integrals, which in turn leads to a batched training scheme as we propose in this work---FastGCN. Enhanced with importance sampling, FastGCN not only is efficient for training but also generalizes well for inference. We show a comprehensive set of experiments to demonstrate its effectiveness compared with GCN and related models. In particular, training is orders of magnitude more efficient while predictions remain comparably accurate.

Adaptive Sampling Towards Fast Graph Representation Learning

Abstract: Graph Convolutional Networks (GCNs) have become a crucial tool on learning representations of graph vertices. The main challenge of adapting GCNs on large-scale graphs is the scalability issue that it incurs heavy cost both in computation and memory due to the uncontrollable neighborhood expansion across layers. In this paper, we accelerate the training of GCNs through developing an adaptive layer-wise sampling method. By constructing the network layer by layer in a top-down passway, we sample the lower layer conditioned on the top one, where the sampled neighborhoods are shared by different parent nodes and the over expansion is avoided owing to the fixed-size sampling. More importantly, the proposed sampler is adaptive and applicable for explicit variance reduction, which in turn enhances the training of our method. Furthermore, we propose a novel and economical approach to promote the message passing over distant nodes by applying skip connections. Intensive experiments on several benchmarks verify the effectiveness of our method regarding the classification accuracy while enjoying faster convergence speed.

Layer-Dependent Importance Sampling for Training Deep and Large Graph Convolutional Networks

Abstract: Graph convolutional networks (GCNs) have recently received wide attentions, due to their successful applications in different graph tasks and different domains. Training GCNs for a large graph, however, is still a challenge. Original full-batch GCN training requires calculating the representation of all the nodes in the graph per GCN layer, which brings in high computation and memory costs. To alleviate this issue, several sampling-based methods are proposed to train GCNs on a subset of nodes. Among them, the node-wise neighbor-sampling method recursively samples a fixed number of neighbor nodes, and thus its computation cost suffers from exponential growing neighbor size across layers; while the layer-wise importance-sampling method discards the neighbor-dependent constraints, and thus the nodes sampled across layer suffer from sparse connection problem. To deal with the above two problems, we propose a new effective sampling algorithm called LAyer-Dependent ImportancE Sampling (LADIES). Based on the sampled nodes in the upper layer, LADIES selects nodes that are in the neighborhood of these nodes and uses the constructed bipartite graph to compute the importance probability. Then, it samples a fixed number of nodes according to the probability for the whole layer, and recursively conducts such procedure per layer to construct the whole computation graph. We prove theoretically and experimentally, that our proposed sampling algorithm outperforms the previous sampling methods regarding both time and memory. Furthermore, LADIES is shown to have better generalization accuracy than original full-batch GCN, due to its stochastic nature.

Cluster-GCN: An Efficient Algorithm for Training Deep and Large Graph Convolutional Networks

Abstract: Graph convolutional network (GCN) has been successfully applied to many graph-based applications; however, training a large-scale GCN remains challenging. Current SGD-based algorithms suffer from either a high computational cost that exponentially grows with number of GCN layers, or a large space requirement for keeping the entire graph and the embedding of each node in memory. In this paper, we propose Cluster-GCN, a novel GCN algorithm that is suitable for SGD-based training by exploiting the graph clustering structure. Cluster-GCN works as the following: at each step, it samples a block of nodes that associate with a dense subgraph identified by a graph clustering algorithm, and restricts the neighborhood search within this subgraph. This simple but effective strategy leads to significantly improved memory and computational efficiency while being able to achieve comparable test accuracy with previous algorithms. To test the scalability of our algorithm, we create a new Amazon2M data with 2 million nodes and 61 million edges which is more than 5 times larger than the previous largest publicly available dataset (Reddit). For training a 3-layer GCN on this data, Cluster-GCN is faster than the previous state-of-the-art VR-GCN (1523 seconds vs 1961 seconds) and using much less memory (2.2GB vs 11.2GB). Furthermore, for training 4 layer GCN on this data, our algorithm can finish in around 36 minutes while all the existing GCN training algorithms fail to train due to the out-of-memory issue. Furthermore, Cluster-GCN allows us to train much deeper GCN without much time and memory overhead, which leads to improved prediction accuracy---using a 5-layer Cluster-GCN, we achieve state-of-the-art test F1 score 99.36 on the PPI dataset, while the previous best result was 98.71 by [16]. Our codes are publicly available at https://github.com/google-research/google-research/tree/master/cluster_gcn.

Accurate, Efficient and Scalable Graph Embedding

Abstract: The Graph Convolutional Network (GCN) model and its variants are powerful graph embedding tools for facilitating classification and clustering on graphs. However, a major challenge is to reduce the complexity of layered GCNs and make them parallelizable and scalable on very large graphs -- state-of the art techniques are unable to achieve scalability without losing accuracy and efficiency. In this paper, we propose novel parallelization techniques for graph sampling-based GCNs that achieve superior scalable performance on very large graphs without compromising accuracy. Specifically, our GCN guarantees work-efficient training and produces order of magnitude savings in computation and communication. To scale GCN training on tightly-coupled shared memory systems, we develop parallelization strategies for the key steps in training: For the graph sampling step, we exploit parallelism within and across multiple sampling instances, and devise an efficient data structure for concurrent accesses that provides theoretical guarantee of near-linear speedup with number of processing units. For the feature propagation step within the sampled graph, we improve cache utilization and reduce DRAM communication by data partitioning. We prove that our partitioning strategy is a 2-approximation for minimizing the communication time compared to the optimal strategy. We demonstrate that our parallel graph embedding outperforms state-of-the-art methods in scalability (with respect to number of processors, graph size and GCN model size), efficiency and accuracy on several large datasets. On a 40-core Xeon platform, our parallel training achieves $64\times$ speedup (with AVX) in the sampling step and $25\times$ speedup in the feature propagation step, compared to the serial implementation, resulting in a net speedup of $21\times$.

Heterogeneous Graph Transformer

Abstract: Recent years have witnessed the emerging success of graph neural networks (GNNs) for modeling structured data. However, most GNNs are designed for homogeneous graphs, in which all nodes and edges belong to the same types, making it infeasible to represent heterogeneous structures. In this paper, we present the Heterogeneous Graph Transformer (HGT) architecture for modeling Web-scale heterogeneous graphs. To model heterogeneity, we design node- and edge-type dependent parameters to characterize the heterogeneous attention over each edge, empowering HGT to maintain dedicated representations for different types of nodes and edges. To handle Web-scale graph data, we design the heterogeneous mini-batch graph sampling algorithm—HGSampling—for efficient and scalable training. Extensive experiments on the Open Academic Graph of 179 million nodes and 2 billion edges show that the proposed HGT model consistently outperforms all the state-of-the-art GNN baselines by 9–21 on various downstream tasks. The dataset and source code of HGT are publicly available at https://github.com/acbull/pyHGT.

Available links and the corresponding models

We conduct extensive experiments of vanilla GCN and various sampling methods on common benchmark datasets (which are introduced in TABLE {{table:424b1e87-2732-4ba4-9596-a11d02f7e153}} ) and exhibit the time-accuracy plots for comparison and analysis. As HetGNN {{cite:4fe47b3}} and HGSampling {{cite:08cd1e0}} place emphasis on different applications of the heterogeneous graph respectively and have not conducted experiments on common benchmark datasets, we do not consider conducting experiments for the two methods separately for lacking the control group.
Besides, we record some significant factors in GCN training to provide a detailed performance comparison, including validation accuracy, total epoch number and time cost before convergence, the proportion of sampling in the total time cost (excluding the time of loading data), and the proportion of model training (without regard to model evaluation) time in the total time cost. To impartially exhibit the impact of sampling methods in GCN training, we conduct all experiments on a two-layer GCN model and mainly use their official configurations in model training. We evaluate all the models with available code in GitHub on a Linux server equipped with dual 14-core Intel Xeon E5-2683 v3 CPUs (2.00 GHz) and an NVIDIA Tesla V100 GPU (16 GB memory). Available links and the corresponding sampling methods are listed in TABLE {{table:4c96e279-d2be-4e5b-9959-f40d4473fbe9}} .


Signed Networks in Social Media

Abstract: Relations between users on social media sites often reflect a mixture of positive (friendly) and negative (antagonistic) interactions. In contrast to the bulk of research on social networks that has focused almost exclusively on positive interpretations of links between people, we study how the interplay between positive and negative relationships affects the structure of on-line social networks. We connect our analyses to theories of signed networks from social psychology. We find that the classical theory of structural balance tends to capture certain common patterns of interaction, but that it is also at odds with some of the fundamental phenomena we observe --- particularly related to the evolving, directed nature of these on-line networks. We then develop an alternate theory of status that better explains the observed edge signs and provides insights into the underlying social mechanisms. Our work provides one of the first large-scale evaluations of theories of signed networks using on-line datasets, as well as providing a perspective for reasoning about social media sites.

Signed Networks in Social Media

Graph Evolution: Densification and Shrinking Diameters

Abstract: How do real graphs evolve over time? What are ``normal'' growth patterns in social, technological, and information networks? Many studies have discovered patterns in static graphs, identifying properties in a single snapshot of a large network, or in a very small number of snapshots; these include heavy tails for in- and out-degree distributions, communities, small-world phenomena, and others. However, given the lack of information about network evolution over long periods, it has been hard to convert these findings into statements about trends over time. Here we study a wide range of real graphs, and we observe some surprising phenomena. First, most of these graphs densify over time, with the number of edges growing super-linearly in the number of nodes. Second, the average distance between nodes often shrinks over time, in contrast to the conventional wisdom that such distance parameters should increase slowly as a function of the number of nodes (like O(log n) or O(log(log n)). Existing graph generation models do not exhibit these types of behavior, even at a qualitative level. We provide a new graph generator, based on a ``forest fire'' spreading process, that has a simple, intuitive justification, requires very few parameters (like the ``flammability'' of nodes), and produces graphs exhibiting the full range of properties observed both in prior work and in the present study. We also notice that the ``forest fire'' model exhibits a sharp transition between sparse graphs and graphs that are densifying. Graphs with decreasing distance between the nodes are generated around this transition point.

Signed Networks in Social Media

Motifs in Temporal Networks

Abstract: Networks are a fundamental tool for modeling complex systems in a variety of domains including social and communication networks as well as biology and neuroscience. Small subgraph patterns in networks, called network motifs, are crucial to understanding the structure and function of these systems. However, the role of network motifs in temporal networks, which contain many timestamped links between the nodes, is not yet well understood. Here we develop a notion of a temporal network motif as an elementary unit of temporal networks and provide a general methodology for counting such motifs. We define temporal network motifs as induced subgraphs on sequences of temporal edges, design fast algorithms for counting temporal motifs, and prove their runtime complexity. Our fast algorithms achieve up to 56.5x speedup compared to a baseline method. Furthermore, we use our algorithms to count temporal motifs in a variety of networks. Results show that networks from different domains have significantly different motif counts, whereas networks from the same domain tend to have similar motif counts. We also find that different motifs occur at different time scales, which provides further insights into structure and function of temporal networks.

Motifs in Temporal Networks

Motifs in Temporal Networks

The Dynamics of Viral Marketing

Abstract: We present an analysis of a person-to-person recommendation network, consisting of 4 million people who made 16 million recommendations on half a million products. We observe the propagation of recommendations and the cascade sizes, which we explain by a simple stochastic model. We analyze how user behavior varies within user communities defined by a recommendation network. Product purchases follow a 'long tail' where a significant share of purchases belongs to rarely sold items. We establish how the recommendation network grows over time and how effective it is from the viewpoint of the sender and receiver of the recommendations. While on average recommendations are not very effective at inducing purchases and do not spread very far, we present a model that successfully identifies communities, product and pricing categories for which viral marketing seems to be very effective.

The Dynamics of Viral Marketing

Specifications of our large networks.

The experiments are done on an Intel
processor clocked at {{formula:be580e84-7fbc-4c33-a4b7-6a15f6f95e6d}}  GHz with 3 GB
main memory, running Ubuntu Linux {{formula:726ff30d-69fc-414f-abc1-f629c3194b54}}  LTS.
We perform our tests over 16 real-world datasets
from the SNAP repositoryhttps://snap.stanford.edu/data/.
The specifications of the datasets
are summarized in Table {{table:5c8b96f2-0cc9-492e-904b-517ae36a00ce}} .
Over each dataset,
we choose 500 pairs of vertices {{formula:57104f0c-1215-4c15-baef-ecb0f4787c4c}}  and {{formula:c0c256d3-4106-4108-a45b-c2046b1a4700}} ,
uniformly at random.
Then, we run our proposed pruning technique
for each pair.
In the end, we report maximum size of
{{formula:c1671ff7-1ca3-4864-9e31-50fed26f9fd7}}
divided by {{formula:9cf85430-547a-4c53-b06a-46ba96712e37}}
(in the {{formula:f7ee2004-aca1-479f-a909-90668ee8b4cd}}  column)
and the maximum time to compute
{{formula:9e83e788-4f0b-4fb6-a2ed-cb262f712f4f}}
(in the {{formula:20f5a811-62e9-4960-b3f6-4a13cc5164c2}}  column),
over all the sampled pairs.


Large-Margin Softmax Loss for Convolutional Neural Networks

Abstract: Cross-entropy loss together with softmax is arguably one of the most common used supervision components in convolutional neural networks (CNNs). Despite its simplicity, popularity and excellent performance, the component does not explicitly encourage discriminative learning of features. In this paper, we propose a generalized large-margin softmax (L-Softmax) loss which explicitly encourages intra-class compactness and inter-class separability between learned features. Moreover, L-Softmax not only can adjust the desired margin but also can avoid overfitting. We also show that the L-Softmax loss can be optimized by typical stochastic gradient descent. Extensive experiments on four benchmark datasets demonstrate that the deeply-learned features with L-softmax loss become more discriminative, hence significantly boosting the performance on a variety of visual classification and verification tasks.

Ensemble Soft-Margin Softmax Loss for Image Classification

Abstract: Softmax loss is arguably one of the most popular losses to train CNN models for image classification. However, recent works have exposed its limitation on feature discriminability. This paper casts a new viewpoint on the weakness of softmax loss. On the one hand, the CNN features learned using the softmax loss are often inadequately discriminative. We hence introduce a soft-margin softmax function to explicitly encourage the discrmination between different classes. On the other hand, the learned classifier of softmax loss is weak. We propose to assemble multiple these weak classifiers to a strong one, inspired by the recognition that the diversity among weak classifiers is critical to a good ensemble. To achieve the diversity, we adopt the Hilbert-Schmidt Independence Criterion (HSIC). Considering these two aspects in one framework, we design a novel loss, named as Ensemble Soft-Margin Softmax (EM-Softmax). Extensive experiments on benchmark datasets are conducted to show the superiority of our design over the baseline softmax loss and several state-of-the-art alternatives.

Virtual Class Enhanced Discriminative Embedding Learning

Abstract: Recently, learning discriminative features to improve the recognition performances gradually becomes the primary goal of deep learning, and numerous remarkable works have emerged. In this paper, we propose a novel yet extremely simple method Virtual Softmax to enhance the discriminative property of learned features by injecting a dynamic virtual negative class into the original softmax. Injecting virtual class aims to enlarge inter-class margin and compress intra-class distribution by strengthening the decision boundary constraint. Although it seems weird to optimize with this additional virtual class, we show that our method derives from an intuitive and clear motivation, and it indeed encourages the features to be more compact and separable. This paper empirically and experimentally demonstrates the superiority of Virtual Softmax, improving the performances on a variety of object classification and face verification tasks.

Additive Margin Softmax for Face Verification

Abstract: In this paper, we propose a conceptually simple and geometrically interpretable objective function, i.e. additive margin Softmax (AM-Softmax), for deep face verification. In general, the face verification task can be viewed as a metric learning problem, so learning large-margin face features whose intra-class variation is small and inter-class difference is large is of great importance in order to achieve good performance. Recently, Large-margin Softmax and Angular Softmax have been proposed to incorporate the angular margin in a multiplicative manner. In this work, we introduce a novel additive angular margin for the Softmax loss, which is intuitively appealing and more interpretable than the existing works. We also emphasize and discuss the importance of feature normalization in the paper. Most importantly, our experiments on LFW BLUFR and MegaFace show that our additive margin softmax loss consistently performs better than the current state-of-the-art methods using the same network architecture and training dataset. Our code has also been made available at https://github.com/happynear/AMSoftmax

CosFace: Large Margin Cosine Loss for Deep Face Recognition

Abstract: Face recognition has made extraordinary progress owing to the advancement of deep convolutional neural networks (CNNs). The central task of face recognition, including face verification and identification, involves face feature discrimination. However, the traditional softmax loss of deep CNNs usually lacks the power of discrimination. To address this problem, recently several loss functions such as center loss, large margin softmax loss, and angular softmax loss have been proposed. All these improved losses share the same idea: maximizing inter-class variance and minimizing intra-class variance. In this paper, we propose a novel loss function, namely large margin cosine loss (LMCL), to realize this idea from a different perspective. More specifically, we reformulate the softmax loss as a cosine loss by $L_2$ normalizing both features and weight vectors to remove radial variations, based on which a cosine margin term is introduced to further maximize the decision margin in the angular space. As a result, minimum intra-class variance and maximum inter-class variance are achieved by virtue of normalization and cosine decision margin maximization. We refer to our model trained with LMCL as CosFace. Extensive experimental evaluations are conducted on the most popular public-domain face recognition datasets such as MegaFace Challenge, Youtube Faces (YTF) and Labeled Face in the Wild (LFW). We achieve the state-of-the-art performance on these benchmarks, which confirms the effectiveness of our proposed approach.

Large margin classification for face recognition. The methods differ from each other by introducing margins in angular, logit, or cosine spaces and by incorporating scaling factors (Mul) or additive terms (Add). Given {{formula:ada6bbe4-974f-4c15-90fa-1bffce0faa6d}}  as a learnable matrix before softmax and {{formula:5d1555e8-9c11-48fd-92f0-8ac36d1320b1}}  as the feature representation of a DNN, if {{formula:080649f1-e910-47d1-999a-f9499786e481}}  is fixed, then we can rewrite the linear transformation {{formula:8dc47f69-7f07-401e-aff1-cefc4319b23a}}  as {{formula:4a23e0fd-215a-46a5-a8be-1b1fb0ab57d7}}  and incorporate a scaling factor {{formula:ceac5b02-a695-4bd1-8df0-b700c1933ca0}}  and an additive term {{formula:a0a523a4-0436-4d28-b03c-3e6051155fc1}}  into it to encourage the angular margin as {{formula:bb522545-2d75-4b0f-9dcc-353d65c991a4}} . Note that although M-Softmax targets to image classification, it is closely related to the other methods and thus we list it here as well.

Regularization on Lipschitz Constant and Output Margins

As mentioned, there is also a line of work focusing on “margins” in the decision or a representation space (i.e., output margin) of DNNs.
For instance, Tang {{cite:3b06a95}} proposed to replace the softmax cross-entropy loss with an SVM-derived one, leading to a positive margin in the representation space characterized by the penultimate layer and a winning solution to the ICML'13 representation learning challenge {{cite:4b8d51d}}.
Sun et al. {{cite:f8c868d}} introduced margin-based penalties to the objective of training DNNs, motivated by theoretical analyses from the perspective of the margin bound.
The triplet loss {{cite:4ce0cc2}} that imposes a margin of representation distance between each positive sample pair and each negative sample pair was also considered, e.g., in FaceNet {{cite:51f4bab}}.
Since then such output margin has been actively discussed and studied in the face recognition community.
Beside what was applied in FaceNet, angular softmax (A-Softmax)See also large margin softmax (L-Softmax) in {{cite:ace32b6}}, which is very similar to A-Softmax. and additive margin softmax (AM-Softmax) were further developed and used in SphereFace {{cite:790eaaf}} and CosineFace {{cite:925382c}}, respectively, to directly enhance the conventional cross-entropy loss with softmax.
There are also ensemble soft-margin softmax (M-Softma) {{cite:999c4bc}}, virtual softmax (V-Softmax) {{cite:5f82e34}}, large margin cosine loss (LMCL) {{cite:7509aed}}, and additive angular margin loss (AAML) {{cite:028077b}}, just to name a few.
They achieved remarkable success in the task of face identification and verification, and some of them also show promising accuracy for classifying natural scene images.
The difference between these methods mostly lie in the way of decomposing the cross-entropy loss. Table {{table:e3914ab9-6e71-4a7d-88e7-b2ed3d99449c}}  compares them.
It is also worth mentioning that the cross-entropy loss itself can also be interpreted as a margin-based loss, with input-specific margins in the output space, and it was shown that enlarging such input-specific margins could be used as a regularization and led to improved test-set accuracy {{cite:fe0073c}}.


Fine-Grained Visual Classification of Aircraft

Abstract: This paper introduces FGVC-Aircraft, a new dataset containing 10,000 images of aircraft spanning 100 aircraft models, organised in a three-level hierarchy. At the finer level, differences between models are often subtle but always visually measurable, making visual recognition challenging but possible. A benchmark is obtained by defining corresponding classification tasks and evaluation protocols, and baseline results are presented. The construction of this dataset was made possible by the work of aircraft enthusiasts, a strategy that can extend to the study of number of other object classes. Compared to the domains usually considered in fine-grained visual classification (FGVC), for example animals, aircraft are rigid and hence less deformable. They, however, present other interesting modes of variation, including purpose, size, designation, structure, historical style, and branding.

Describing Textures in the Wild

Abstract: Patterns and textures are key characteristics of many natural objects: a shirt can be striped, the wings of a butterfly can be veined, and the skin of an animal can be scaly. Aiming at supporting this dimension in image understanding, we address the problem of describing textures with semantic attributes. We identify a vocabulary of forty-seven texture terms and use them to describe a large dataset of patterns collected "in the wild". The resulting Describable Textures Dataset (DTD) is a basis to seek the best representation for recognizing describable texture attributes in images. We port from object recognition to texture recognition the Improved Fisher Vector (IFV) and Deep Convolutional-network Activation Features (DeCAF), and show that surprisingly, they both outperform specialized texture descriptors not only on our problem, but also in established material recognition datasets. We also show that our describable attributes are excellent texture descriptors, transferring between datasets and tasks, in particular, combined with IFV and DeCAF, they significantly outperform the state-of-the-art by more than 10% on both FMD and KTH-TIPS-2b benchmarks. We also demonstrate that they produce intuitive descriptions of materials and Internet images.

Statistics of the 7 small datasets used in the paper.

We used 7 small datasets for our experiments, as shown in Table {{table:e739313d-a085-4321-821a-7528530ea946}} . First, we demonstrate the effectiveness of small resolution on small datasets as well as the large-scale ImageNet {{cite:2172fc3}} in Section REF . Then, we investigate the effect of removing the last residual block in Section REF . Finally, we explore the power of small data in Section REF . All our experiments were conducted using PyTorch and we used Titan Xp GPUs for ImageNet experiments and Tesla K80 GPUs for small datasets. Codes will be made publicly available.

Results on CUB200 and other small datasets

We experimented with all the datasets in Table {{table:e739313d-a085-4321-821a-7528530ea946}}  and more results are shown in Figure {{figure:8d52952e-ce45-4220-aadb-974e8e1c024e}}  (c.f. appendix for precise values). We can find consistent improvements of small resolution on all these datasets with much less training costs.


TSP, SAT, Max-Cut, partitioning, maximum independent set problem

Computer fluid dynamics (CFD), crash simulation, automotive structure design

Jorda-Wigner, Bravyi-Kitaev Superfast, Parity encoding

Sets of differential equations, system of linear equations

A Quantum Approximate Optimization Algorithm

Abstract: We introduce a quantum algorithm that produces approximate solutions for combinatorial optimization problems. The algorithm depends on a positive integer p and the quality of the approximation improves as p is increased. The quantum circuit that implements the algorithm consists of unitary gates whose locality is at most the locality of the objective function whose optimum is sought. The depth of the circuit grows linearly with p times (at worst) the number of constraints. If p is fixed, that is, independent of the input size, the algorithm makes use of efficient classical preprocessing. If p grows with the input size a different strategy is proposed. We study the algorithm as applied to MaxCut on regular graphs and analyze its performance on 2-regular and 3-regular graphs for fixed p. For p = 1, on 3-regular graphs the quantum algorithm always finds a cut that is at least 0.6924 times the size of the optimal cut.

Simulation of Many-Body Fermi Systems on a Universal Quantum Computer

Abstract: We provide fast algorithms for simulating many body Fermi systems on a universal quantum computer. Both first and second quantized descriptions are considered, and the relative computational complexities are determined in each case. In order to accommodate fermions using a first quantized Hamiltonian, an efficient quantum algorithm for anti-symmetrization is given. Finally, a simulation of the Hubbard model is discussed in detail.

Quantum algorithm for solving linear systems of equations

Abstract: Solving linear systems of equations is a common problem that arises both on its own and as a subroutine in more complex problems: given a matrix A and a vector b, find a vector x such that Ax=b. We consider the case where one doesn't need to know the solution x itself, but rather an approximation of the expectation value of some operator associated with x, e.g., x'Mx for some matrix M. In this case, when A is sparse, N by N and has condition number kappa, classical algorithms can find x and estimate x'Mx in O(N sqrt(kappa)) time. Here, we exhibit a quantum algorithm for this task that runs in poly(log N, kappa) time, an exponential improvement over the best classical algorithm.

Supervised learning with quantum enhanced feature spaces

Abstract: Machine learning and quantum computing are two technologies each with the potential for altering how computation is performed to address previously untenable problems. Kernel methods for machine learning are ubiquitous for pattern recognition, with support vector machines (SVMs) being the most well-known method for classification problems. However, there are limitations to the successful solution to such problems when the feature space becomes large, and the kernel functions become computationally expensive to estimate. A core element to computational speed-ups afforded by quantum algorithms is the exploitation of an exponentially large quantum state space through controllable entanglement and interference. Here, we propose and experimentally implement two novel methods on a superconducting processor. Both methods represent the feature space of a classification problem by a quantum state, taking advantage of the large dimensionality of quantum Hilbert space to obtain an enhanced solution. One method, the quantum variational classifier builds on [1,2] and operates through using a variational quantum circuit to classify a training set in direct analogy to conventional SVMs. In the second, a quantum kernel estimator, we estimate the kernel function and optimize the classifier directly. The two methods present a new class of tools for exploring the applications of noisy intermediate scale quantum computers [3] to machine learning.

Application of Quantum Approaches to Automotive Applications: Various mid- and long-term approaches for addressing specific optimization, simulation, and machine learning problems exist.

Table {{table:c3ebbf90-d29d-4788-baf2-ecbeee0932df}}  investigates problem domains, classes, mathematical formulation as well as quantum-based approaches for addressing these. Further, we map these to the described industry applications. The majority of applications are in the optimization domain, but notable applications in quantum chemistry, numerical simulations, and machine learning exist.

Quantum chemistry relies heavily on the simulation of Hamiltonian dynamics. By describing a system in the form of a many-body wave function, quantum computers can efficiently simulate their time evolution {{cite:4c287ce}}. A molecular Hamiltonian can be mapped onto a gate-based quantum computer using the mathematical transformations depicted in Table {{table:c3ebbf90-d29d-4788-baf2-ecbeee0932df}} . Different algorithms to simulate the evolution of the many-body Hamiltonian exist. The Variational Quantum Eigensolver (VQE) is designed for NISQ devices and tries to find a ground state of the Hamiltonian by using a classical optimizer in combination with a quantum computer. Quantum Phase Estimation (QPE) is an algorithm frequently used in many areas of quantum computing. It allows to derive the eigenstates and therefore eigenenergies of any Hamiltonian. However, it uses a circuit with high depth and therefore requires error-corrected quantum computers {{cite:4c287ce}}.