Unsupervised Keyphrase Extraction with Multipartite Graphs

Abstract: We propose an unsupervised keyphrase extraction model that encodes topical information within a multipartite graph structure. Our model represents keyphrase candidates and topics in a single graph and exploits their mutually reinforcing relationship to improve candidate ranking. We further introduce a novel mechanism to incorporate keyphrase selection preferences into the model. Experiments conducted on three widely used datasets show significant improvements over state-of-the-art graph-based models.

KEA: Practical Automatic Keyphrase Extraction

Abstract: Keyphrases provide semantic metadata that summarize and characterize documents. This paper describes Kea, an algorithm for automatically extracting keyphrases from text. Kea identifies candidate keyphrases using lexical methods, calculates feature values for each candidate, and uses a machine-learning algorithm to predict which candidates are good keyphrases. The machine learning scheme first builds a prediction model using training documents with known keyphrases, and then uses the model to find keyphrases in new documents. We use a large test corpus to evaluate Kea's effectiveness in terms of how many author-assigned keyphrases are correctly identified. The system is simple, robust, and publicly available.

Pre-trained Language Models for Keyphrase Generation: A Thorough Empirical Study

Abstract: Neural models that do not rely on pre-training have excelled in the keyphrase generation task with large annotated datasets. Meanwhile, new approaches have incorporated pre-trained language models (PLMs) for their data efficiency. However, there lacks a systematic study of how the two types of approaches compare and how different design choices can affect the performance of PLM-based models. To fill in this knowledge gap and facilitate a more informed use of PLMs for keyphrase extraction and keyphrase generation, we present an in-depth empirical study. Formulating keyphrase extraction as sequence labeling and keyphrase generation as sequence-to-sequence generation, we perform extensive experiments in three domains. After showing that PLMs have competitive high-resource performance and state-of-the-art low-resource performance, we investigate important design choices including in-domain PLMs, PLMs with different pre-training objectives, using PLMs with a parameter budget, and different formulations for present keyphrases. Further results show that (1) in-domain BERT-like PLMs can be used to build strong and data-efficient keyphrase generation models; (2) with a fixed parameter budget, prioritizing model depth over width and allocating more layers in the encoder leads to better encoder-decoder models; and (3) introducing four in-domain PLMs, we achieve a competitive performance in the news domain and the state-of-the-art performance in the scientific domain.

Hyperbolic Relevance Matching for Neural Keyphrase Extraction

Abstract: Keyphrase extraction is a fundamental task in natural language processing that aims to extract a set of phrases with important information from a source document. Identifying important keyphrases is the central component of keyphrase extraction, and its main challenge is learning to represent information comprehensively and discriminate importance accurately. In this paper, to address the above issues, we design a new hyperbolic matching model (HyperMatch) to explore keyphrase extraction in hyperbolic space. Concretely, to represent information comprehensively, HyperMatch first takes advantage of the hidden representations in the middle layers of RoBERTa and integrates them as the word embeddings via an adaptive mixing layer to capture the hierarchical syntactic and semantic structures. Then, considering the latent structure information hidden in natural languages, HyperMatch embeds candidate phrases and documents in the same hyperbolic space via a hyperbolic phrase encoder and a hyperbolic document encoder. To discriminate importance accurately, HyperMatch estimates the importance of each candidate phrase by explicitly modeling the phrase-document relevance via the Poincaré distance and optimizes the whole model by minimizing the hyperbolic margin-based triplet loss. Extensive experiments are conducted on six benchmark datasets and demonstrate that HyperMatch outperforms the recent state-of-the-art baselines.

Neural Keyphrase Generation via Reinforcement Learning with Adaptive Rewards

Abstract: Generating keyphrases that summarize the main points of a document is a fundamental task in natural language processing. Although existing generative models are capable of predicting multiple keyphrases for an input document as well as determining the number of keyphrases to generate, they still suffer from the problem of generating too few keyphrases. To address this problem, we propose a reinforcement learning (RL) approach for keyphrase generation, with an adaptive reward function that encourages a model to generate both sufficient and accurate keyphrases. Furthermore, we introduce a new evaluation method that incorporates name variations of the ground-truth keyphrases using the Wikipedia knowledge base. Thus, our evaluation method can more robustly evaluate the quality of predicted keyphrases. Extensive experiments on five real-world datasets of different scales demonstrate that our RL approach consistently and significantly improves the performance of the state-of-the-art generative models with both conventional and new evaluation methods.

Exclusive Hierarchical Decoding for Deep Keyphrase Generation

Abstract: Keyphrase generation (KG) aims to summarize the main ideas of a document into a set of keyphrases. A new setting is recently introduced into this problem, in which, given a document, the model needs to predict a set of keyphrases and simultaneously determine the appropriate number of keyphrases to produce. Previous work in this setting employs a sequential decoding process to generate keyphrases. However, such a decoding method ignores the intrinsic hierarchical compositionality existing in the keyphrase set of a document. Moreover, previous work tends to generate duplicated keyphrases, which wastes time and computing resources. To overcome these limitations, we propose an exclusive hierarchical decoding framework that includes a hierarchical decoding process and either a soft or a hard exclusion mechanism. The hierarchical decoding process is to explicitly model the hierarchical compositionality of a keyphrase set. Both the soft and the hard exclusion mechanisms keep track of previously-predicted keyphrases within a window size to enhance the diversity of the generated keyphrases. Extensive experiments on multiple KG benchmark datasets demonstrate the effectiveness of our method to generate less duplicated and more accurate keyphrases.

Select, Extract and Generate: Neural Keyphrase Generation with Layer-wise Coverage Attention

Abstract: Natural language processing techniques have demonstrated promising results in keyphrase generation. However, one of the major challenges in \emph{neural} keyphrase generation is processing long documents using deep neural networks. Generally, documents are truncated before given as inputs to neural networks. Consequently, the models may miss essential points conveyed in the target document. To overcome this limitation, we propose \emph{SEG-Net}, a neural keyphrase generation model that is composed of two major components, (1) a selector that selects the salient sentences in a document and (2) an extractor-generator that jointly extracts and generates keyphrases from the selected sentences. SEG-Net uses Transformer, a self-attentive architecture, as the basic building block with a novel \emph{layer-wise} coverage attention to summarize most of the points discussed in the document. The experimental results on seven keyphrase generation benchmarks from scientific and web documents demonstrate that SEG-Net outperforms the state-of-the-art neural generative methods by a large margin.

One2Set: Generating Diverse Keyphrases as a Set

Abstract: Recently, the sequence-to-sequence models have made remarkable progress on the task of keyphrase generation (KG) by concatenating multiple keyphrases in a predefined order as a target sequence during training. However, the keyphrases are inherently an unordered set rather than an ordered sequence. Imposing a predefined order will introduce wrong bias during training, which can highly penalize shifts in the order between keyphrases. In this work, we propose a new training paradigm One2Set without predefining an order to concatenate the keyphrases. To fit this paradigm, we propose a novel model that utilizes a fixed set of learned control codes as conditions to generate a set of keyphrases in parallel. To solve the problem that there is no correspondence between each prediction and target during training, we propose a $K$-step target assignment mechanism via bipartite matching, which greatly increases the diversity and reduces the duplication ratio of generated keyphrases. The experimental results on multiple benchmarks demonstrate that our approach significantly outperforms the state-of-the-art methods.

Learning Rich Representation of Keyphrases from Text

Abstract: In this work, we explore how to train task-specific language models aimed towards learning rich representation of keyphrases from text documents. We experiment with different masking strategies for pre-training transformer language models (LMs) in discriminative as well as generative settings. In the discriminative setting, we introduce a new pre-training objective - Keyphrase Boundary Infilling with Replacement (KBIR), showing large gains in performance (upto 8.16 points in F1) over SOTA, when the LM pre-trained using KBIR is fine-tuned for the task of keyphrase extraction. In the generative setting, we introduce a new pre-training setup for BART - KeyBART, that reproduces the keyphrases related to the input text in the CatSeq format, instead of the denoised original input. This also led to gains in performance (upto 4.33 points in F1@M) over SOTA for keyphrase generation. Additionally, we also fine-tune the pre-trained language models on named entity recognition (NER), question answering (QA), relation extraction (RE), abstractive summarization and achieve comparable performance with that of the SOTA, showing that learning rich representation of keyphrases is indeed beneficial for many other fundamental NLP tasks.

KPEval: Towards Fine-Grained Semantic-Based Keyphrase Evaluation

Abstract: Despite the significant advancements in keyphrase extraction and keyphrase generation methods, the predominant approach for evaluation mainly relies on exact matching with human references. This scheme fails to recognize systems that generate keyphrases semantically equivalent to the references or diverse keyphrases that carry practical utility. To better assess the capability of keyphrase systems, we propose KPEval, a comprehensive evaluation framework consisting of four critical aspects: reference agreement, faithfulness, diversity, and utility. For each aspect, we design semantic-based metrics to reflect the evaluation objectives. Meta-evaluation studies demonstrate that our evaluation strategy correlates better with human preferences compared to a range of previously proposed metrics. Using KPEval, we re-evaluate 21 keyphrase systems and discover that (1) established model comparison results have blind-spots especially when considering reference-free evaluation; (2) large language models are underestimated by prior evaluation works; and (3) there is no single best model that can excel in all the aspects.

Descriptions of the studied models. "KPE" = "Keyphrase Extraction". "KPG" = "Keyphrase Generation". "PLM" = "Pre-trained Language Model". "TAPT" = "Task-adaptive Pre-training".

Detailed information of the considered keyphrase systems

We summarize the properties of all the considered keyphrase systems in Table {{table:992168f4-9408-4da2-84b1-448e7f171108}} . We benchmark 7 keyphrase extraction models and 11 keyphrase generation models. Among the keyphrase generation models, 5 models are trained from scratch, 4 models are fine-tuned from pre-trained language models, and 2 are prompting GPT-3.5 either zero-shot or with with examples.


What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models

Abstract: Pre-training by language modeling has become a popular and successful approach to NLP tasks, but we have yet to understand exactly what linguistic capacities these pre-training processes confer upon models. In this paper we introduce a suite of diagnostics drawn from human language experiments, which allow us to ask targeted questions about the information used by language models for generating predictions in context. As a case study, we apply these diagnostics to the popular BERT model, finding that it can generally distinguish good from bad completions involving shared category or role reversal, albeit with less sensitivity than humans, and it robustly retrieves noun hypernyms, but it struggles with challenging inferences and role-based event prediction -- and in particular, it shows clear insensitivity to the contextual impacts of negation.

BLiMP: The Benchmark of Linguistic Minimal Pairs for English

Abstract: We introduce The Benchmark of Linguistic Minimal Pairs (shortened to BLiMP), a challenge set for evaluating what language models (LMs) know about major grammatical phenomena in English. BLiMP consists of 67 sub-datasets, each containing 1000 minimal pairs isolating specific contrasts in syntax, morphology, or semantics. The data is automatically generated according to expert-crafted grammars, and aggregate human agreement with the labels is 96.4%. We use it to evaluate n-gram, LSTM, and Transformer (GPT-2 and Transformer-XL) LMs. We find that state-of-the-art models identify morphological contrasts reliably, but they struggle with semantic restrictions on the distribution of quantifiers and negative polarity items and subtle syntactic phenomena such as extraction islands.

Language Models as Knowledge Bases?

Abstract: Recent progress in pretraining language models on large textual corpora led to a surge of improvements for downstream NLP tasks. Whilst learning linguistic knowledge, these models may also be storing relational knowledge present in the training data, and may be able to answer queries structured as "fill-in-the-blank" cloze statements. Language models have many advantages over structured knowledge bases: they require no schema engineering, allow practitioners to query about an open class of relations, are easy to extend to more data, and require no human supervision to train. We present an in-depth analysis of the relational knowledge already present (without fine-tuning) in a wide range of state-of-the-art pretrained language models. We find that (i) without fine-tuning, BERT contains relational knowledge competitive with traditional NLP methods that have some access to oracle knowledge, (ii) BERT also does remarkably well on open-domain question answering against a supervised baseline, and (iii) certain types of factual knowledge are learned much more readily than others by standard language model pretraining approaches. The surprisingly strong ability of these models to recall factual knowledge without any fine-tuning demonstrates their potential as unsupervised open-domain QA systems. The code to reproduce our analysis is available at https://github.com/facebookresearch/LAMA.

X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models

Abstract: Language models (LMs) have proven surprisingly successful at capturing factual knowledge by completing cloze-style fill-in-the-blank questions such as "Punta Cana is located in _." However, while knowledge is both written and queried in many languages, studies on LMs' factual representation ability have almost invariably been performed on English. To assess factual knowledge retrieval in LMs in different languages, we create a multilingual benchmark of cloze-style probes for 23 typologically diverse languages. To properly handle language variations, we expand probing methods from single- to multi-word entities, and develop several decoding algorithms to generate multi-token predictions. Extensive experimental results provide insights about how well (or poorly) current state-of-the-art LMs perform at this task in languages with more or fewer available resources. We further propose a code-switching-based method to improve the ability of multilingual LMs to access knowledge, and verify its effectiveness on several benchmark languages. Benchmark data and code have been released at https://x-factr.github.io.

Multilingual LAMA: Investigating Knowledge in Multilingual Pretrained Language Models

Abstract: Recently, it has been found that monolingual English language models can be used as knowledge bases. Instead of structural knowledge base queries, masked sentences such as "Paris is the capital of [MASK]" are used as probes. We translate the established benchmarks TREx and GoogleRE into 53 languages. Working with mBERT, we investigate three questions. (i) Can mBERT be used as a multilingual knowledge base? Most prior work only considers English. Extending research to multiple languages is important for diversity and accessibility. (ii) Is mBERT's performance as knowledge base language-independent or does it vary from language to language? (iii) A multilingual model is trained on more text, e.g., mBERT is trained on 104 Wikipedias. Can mBERT leverage this for better performance? We find that using mBERT as a knowledge base yields varying performance across languages and pooling predictions across languages improves performance. Conversely, mBERT exhibits a language bias; e.g., when queried in Italian, it tends to predict Italy as the country of origin.

Can Language Models be Biomedical Knowledge Bases?

Abstract: Pre-trained language models (LMs) have become ubiquitous in solving various natural language processing (NLP) tasks. There has been increasing interest in what knowledge these LMs contain and how we can extract that knowledge, treating LMs as knowledge bases (KBs). While there has been much work on probing LMs in the general domain, there has been little attention to whether these powerful LMs can be used as domain-specific KBs. To this end, we create the BioLAMA benchmark, which is comprised of 49K biomedical factual knowledge triples for probing biomedical LMs. We find that biomedical LMs with recently proposed probing methods can achieve up to 18.51% Acc@5 on retrieving biomedical knowledge. Although this seems promising given the task difficulty, our detailed analyses reveal that most predictions are highly correlated with prompt templates without any subjects, hence producing similar results on each relation and hindering their capabilities to be used as domain-specific KBs. We hope that BioLAMA can serve as a challenging benchmark for biomedical factual probing.

Evaluating Commonsense in Pre-trained Language Models

Abstract: Contextualized representations trained over large raw text data have given remarkable improvements for NLP tasks including question answering and reading comprehension. There have been works showing that syntactic, semantic and word sense knowledge are contained in such representations, which explains why they benefit such tasks. However, relatively little work has been done investigating commonsense knowledge contained in contextualized representations, which is crucial for human question answering and reading comprehension. We study the commonsense ability of GPT, BERT, XLNet, and RoBERTa by testing them on seven challenging benchmarks, finding that language modeling and its variants are effective objectives for promoting models' commonsense ability while bi-directional context and larger training set are bonuses. We additionally find that current models do poorly on tasks require more necessary inference steps. Finally, we test the robustness of models by making dual test cases, which are correlated so that the correct prediction of one sample should lead to correct prediction of the other. Interestingly, the models show confusion on these test cases, which suggests that they learn commonsense at the surface rather than the deep level. We release a test set, named CATs publicly, for future research.

Birds have four legs?! NumerSense: Probing Numerical Commonsense Knowledge of Pre-trained Language Models

Abstract: Recent works show that pre-trained language models (PTLMs), such as BERT, possess certain commonsense and factual knowledge. They suggest that it is promising to use PTLMs as "neural knowledge bases" via predicting masked words. Surprisingly, we find that this may not work for numerical commonsense knowledge (e.g., a bird usually has two legs). In this paper, we investigate whether and to what extent we can induce numerical commonsense knowledge from PTLMs as well as the robustness of this process. To study this, we introduce a novel probing task with a diagnostic dataset, NumerSense, containing 13.6k masked-word-prediction probes (10.5k for fine-tuning and 3.1k for testing). Our analysis reveals that: (1) BERT and its stronger variant RoBERTa perform poorly on the diagnostic dataset prior to any fine-tuning; (2) fine-tuning with distant supervision brings some improvement; (3) the best supervised model still performs poorly as compared to human performance (54.06% vs 96.3% in accuracy).

oLMpics -- On what Language Model Pre-training Captures

Abstract: Recent success of pre-trained language models (LMs) has spurred widespread interest in the language capabilities that they possess. However, efforts to understand whether LM representations are useful for symbolic reasoning tasks have been limited and scattered. In this work, we propose eight reasoning tasks, which conceptually require operations such as comparison, conjunction, and composition. A fundamental challenge is to understand whether the performance of a LM on a task should be attributed to the pre-trained representations or to the process of fine-tuning on the task data. To address this, we propose an evaluation protocol that includes both zero-shot evaluation (no fine-tuning), as well as comparing the learning curve of a fine-tuned LM to the learning curve of multiple controls, which paints a rich picture of the LM capabilities. Our main findings are that: (a) different LMs exhibit qualitatively different reasoning abilities, e.g., RoBERTa succeeds in reasoning tasks where BERT fails completely; (b) LMs do not reason in an abstract manner and are context-dependent, e.g., while RoBERTa can compare ages, it can do so only when the ages are in the typical range of human ages; (c) On half of our reasoning tasks all models fail completely. Our findings and infrastructure can help future work on designing new datasets, models and objective functions for pre-training.

Open Sesame: Getting inside BERT’s Linguistic Knowledge

Abstract: How and to what extent does BERT encode syntactically-sensitive hierarchical information or positionally-sensitive linear information? Recent work has shown that contextual representations like BERT perform well on tasks that require sensitivity to linguistic structure. We present here two studies which aim to provide a better understanding of the nature of BERT’s representations. The first of these focuses on the identification of structurally-defined elements using diagnostic classifiers, while the second explores BERT’s representation of subject-verb agreement and anaphor-antecedent dependencies through a quantitative assessment of self-attention vectors. In both cases, we find that BERT encodes positional information about word tokens well on its lower layers, but switches to a hierarchically-oriented encoding on higher layers. We conclude then that BERT’s representations do indeed model linguistically relevant aspects of hierarchical structure, though they do not appear to show the sharp sensitivity to hierarchical structure that is found in human processing of reflexive anaphora.

Linguistic Knowledge and Transferability of Contextual Representations

Abstract: Contextual word representations derived from large-scale neural language models are successful across a diverse set of NLP tasks, suggesting that they encode useful and transferable features of language. To shed light on the linguistic knowledge they capture, we study the representations produced by several recent pretrained contextualizers (variants of ELMo, the OpenAI transformer language model, and BERT) with a suite of seventeen diverse probing tasks. We find that linear models trained on top of frozen contextual representations are competitive with state-of-the-art task-specific models in many cases, but fail on tasks requiring fine-grained linguistic knowledge (e.g., conjunct identification). To investigate the transferability of contextual word representations, we quantify differences in the transferability of individual layers within contextualizers, especially between recurrent neural networks (RNNs) and transformers. For instance, higher layers of RNNs are more task-specific, while transformer layers do not exhibit the same monotonic trend. In addition, to better understand what makes contextual word representations transferable, we compare language model pretraining with eleven supervised pretraining tasks. For any given task, pretraining on a closely related task yields better performance than language model pretraining (which is better on average) when the pretraining dataset is fixed. However, language model pretraining on more data gives the best results.

Investigating BERT’s Knowledge of Language: Five Analysis Methods with NPIs

Abstract: Though state-of-the-art sentence representation models can perform tasks requiring significant knowledge of grammar, it is an open question how best to evaluate their grammatical knowledge. We explore five experimental methods inspired by prior work evaluating pretrained sentence representation models. We use a single linguistic phenomenon, negative polarity item (NPI) licensing, as a case study for our experiments. NPIs like any are grammatical only if they appear in a licensing environment like negation (Sue doesn’t have any cats vs. *Sue has any cats). This phenomenon is challenging because of the variety of NPI licensing environments that exist. We introduce an artificially generated dataset that manipulates key features of NPI licensing for the experiments. We find that BERT has significant knowledge of these features, but its success varies widely across different experimental methods. We conclude that a variety of methods is necessary to reveal all relevant aspects of a model’s grammatical knowledge in a given domain.

What do you learn from context? Probing for sentence structure in contextualized word representations

Abstract: Contextualized representation models such as ELMo (Peters et al., 2018a) and BERT (Devlin et al., 2018) have recently achieved state-of-the-art results on a diverse array of downstream NLP tasks. Building on recent token-level probing work, we introduce a novel edge probing task design and construct a broad suite of sub-sentence tasks derived from the traditional structured NLP pipeline. We probe word-level contextual representations from four recent models and investigate how they encode sentence structure across a range of syntactic, semantic, local, and long-range phenomena. We find that existing models trained on language modeling and translation produce strong representations for syntactic phenomena, but only offer comparably small improvements on semantic tasks over a non-contextual baseline.

Information-Theoretic Probing with Minimum Description Length

Abstract: To measure how well pretrained representations encode some linguistic property, it is common to use accuracy of a probe, i.e. a classifier trained to predict the property from the representations. Despite widespread adoption of probes, differences in their accuracy fail to adequately reflect differences in representations. For example, they do not substantially favour pretrained representations over randomly initialized ones. Analogously, their accuracy can be similar when probing for genuine linguistic labels and probing for random synthetic tasks. To see reasonable differences in accuracy with respect to these random baselines, previous work had to constrain either the amount of probe training data or its model size. Instead, we propose an alternative to the standard probes, information-theoretic probing with minimum description length (MDL). With MDL probing, training a probe to predict labels is recast as teaching it to effectively transmit the data. Therefore, the measure of interest changes from probe accuracy to the description length of labels given representations. In addition to probe quality, the description length evaluates "the amount of effort" needed to achieve the quality. This amount of effort characterizes either (i) size of a probing model, or (ii) the amount of data needed to achieve the high quality. We consider two methods for estimating MDL which can be easily implemented on top of the standard probing pipelines: variational coding and online coding. We show that these methods agree in results and are more informative and stable than the standard probes.

Do Neural Language Representations Learn Physical Commonsense?

Abstract: Humans understand language based on the rich background knowledge about how the physical world works, which in turn allows us to reason about the physical world through language. In addition to the properties of objects (e.g., boats require fuel) and their affordances, i.e., the actions that are applicable to them (e.g., boats can be driven), we can also reason about if-then inferences between what properties of objects imply the kind of actions that are applicable to them (e.g., that if we can drive something then it likely requires fuel). In this paper, we investigate the extent to which state-of-the-art neural language representations, trained on a vast amount of natural language text, demonstrate physical commonsense reasoning. While recent advancements of neural language models have demonstrated strong performance on various types of natural language inference tasks, our study based on a dataset of over 200k newly collected annotations suggests that neural language representations still only learn associations that are explicitly written down.

Summary about some representative knowledge probing benchmarks.

To assess the knowledge in PLMs, lots of benchmarks have been proposed to probe various knowledge contained in PLMs, for example, linguistic knowledge {{cite:e3e6d90}}, {{cite:b2ac481}}, {{cite:b51f5a5}}, {{cite:9e06a67}}, {{cite:4283b0e}}, syntactic knowledge {{cite:328c9f8}}, {{cite:4d67b33}}, factual knowledge {{cite:a742243}}, {{cite:9df6b65}}, {{cite:1c44942}}, {{cite:cab4f1b}}, commonsense knowledge {{cite:63061db}}, {{cite:6ddddd4}}, etc. Table {{table:346042c7-aae6-4a0d-9a73-8cbe7bb9ae8a}}  summarizes several representative knowledge probing benchmarks.

Prompt-based probing is one of the most popular approaches for knowledge probing. To evaluate whether LMs know a specific knowledge such as the birthplace of Michael Jordan, we could query LMs with knowledge queries such as “Michael Jordan was born in .”, where “was born in” is a prompt for a specific type of knowledge.
As shown in Table {{table:346042c7-aae6-4a0d-9a73-8cbe7bb9ae8a}} , prompt-based probing has been widely used in benchmarks such as LAMA {{cite:a742243}}, oLMpics {{cite:16c19e9}}, LM diagnostics {{cite:e3e6d90}}, BIG-bench {{cite:b711ac1}}, etc.

In the future, the main directions of knowledge probing may lie in:
1) Comprehensive benchmark construction.
As we demonstrate in table {{table:346042c7-aae6-4a0d-9a73-8cbe7bb9ae8a}} , current knowledge probing benchmarks are mostly too specialized, which may lead to inconsistent, biased or unreliable results. Therefore it is critical to build a comprehensive and unbiased benchmark.
2) Debiased probing approaches.
Currently prompt-based probing is the dominant knowledge probing methods due to its simplicity. However, there still exist lots of issues in prompt-based probing. Therefore, the design of unbiased datasets and better probing frameworks is another useful direction worth investigating.


End-to-end Optimized Image Compression

Abstract: We describe an image compression method, consisting of a nonlinear analysis transformation, a uniform quantizer, and a nonlinear synthesis transformation. The transforms are constructed in three successive stages of convolutional linear filters and nonlinear activation functions. Unlike most convolutional neural networks, the joint nonlinearity is chosen to implement a form of local gain control, inspired by those used to model biological neurons. Using a variant of stochastic gradient descent, we jointly optimize the entire model for rate-distortion performance over a database of training images, introducing a continuous proxy for the discontinuous loss function arising from the quantizer. Under certain conditions, the relaxed loss function may be interpreted as the log likelihood of a generative model, as implemented by a variational autoencoder. Unlike these models, however, the compression model must operate at any given point along the rate-distortion curve, as specified by a trade-off parameter. Across an independent set of test images, we find that the optimized method generally exhibits better rate-distortion performance than the standard JPEG and JPEG 2000 compression methods. More importantly, we observe a dramatic improvement in visual quality for all images at all bit rates, which is supported by objective quality estimates using MS-SSIM.

Variational image compression with a scale hyperprior

Abstract: We describe an end-to-end trainable model for image compression based on variational autoencoders. The model incorporates a hyperprior to effectively capture spatial dependencies in the latent representation. This hyperprior relates to side information, a concept universal to virtually all modern image codecs, but largely unexplored in image compression using artificial neural networks (ANNs). Unlike existing autoencoder compression methods, our model trains a complex prior jointly with the underlying autoencoder. We demonstrate that this model leads to state-of-the-art image compression when measuring visual quality using the popular MS-SSIM index, and yields rate-distortion performance surpassing published ANN-based methods when evaluated using a more traditional metric based on squared error (PSNR). Furthermore, we provide a qualitative comparison of models trained for different distortion metrics.

Network architectures in our experiments.

We made evaluations using three network architectures: Ballé17 {{cite:d41bf81}}, Ballé18 {{cite:3207899}}, and Cheng20 {{cite:9c68c10}}.
We summarized the network architectures in Table {{table:3b67e963-5bf0-4c2b-ad54-06bd06d342c5}} .
The latter increases the complexity of the architecture.
Specifically, Cheng20 uses six and seven residual blocks {{cite:444b840}} for the encoder and decoder, respectively.
They used simplified non-local attention modules in these layers.
The entropy model is based on Gaussian mixtures, whose parameters are predicted by a hyper-prior and an autoregressive context model.
The quantizer is approximated using AUN-Q.
Some convolutional layers in the encoder and decoder of these three networks are followed by generalized divisible normalization {{cite:d22749b}}.


Introduction to the CoNLL-2003 Shared Task: Language-Independent Named Entity Recognition

Abstract: We describe the CoNLL-2003 shared task: language-independent named entity recognition. We give background information on the data sets (English and German) and the evaluation method, present a general overview of the systems that have taken part in the task and discuss their performance.

SQuAD: 100,000+ Questions for Machine Comprehension of Text

Abstract: We present the Stanford Question Answering Dataset (SQuAD), a new reading comprehension dataset consisting of 100,000+ questions posed by crowdworkers on a set of Wikipedia articles, where the answer to each question is a segment of text from the corresponding reading passage. We analyze the dataset to understand the types of reasoning required to answer the questions, leaning heavily on dependency and constituency trees. We build a strong logistic regression model, which achieves an F1 score of 51.0%, a significant improvement over a simple baseline (20%). However, human performance (86.8%) is much higher, indicating that the dataset presents a good challenge problem for future research. The dataset is freely available at https://stanford-qa.com

Visual Entailment: A Novel Task for Fine-Grained Image Understanding

Abstract:
Existing visual reasoning datasets such as Visual Question Answering (VQA), often suffer from biases conditioned on the question, image or answer distributions. The recently proposed CLEVR dataset addresses these limitations and requires fine-grained reasoning but the dataset is synthetic and consists of similar objects and sentence structures across the dataset.
In this paper, we introduce a new inference task, Visual Entailment (VE) - consisting of image-sentence pairs whereby a premise is defined by an image, rather than a natural language sentence as in traditional Textual Entailment tasks. The goal of a trained VE model is to predict whether the image semantically entails the text. To realize this task, we build a dataset SNLI-VE based on the Stanford Natural Language Inference corpus and Flickr30k dataset. We evaluate various existing VQA baselines and build a model called Explainable Visual Entailment (EVE) system to address the VE task. EVE achieves up to 71% accuracy and outperforms several other state-of-the-art VQA based models. Finally, we demonstrate the explainability of EVE through cross-modal attention visualizations. The SNLI-VE dataset is publicly available at this https URL necla-ml/SNLI-VE.

Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering

Abstract: Problems at the intersection of vision and language are of significant importance both as challenging research questions and for the rich set of applications they enable. However, inherent structure in our world and bias in our language tend to be a simpler signal for learning than visual modalities, resulting in models that ignore visual information, leading to an inflated sense of their capability. We propose to counter these language priors for the task of Visual Question Answering (VQA) and make vision (the V in VQA) matter! Specifically, we balance the popular VQA dataset by collecting complementary images such that every question in our balanced dataset is associated with not just a single image, but rather a pair of similar images that result in two different answers to the question. Our dataset is by construction more balanced than the original VQA dataset and has approximately twice the number of image-question pairs. Our complete balanced dataset is publicly available at www.visualqa.org as part of the 2nd iteration of the Visual Question Answering Dataset and Challenge (VQA v2.0). We further benchmark a number of state-of-art VQA models on our balanced dataset. All models perform significantly worse on our balanced dataset, suggesting that these models have indeed learned to exploit language priors. This finding provides the first concrete empirical evidence for what seems to be a qualitative sense among practitioners. Finally, our data collection protocol for identifying complementary images enables us to develop a novel interpretable model, which in addition to providing an answer to the given (image, question) pair, also provides a counter-example based explanation. Specifically, it identifies an image that is similar to the original image, but it believes has a different answer to the same question. This can help in building trust for machines among their users.

Datasets for training and evaluation

Table {{table:9ebe2c2d-838c-45d7-a6a6-a648f59c4d1f}}  delineates the domains, tasks, and datasets for training and evaluating FM3.


Shopping100k, DARN, iMaterialist, CelebA

TL, AL, feature matching loss. Cycle consistency loss

CL, triplet ranking loss, global ranking loss

Semi-supervised Feature-Level Attribute Manipulation for Fashion Image Retrieval

Abstract: With a growing demand for the search by image, many works have studied the task of fashion instance-level image retrieval (FIR). Furthermore, the recent works introduce a concept of fashion attribute manipulation (FAM) which manipulates a specific attribute (e.g color) of a fashion item while maintaining the rest of the attributes (e.g shape, and pattern). In this way, users can search not only "the same" items but also "similar" items with the desired attributes. FAM is a challenging task in that the attributes are hard to define, and the unique characteristics of a query are hard to be preserved. Although both FIR and FAM are important in real-life applications, most of the previous studies have focused on only one of these problem. In this study, we aim to achieve competitive performance on both FIR and FAM. To do so, we propose a novel method that converts a query into a representation with the desired attributes. We introduce a new idea of attribute manipulation at the feature level, by matching the distribution of manipulated features with real features. In this fashion, the attribute manipulation can be done independently from learning a representation from the image. By introducing the feature-level attribute manipulation, the previous methods for FIR can perform attribute manipulation without sacrificing their retrieval performance.

FashionSearchNet-v2: Learning Attribute Representations with Localization for Image Retrieval with Attribute Manipulation

Abstract: The focus of this paper is on the problem of image retrieval with attribute manipulation. Our proposed work is able to manipulate the desired attributes of the query image while maintaining its other attributes. For example, the collar attribute of the query image can be changed from round to v-neck to retrieve similar images from a large dataset. A key challenge in e-commerce is that images have multiple attributes where users would like to manipulate and it is important to estimate discriminative feature representations for each of these attributes. The proposed FashionSearchNet-v2 architecture is able to learn attribute specific representations by leveraging on its weakly-supervised localization module, which ignores the unrelated features of attributes in the feature space, thus improving the similarity learning. The network is jointly trained with the combination of attribute classification and triplet ranking loss to estimate local representations. These local representations are then merged into a single global representation based on the instructed attribute manipulation where desired images can be retrieved with a distance metric. The proposed method also provides explainability for its retrieval process to help provide additional information on the attention of the network. Experiments performed on several datasets that are rich in terms of the number of attributes show that FashionSearchNet-v2 outperforms the other state-of-the-art attribute manipulation techniques. Different than our earlier work (FashionSearchNet), we propose several improvements in the learning procedure and show that the proposed FashionSearchNet-v2 can be generalized to different domains other than fashion.

Where to Look and How to Describe: Fashion Image Retrieval with an Attentional Heterogeneous Bilinear Network

Abstract: Fashion products typically feature in compositions of a variety of styles at different clothing parts. In order to distinguish images of different fashion products, we need to extract both appearance (i.e., "how to describe") and localization (i.e.,"where to look") information, and their interactions. To this end, we propose a biologically inspired framework for image-based fashion product retrieval, which mimics the hypothesized twostream visual processing system of human brain. The proposed attentional heterogeneous bilinear network (AHBN) consists of two branches: a deep CNN branch to extract fine-grained appearance attributes and a fully convolutional branch to extract landmark localization information. A joint channel-wise attention mechanism is further applied to the extracted heterogeneous features to focus on important channels, followed by a compact bilinear pooling layer to model the interaction of the two streams. Our proposed framework achieves satisfactory performance on three image-based fashion product retrieval benchmarks.

Comparative analysis of Image-guided CBFIR methods with attribute and style guidance, where BCE = binary cross entropy, mAP = mean Average Precision, CL = Classification Loss, TL = Triplet Loss, CEL = Cross-Entropy Loss, AL = Adversarial Loss

Attribute guidance plays an important role in retrieving some specific cloth images with the desired attributes by the consumers, like a particular color, style, pattern, and many more. A main concern to be addressed is that fashion products or items have many attributes, and it is vital for each attribute to have representative features. Attribute manipulation is also challenging in content-based fashion image retrieval because combining query image fashion features and desired attribute representations is a very complex process. Attribute guidance positively impacts fashion retrieval tasks and leaves too much room for future researchers. Figure {{figure:551f5b06-4b61-4e74-a1bf-c3658ca5901c}}  shows some results of retrieved images with different image-guided CBFIR methods with attribute guidance with the perspective of the query reference image and desired attributes in text form. In the recent development of image-guided CBFIR with attribute guidance, the key contribution of authors are explained below and are analyzed comparatively according to CBFIR networks, publication year, datasets, evaluation metrics, and loss functions in Table {{table:5b27b88a-fea9-49f7-93c0-0ea951b67467}} .

Style guidance plays a vital role in content-based fashion image retrieval tasks, as a retrieved fashion image for the wedding may not be suitable for the graduation ceremony. Style, occasion, and season are some key examples of the cultural and environmental factors that make up an outfit's semantics. The retrieved image is considered as a perfect match if the reference and retrieved images are linked with the same style and suitable for the same seasons and events. In the recent development of image-guided CBFIR with style guidance, the key contribution of authors are explained below and are analyzed comparatively according to CBFIR networks, publication year, datasets, evaluation metrics, and loss functions in Table {{table:5b27b88a-fea9-49f7-93c0-0ea951b67467}} .


Restormer: Efficient Transformer for High-Resolution Image Restoration

Abstract: Since convolutional neural networks (CNNs) perform well at learning generalizable image priors from large-scale data, these models have been extensively applied to image restoration and related tasks. Recently, another class of neural architectures, Transformers, have shown significant performance gains on natural language and high-level vision tasks. While the Transformer model mitigates the shortcomings of CNNs (i.e., limited receptive field and inadaptability to input content), its computational complexity grows quadratically with the spatial resolution, therefore making it infeasible to apply to most image restoration tasks involving high-resolution images. In this work, we propose an efficient Transformer model by making several key designs in the building blocks (multi-head attention and feed-forward network) such that it can capture long-range pixel interactions, while still remaining applicable to large images. Our model, named Restoration Transformer (Restormer), achieves state-of-the-art results on several image restoration tasks, including image deraining, single-image motion deblurring, defocus deblurring (single-image and dual-pixel data), and image denoising (Gaussian grayscale/color denoising, and real image denoising). The source code and pre-trained models are available at https://github.com/swz30/Restormer.

Mask-guided Spectral-wise Transformer for Efficient Hyperspectral Image Reconstruction

Abstract: Hyperspectral image (HSI) reconstruction aims to recover the 3D spatial-spectral signal from a 2D measurement in the coded aperture snapshot spectral imaging (CASSI) system. The HSI representations are highly similar and correlated across the spectral dimension. Modeling the inter-spectra interactions is beneficial for HSI reconstruction. However, existing CNN-based methods show limitations in capturing spectral-wise similarity and long-range dependencies. Besides, the HSI information is modulated by a coded aperture (physical mask) in CASSI. Nonetheless, current algorithms have not fully explored the guidance effect of the mask for HSI restoration. In this paper, we propose a novel framework, Mask-guided Spectral-wise Transformer (MST), for HSI reconstruction. Specifically, we present a Spectral-wise Multi-head Self-Attention (S-MSA) that treats each spectral feature as a token and calculates self-attention along the spectral dimension. In addition, we customize a Mask-guided Mechanism (MM) that directs S- MSA to pay attention to spatial regions with high-fidelity spectral representations. Extensive experiments show that our MST significantly outperforms state-of-the-art (SOTA) methods on simulation and real HSI datasets while requiring dramatically cheaper computational and memory costs. https://github.com/caiyuanhao1998/MST/

Spatial-Spectral Transformer for Hyperspectral Image Denoising

Abstract: Hyperspectral image (HSI) denoising is a crucial preprocessing procedure for the subsequent HSI applications. Unfortunately, though witnessing the development of deep learning in HSI denoising area, existing convolution-based methods face the trade-off between computational efficiency and capability to model non-local characteristics of HSI. In this paper, we propose a Spatial-Spectral Transformer (SST) to alleviate this problem. To fully explore intrinsic similarity characteristics in both spatial dimension and spectral dimension, we conduct non-local spatial self-attention and global spectral self-attention with Transformer architecture. The window-based spatial self-attention focuses on the spatial similarity beyond the neighboring region. While, spectral self-attention exploits the long-range dependencies between highly correlative bands. Experimental results show that our proposed method outperforms the state-of-the-art HSI denoising methods in quantitative quality and visual results.

The competing attention mechanisms.

We present HSDT, an effective and flexible transformer for hyperspectral image denoising. Built upon the hybrid hierarchical architecture, our HSDT is equipped with a novel S3Conv, GSSA, and SM-FFN module to effectively integrate the local spectral-spatial inductive bias and the long-range spectral dependency modeling. Our S3Conv extracts highly correlated local spatial-spectral features without harming efficiency and flexibility, while the GSSA provides stronger capabilities for capturing global spectral correlations, guided by a set of learnable queries that encode the band-wise spectral signatures. With the SM-FFN to further strengthen the aggregated features of more informative regions, our model outperforms the existing SOTA methods on various datasets under simulated and real-world noise.

Boaz Arad and Ohad Ben-Shahar.
Sparse recovery of hyperspectral signal from natural rgb images.
In Eur. Conf. Comput. Vis., pages 19–34. Springer, 2016.
Hangbo Bao, Li Dong, and Furu Wei.
Beit: Bert pre-training of image transformers.
arXiv preprint arXiv:2106.08254, 2021.
José M Bioucas-Dias, Antonio Plaza, Gustavo Camps-Valls, Paul Scheunders,
Nasser Nasrabadi, and Jocelyn Chanussot.
Hyperspectral remote sensing data analysis and future challenges.
IEEE Trans. Geosci. Remote Sens., 1(2):6–36, 2013.
George Alan Blackburn.
Hyperspectral remote sensing of plant pigments.
Journal of Experimental Botany, 58(4):855–867, 2007.
Théo Bodrito, Alexandre Zouaoui, Jocelyn Chanussot, and Julien Mairal.
A trainable spectral-spatial sparse coding model for hyperspectral
image restoration.
Adv. Neural Inform. Process. Syst., 34:5430–5442, 2021.
James Bradbury, Stephen Merity, Caiming Xiong, and Richard Socher.
Quasi-recurrent neural networks.
arXiv preprint arXiv:1611.01576, 2016.
Yuanhao Cai, Jing Lin, Xiaowan Hu, Haoqian Wang, Xin Yuan, Yulun Zhang, Radu
Timofte, and Luc Van Gool.
Mask-guided spectral-wise transformer for efficient hyperspectral
image reconstruction.
In IEEE Conf. Comput. Vis. Pattern Recog., pages 17502–17511,
2022.
Stanley H Chan, Xiran Wang, and Omar A Elgendy.
Plug-and-play admm for image restoration: Fixed-point convergence and
applications.
IEEE Trans. Comput. Imaging., 3(1):84–98, 2016.
Yi Chang, Luxin Yan, Houzhang Fang, Sheng Zhong, and Wenshan Liao.
Hsi-denet: Hyperspectral image restoration via convolutional neural
network.
IEEE Trans. Geosci. Remote Sens., 57(2):667–682, 2018.
Yi Chang, Luxin Yan, Xi-Le Zhao, Houzhang Fang, Zhijun Zhang, and Sheng Zhong.
Weighted low-rank tensor recovery for hyperspectral image
restoration.
IEEE Trans. Cybern., 50(11):4558–4572, 2020.
Yi Chang, Luxin Yan, and Sheng Zhong.
Hyper-laplacian regularized unidirectional low-rank tensor recovery
for multispectral image denoising.
In IEEE Conf. Comput. Vis. Pattern Recog., pages 4260–4268,
2017.
Hanting Chen, Yunhe Wang, Tianyu Guo, Chang Xu, Yiping Deng, Zhenhua Liu, Siwei
Ma, Chunjing Xu, Chao Xu, and Wen Gao.
Pre-trained image processing transformer.
In IEEE Conf. Comput. Vis. Pattern Recog., pages 12299–12310,
2021.
Hongyu Chen, Guangyi Yang, and Hongyan Zhang.
Hider: A hyperspectral image denoising transformer with
spatial–spectral constraints for hybrid noise removal.
IEEE Transactions on Neural Networks and Learning Systems,
2022.
Yang Chen, Xiangyong Cao, Qian Zhao, Deyu Meng, and Zongben Xu.
Denoising hyperspectral image with non-iid noise structure.
IEEE Trans. Cybern., 48(3):1054–1066, 2017.
Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio.
Empirical evaluation of gated recurrent neural networks on sequence
modeling.
arXiv preprint arXiv:1412.3555, 2014.
Weisheng Dong, Huan Wang, Fangfang Wu, Guangming Shi, and Xin Li.
Deep spatial–spectral representation learning for hyperspectral
image denoising.
IEEE Trans. Comput. Imaging., 5(4):635–648, 2019.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn,
Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg
Heigold, Sylvain Gelly, et al.
An image is worth 16x16 words: Transformers for image recognition at
scale.
arXiv preprint arXiv:2010.11929, 2020.
Stefan Elfwing, Eiji Uchibe, and Kenji Doya.
Sigmoid-weighted linear units for neural network function
approximation in reinforcement learning.
Neural Networks, 107:3–11, 2018.
James Ellis.
Searching for oil seeps and oil-impacted soil with hyperspectral
imagery.
Earth Observation Magazine, 1:25–28, 2001.
Ying Fu, Zhiyuan Liang, and Shaodi You.
Bidirectional 3d quasi-recurrent neural networkfor hyperspectral
image super-resolution.
IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens., 14:2674–2688,
2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.
Deep residual learning for image recognition.
In IEEE Conf. Comput. Vis. Pattern Recog., pages 770–778,
2016.
Wei He, Quanming Yao, Chao Li, Naoto Yokoya, and Qibin Zhao.
Non-local meets global: An integrated paradigm for hyperspectral
denoising.
In IEEE Conf. Comput. Vis. Pattern Recog., pages 6868–6877,
2019.
Wei He, Quanming Yao, Naoto Yokoya, Tatsumi Uezato, Hongyan Zhang, and Liangpei
Zhang.
Spectrum-aware and transferable architecture search for hyperspectral
image restoration.
In Computer Vision–ECCV 2022: 17th European Conference, Tel
Aviv, Israel, October 23–27, 2022, Proceedings, Part XIX, pages 19–37.
Springer, 2022.
Wei He, Hongyan Zhang, Liangpei Zhang, and Huanfeng Shen.
Total-variation-regularized low-rank matrix factorization for
hyperspectral image restoration.
IEEE Trans. Geosci. Remote Sens., 54(1):178–188, 2015.
Xin He, Yushi Chen, and Zhouhan Lin.
Spatial-spectral transformer for hyperspectral image classification.
Remote Sens., 13(3):498, 2021.
Sepp Hochreiter and Jürgen Schmidhuber.
Long short-term memory.
Neural Computation, 9(8):1735–1780, 1997.
Danfeng Hong, Zhu Han, Jing Yao, Lianru Gao, Bing Zhang, Antonio Plaza, and
Jocelyn Chanussot.
Spectralformer: Rethinking hyperspectral image classification with
transformers.
IEEE Trans. Geosci. Remote Sens., 60:1–15, 2021.
Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang,
Tobias Weyand, Marco Andreetto, and Hartwig Adam.
Mobilenets: Efficient convolutional neural networks for mobile vision
applications.
arXiv preprint arXiv:1704.04861, 2017.
Ryuji Imamura, Tatsuki Itasaka, and Masahiro Okuda.
Zero-shot hyperspectral image denoising with separable image prior.
In Proceedings of the IEEE/CVF International Conference on
Computer Vision Workshops, pages 0–0, 2019.
Sergey Ioffe and Christian Szegedy.
Batch normalization: Accelerating deep network training by reducing
internal covariate shift.
In Proc Int. Conf. Mach Learn., pages 448–456. PMLR, 2015.
Junjun Jiang, He Sun, Xianming Liu, and Jiayi Ma.
Learning spatial-spectral prior for super-resolution of hyperspectral
imagery.
IEEE Trans. Comput. Imaging., 6:1082–1096, 2020.
Linda S Kalman and Edward M Bassett III.
Classification and material identification in an urban environment
using hydice hyperspectral data.
In Imaging Spectrometry III, volume 3118, pages 57–68. SPIE,
1997.
Diederik P Kingma and Jimmy Ba.
Adam: A method for stochastic optimization.
arXiv preprint arXiv:1412.6980, 2014.
Zeqiang Lai, Kaixuan Wei, and Ying Fu.
Deep plug-and-play prior for hyperspectral image restoration.
Neurocomputing, 481:281–293, 2022.
Miaoyu Li, Ying Fu, and Yulun Zhang.
Spatial-spectral transformer for hyperspectral image denoising.
arXiv preprint arXiv:2211.14090, 2022.
Jingyun Liang, Jiezhang Cao, Guolei Sun, Kai Zhang, Luc Van Gool, and Radu
Timofte.
Swinir: Image restoration using swin transformer.
In Int. Conf. Comput. Vis., pages 1833–1844, 2021.
Jing Lin, Yuanhao Cai, Xiaowan Hu, Haoqian Wang, Xin Yuan, Yulun Zhang, Radu
Timofte, and Luc Van Gool.
Coarse-to-fine sparse transformer for hyperspectral image
reconstruction.
arXiv preprint arXiv:2203.04845, 2022.
Yang Liu, Xin Yuan, Jinli Suo, David J. Brady, and Qionghai Dai.
Rank minimization for snapshot compressive imaging.
IEEE Trans. Pattern Anal. Mach. Intell., 41(12):2990 – 3006,
2019.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and
Baining Guo.
Swin transformer: Hierarchical vision transformer using shifted
windows.
In Int. Conf. Comput. Vis., pages 10012–10022, 2021.
Patrick Llull, Xuejun Liao, Xin Yuan, Jianbo Yang, David Kittle, Lawrence
Carin, Guillermo Sapiro, and David J Brady.
Coded aperture compressive temporal imaging.
Optics express, 21(9):10526–10545, 2013.
Matteo Maggioni, Vladimir Katkovnik, Karen Egiazarian, and Alessandro Foi.
Nonlocal transform-domain filter for volumetric data denoising and
reconstruction.
IEEE Trans. Image Process., 22(1):119–133, 2012.
Nishir Mehta, Sushant P Sahu, Shahensha Shaik, Ram Devireddy, and Manas Ranjan
Gartia.
Dark-field hyperspectral imaging for label free detection of
nano-bio-materials.
Wiley Interdisciplinary Reviews: Nanomedicine and
Nanobiotechnology, 13(1):e1661, 2021.
Hisham Othman and Shen-En Qian.
Noise reduction of hyperspectral imagery using hybrid
spatial-spectral derivative-domain wavelet shrinkage.
IEEE Trans. Geosci. Remote Sens., 44(2):397–408, 2006.
Li Pang, Weizhen Gu, and Xiangyong Cao.
Trq3dnet: A 3d quasi-recurrent and transformer based network for
hyperspectral image denoising.
Remote Sens., 14(18):4598, 2022.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory
Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al.
Pytorch: An imperative style, high-performance deep learning library.
Adv. Neural Inform. Process. Syst., 32, 2019.
J. Peng, Q. Xie, Q. Zhao, Y. Wang, L. Yee, and D. Meng.
Enhanced 3dtv regularization and its applications on hsi denoising
and compressed sensing.
IEEE Trans. Image Process., 29:7889–7903, 2020.
Yi Peng, Deyu Meng, Zongben Xu, Chenqiang Gao, Yi Yang, and Biao Zhang.
Decomposable nonlocal tensor dictionary learning for multispectral
image denoising.
In IEEE Conf. Comput. Vis. Pattern Recog., pages 2949–2956,
2014.
Xie Qi, Zhao Qian, Deyu Meng, and Zongben Xu.
Kronecker-basis-representation based tensor sparsity and its
applications to tensor recovery.
IEEE Trans. Pattern Anal. Mach. Intell., PP(99):1–1, 2017.
Yuhao Qing, Wenyi Liu, Liuyan Feng, and Wanjia Gao.
Improved transformer net for hyperspectral image classification.
Remote Sens., 13(11):2216, 2021.
Haiquan Qiu, Yao Wang, and Deyu Meng.
Effective snapshot compressive-spectral imaging via deep denoising
and total variation priors.
In IEEE Conf. Comput. Vis. Pattern Recog., pages 9127–9136,
2021.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox.
U-net: Convolutional networks for biomedical image segmentation.
In Proceedings of the International Conference on Medical Image
Computing and Computer-Assisted Intervention, pages 234–241. Springer,
2015.
Randall B Smith.
Introduction to hyperspectral imaging with tmips.
MicroImages Tutorial Web site, 14, 2006.
Le Sun, Byeungwoo Jeon, Yuhui Zheng, and Zebin Wu.
Hyperspectral image restoration using low-rank representation on
spectral difference image.
IEEE Geosci. Remote. Sens. Lett., 14(7):1151–1155, 2017.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre
Sablayrolles, and Hervé Jégou.
Training data-efficient image transformers & distillation through
attention.
In Proc Int. Conf. Mach Learn., pages 10347–10357. PMLR, 2021.
Muhammad Uzair, Arif Mahmood, and Ajmal Mian.
Hyperspectral face recognition with spatiospectral information fusion
and pls regression.
IEEE Trans. Image Process., 24(3):1127–1137, 2015.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones,
Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.
Attention is all you need.
Adv. Neural Inform. Process. Syst., 30, 2017.
Ashwin A Wagadarikar, Nikos P Pitsianis, Xiaobai Sun, and David J Brady.
Video rate spectral imaging using a coded aperture snapshot spectral
imager.
Optics Express, 17(8):6368–6388, 2009.
Yao Wang, Jiangjun Peng, Qian Zhao, Yee Leung, Xi-Le Zhao, and Deyu Meng.
Hyperspectral image restoration via total variation regularized
low-rank tensor decomposition.
IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens.,
11(4):1227–1243, 2017.
Zhendong Wang, Xiaodong Cun, Jianmin Bao, Wengang Zhou, Jianzhuang Liu, and
Houqiang Li.
Uformer: A general u-shaped transformer for image restoration.
In IEEE Conf. Comput. Vis. Pattern Recog., pages 17683–17693,
2022.
Kaixuan Wei and Ying Fu.
Low-rank bayesian tensor factorization for hyperspectral image
denoising.
Neurocomputing, 331:412–423, 2019.
Kaixuan Wei, Ying Fu, and Hua Huang.
3-d quasi-recurrent neural network for hyperspectral image denoising.
IEEE Trans Neural Netw Learn Syst., 32(1):363–375, 2021.
F. Yasuma, T. Mitsunaga, D. Iso, and S.K. Nayar.
Generalized Assorted Pixel Camera: Post-Capture Control
of Resolution, Dynamic Range and Spectrum.
Technical report, Nov 2008.
Dabing Yu, Qingwu Li, Xiaolin Wang, Zhiliang Zhang, Yixi Qian, and Chang Xu.
Dstrans: Dual-stream transformer for hyperspectral image restoration.
In Proceedings of the IEEE/CVF Winter Conference on Applications
of Computer Vision, pages 3739–3749, 2023.
Weihao Yu, Mi Luo, Pan Zhou, Chenyang Si, Yichen Zhou, Xinchao Wang, Jiashi
Feng, and Shuicheng Yan.
Metaformer is actually what you need for vision.
In IEEE Conf. Comput. Vis. Pattern Recog., pages 10819–10829,
2022.
Qiangqiang Yuan, Liangpei Zhang, and Huanfeng Shen.
Hyperspectral image denoising employing a spectral–spatial adaptive
total variation model.
IEEE Trans. Geosci. Remote Sens., 50(10):3660–3677, 2012.
Qiangqiang Yuan, Qiang Zhang, Jie Li, Huanfeng Shen, and Liangpei Zhang.
Hyperspectral image denoising employing a spatial–spectral deep
residual convolutional neural network.
IEEE Trans. Geosci. Remote Sens., 57(2):1205–1218, 2018.
Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan,
and Ming-Hsuan Yang.
Restormer: Efficient transformer for high-resolution image
restoration.
In IEEE Conf. Comput. Vis. Pattern Recog., pages 5728–5739,
2022.
Kai Zhang, Yawei Li, Wangmeng Zuo, Lei Zhang, Luc Van Gool, and Radu Timofte.
Plug-and-play image restoration with deep denoiser prior.
IEEE Trans. Pattern Anal. Mach. Intell., 44:6360–6376, 2021.
Tao Zhang, Ying Fu, and Cheng Li.
Hyperspectral image denoising with realistic data.
In Int. Conf. Comput. Vis., pages 2248–2257, 2021.
XiLe Zhao, Hao Zhang, TaiXiang Jiang, Michael K Ng, and Xiong-Jun Zhang.
Fast algorithm with theoretical guarantees for constrained
low-tubal-rank tensor recovery in hyperspectral images denoising.
Neurocomputing, 413:397–409, 2020.

More Details about GSSA
Computational Complexity of GSSA.
Given an input {{formula:a19158b0-88c7-41cf-ba08-ccf0914a1974}}  where {{formula:81d4f642-4293-4f20-9928-653226694ec3}}  denote height and width, {{formula:e19b529e-fc32-46c2-821c-745a79335e51}}  denotes the number of spectral bands, {{formula:81730cfc-048a-4865-a8fa-7c7441790047}}  denotes the features channels, the computational complexity of each step of GSSA is summarized in Tab. {{table:820042de-4cae-4f56-a270-a403eb58263c}} . Since the feature channels are typically larger than the number of spectral bands, the asymptotic computational complexity of GSSA is dominated by two linear transformations, , Linear for {{formula:471834ed-30de-4c2a-9fa7-2e14a5529f80}} and Post linear.
The computational complexity of GSSA. The overall complexity of GSSA is linear with respect to image size.
Step
Complexity
Linear for {{formula:6060e667-1c71-43d6-8b97-9b87aa2832cd}}
{{formula:3e648b86-92fa-4188-8c0c-ae0fea558fc0}}
Pooling for {{formula:65ecb937-1848-4534-b391-0144cf6bb24b}} ,{{formula:92dbe036-a01e-424d-8fce-f28a422c6d54}}
{{formula:73fdbcb8-1f91-48a1-a9bc-555538dfe4f3}}
Compute attention matrix
{{formula:bf5e7a56-2f6d-4a49-8620-a32aff3d6cba}}
Feature aggregation
{{formula:859b02af-33c8-4542-967d-5c57f10a942d}}
Post linear
{{formula:87eb448c-57c5-4e91-b8b5-4585104bb8d2}}
Total
{{formula:6a530506-0bba-4988-8db0-ed0b0ff42566}}
{{table:05cb4455-f8f5-40fe-b670-1afc06bf76d1}}{{table:820042de-4cae-4f56-a270-a403eb58263c}}

Fast Implementation.
With the simplification of pixel-wise attention via global average pooling, our GSSA can be efficiently implemented with a depth-wise convolution by treating the shared attention map as a convolution filter and swapping the spectral and channel dimensions. The speed comparison is shown in Tab. {{table:16f8ac57-feef-4591-a4ff-2516d3990c9b}} , and it can be seen that the Conv-based implementation is approximately 20% faster than the naive Matmul-based one.
Speed of different implementations of GSSA. Our Conv-based implementation reduces the running time without harming the performance.
Implementation
 Runtime (s)
PSNR
Matmul-based
0.60
41.82
Conv-based
0.47
41.82
{{table:77bf2105-ec94-4ccf-a587-39da738aacb8}}{{table:16f8ac57-feef-4591-a4ff-2516d3990c9b}}Comparison of different spectral/channel attention mechanisms. Our GSSA is significantly different from previous attention mechanisms. We could observe MDTA and MGSA are almost identical; MS-MSA and GSA are almost identical. Besides, MS-MSA and GSA are also basically simpler version of MDTA without depthwise convolution. Please refer to the text for detailed explanation.
{{figure:7b82e88e-6215-406e-9f0b-f640955cd81a}}

Comparison against other Attention.
Here, we provide a more detailed explanation regarding the differences between our GSSA and existing channel or spectral attention mechanisms.
We highlight that our GSSA is significantly different from previous attention mechanisms in a variety aspects. Since GSSA performs attention along spectral rather than spatial dimensions, we here compare it with four previous attention mechanisms that apply along spectral or channel dimensions including:
The competing attention mechanisms.
Attention
Method
Task
MDTA
Restormer {{cite:e4b9502}}
Color image restoration
MS-MSA
MST {{cite:3050a71}}
Spectral Reconstruction
GSA
SST {{cite:6878378}}
HSI denoising
MGSA
Hider {{cite:20d3161}}
HSI denoising
{{table:704a14f4-8d72-4ba8-8e7b-83455a247c78}}{{table:f78ba592-57d2-4a9b-96d3-b48036a03bdc}}Fig. {{figure:7b82e88e-6215-406e-9f0b-f640955cd81a}}  illustrates the structures of the aforementioned attention mechanisms. It is worth noting that all previous methods are essentially variants of MDTA proposed in Restormer, whereas our GSSA is fundamentally distinct from them. In the following, we will provide a detailed explanation of the main differences between the previous methods and our GSSA.
3D vs 2D Data Format. The first notable difference, which can be easily confused with previous work, is that our GSSA performs attention on the spectral dimension, i.e., the {{formula:64c05ba5-cc71-45ee-a41f-cf54fa46ba64}}  dimension of a 5D data cube {{formula:bdae0a51-475c-490a-b2fc-319e56f19182}} . In contrast, previous works, such as MST, and SST, even though they refer to their attention mechanisms as spectral attention, essentially apply channel attention along the {{formula:196c8522-357f-42d7-8295-b75169b0f890}}  dimension of a 4D data cube {{formula:8574ba57-9f34-4caa-9f28-ee21e21711b5}} , which is the same as MDTA.
Our 3D approach provides the flexibility to handle HSIs with different bands within a single model. Additionally, it achieves superior performance by preserving the structures of different bands, , each band possesses its own feature set, and their relationship remains unchanged across layers of the entire model.
QKV Projection. The second key difference pertains to the projections used for the query, key, and value. Conventional attention mechanisms typically employ three linear projections to project the input into query, key, and value. This approach is utilized in all of the compared methods, with the exception of our GSSA. Instead, our GSSA applies linear projection solely for the value, which greatly simplifies the design. By contrast, MDTA needs a extra 3x3 depthwise convolution after the linear projection. MGSA is identical to MDTA, except that it employs a 3D convolution. MS-MSA and GSA are the same and solely utilize linear projections, with the exception that MS-MSA employs an additional mask attention specifically designed for spectral reconstruction.
Pooling vs Reshape. The third difference is that our GSSA uses global average pooling to obtain feature maps for each band. This differs from previous methods that adopt a reshape approach. Our method is significantly more computationally efficient compared to previous approaches. Previous methods reshape the Q, K, and V tensors from a shape of {{formula:8bba43e4-5c90-40f7-a3dd-3c348b4a2f38}}  into {{formula:de6891f9-fc16-4dc2-9239-d0370b1bcc38}} , treating {{formula:eee7485b-ac0f-465b-9754-d0bb32781636}}  as the features for each channel. This leads to a time complexity of dot-product attention that is linear with respect to the image size, , {{formula:1a93b9ef-2aad-463e-903b-bf667e94ac83}} . In contrast, our GSSA approach only has a constant time complexity {{formula:2befc053-8c83-472f-9fc2-76e85bd59099}} , where {{formula:3317f338-12c3-4496-90cc-d11c4af3b63d}}  denotes the number of bands.
Learnable Query. The fourth notable difference is the introduction of the learnable query (LQ), which is motivated by the fixed patterns of pixel values across different bands. For example, the values of band 100 nm and 200 nm are correlated. Our LQ helps to identify these correlations and the alternative training strategy enables improvements without any extra cost on the number of parameters, inference time, and the flexibility to handle HSIs with different bands.

More Ablation Studies
To evaluate the effectiveness of the proposed components, we conduct a series of experiments to explore the different design choices for each part of our HSDT architecture.
Specifically, we compare the proposed blocks, which include GSSA, S3Conv, and SM-FNN, by separately replacing them with existing blocks that share the same functionality, , replacing S3Conv with Conv3D. We use HSDT-M as the base model and evaluate the performance of the different blocks by replacing them one at a time. For blocks that cannot be incorporated into our 3D architectural design of HSDT, such as 2D spectral attention {{cite:6878378}}, we report the results obtained using their respective models.
Comparison of different S3Conv variants against 3D convolution and previous separable convolution. Our S3Conv achieves significant better performance with fewer parameters. Our methods are highlighted as gray. #P denotes the model parameters.
Model
#P(Conv)
#P(Total)
PSNR
SAM
Conv3D
0.43M
0.58M
41.62
0.052
Sep3D {{cite:806fd04}}
0.37M
0.53M
41.44
0.054
graycolor S3Conv-S
0.26M
0.42M
41.47
0.052
graycolor S3Conv-Seq
0.26M
0.42M
41.58
0.052
graycolor S3Conv
0.36M
0.52M
41.82
0.049
{{table:a46f25f1-8a59-48e7-8b09-36a3931aca8e}}{{table:308df0e4-ab0e-46cd-80e9-e5f77ddda98c}}
Spatial-Spectral Separable Convolution.
We evaluate several variants of our S3Conv. The most straightforward variant, S3Conv-S, sets the number of spatial convolutions to 1, while the S3Conv variant that we adopt uses 2. Another variant, S3Conv-Seq, applies spatial and spectral convolutions sequentially instead of in parallel. As shown in Table {{table:308df0e4-ab0e-46cd-80e9-e5f77ddda98c}} , both variants achieve comparable performance with roughly 60% of the parameters used by Conv3D. Our adopted version achieves a 0.2 dB PSNR gain with only 80% of the parameters used by Conv3D. Notably, our S3Conv approach significantly outperforms previous HSI separable convolution approaches {{cite:806fd04}}, achieving over 0.4 dB PSNR improvement with even fewer parameters.

Guided Spectral Self-Attention.
We compare the proposed GSSA approach with existing spectral fusion techniques, including QRU {{cite:e2c0164}}, GSA {{cite:6878378}}, MS-MSA {{cite:3050a71}}, MDTA {{cite:e4b9502}}, and MGSA {{cite:20d3161}}. It is worth noting that although GSA and MS-MSA are named as spectral attention, they are essentially channel attentions derived from MDTA, as discussed earlier. Furthermore, GSA, MS-MSA, and MDTA are all 2D attention approaches that work with 4D data formats instead of the 5D data format used by HSDT. Therefore, we report the results of their models when compared with GSA, MS-MSA, and MDTA. For 3D spectral fusion techniques such as QRU and MGSA, we report the results of models that replace the GSSA of HSDT-M with them. Table {{table:11075718-4ace-40f2-ade9-3dac8fa3a141}}  presents the results of different attention mechanisms. Our GSSA approach achieves the best results against the other approaches. Notably, our GSSA outperforms previous GSA and MGSA approaches (which are also designed for HSI denoising) by a large margin, demonstrating the effectiveness of our designs.
Results of our GSSA in comparison with other attention blocks. Our GSSA achieves a prominent improvement against QRU by over 0.5 PSNR improvement, while previous HSI denoising transformer with GSA only outperforms QRU by only 0.1 PSNR.
Model
Params
PSNR
SAM
QRU {{cite:e2c0164}}
0.57M
41.31
0.064
GSA {{cite:6878378}} & MS-MSA {{cite:3050a71}}
4.14M
41.41
0.052
MDTA {{cite:e4b9502}}
26.2M
41.03
0.062
MGSA {{cite:20d3161}}
0.50M
39.74
0.102
graycolor GSSA
0.52M
41.82
0.049
{{table:36d14201-b88e-41e4-b46d-fec4fb4bf070}}{{table:11075718-4ace-40f2-ade9-3dac8fa3a141}}

Self-Modulated Feed-Forward Network.
The proposed SM-Branch can be used without additional conventional FFN. As shown in Tab. {{table:0a34e672-91a3-47ef-8cec-3996628facab}} , the sole use of SM-Branch also outperforms the conventional FFN, and the combination of them both yields the best results with very few extra parameters. The GDFN {{cite:e4b9502}} developed for RGB restoration performs poorly and might be unsuitable for our model.
Comparison of the existing FFN with our SM-FFN and SM-Branch.
Model
Params
PSNR
SAM
FFN
0.49M
41.67
0.050
GDFN {{cite:e4b9502}}
0.49M
37.38
0.094
graycolor SM-Branch
0.45M
41.74
0.051
graycolor SM-FFN
0.52M
41.82
0.049
{{table:136827d5-cc2c-4e2a-bd69-51d000564e1a}}{{table:0a34e672-91a3-47ef-8cec-3996628facab}}
More Discussions
Visualization of S3Conv.
To demonstrate the effectiveness of our S3Conv. We provide a comparison of the features map between S3Conv and conventional 3D convolution. As shown in Fig. {{figure:ec315255-33b1-48f2-9baa-c2dfcf5fc689}} , our S3Conv extracts more spatial meaningful features.
Comparison of the feature maps extracted by conventional 3D convolution and our S3Conv.
{{figure:ec315255-33b1-48f2-9baa-c2dfcf5fc689}}

Analysis of SM-FFN.
The proposed SM-FFN is designed for strengthening the features with higher activation via a self-modulation operation. The improvement provided by SM-FFN could be intuitively explained by the emphasis on more informative regions that typically have higher activation. In the following, we provide some possible relations between our SM-FFN and the SiLU {{cite:8f3299c}} activation, which might further imply why our SM-FFN works better. Specifically, The SiLU activation is,
{{formula:73df0b69-0306-4f44-b04f-e6641452cfda}}
where {{formula:1b83475e-0472-4782-a4fc-cd05b6debd1a}}  and {{formula:2ec4e5e6-a620-4b07-baef-9f058b2bd182}}  are the input and output feature maps. It can be observed that SiLU could be treated as a kind of self-modulation where the modulation weight is computed from the input itself. However, such homogeneous self-modulation might be limited in expressive abilities. Instead, our SM-FFN employs a heterogeneous self-modulation,
{{formula:ad22ba40-dab3-40ae-ad6a-0ae41bef1c76}}
where we adopt two extra linear projections to project input x into two different spaces. This removes the restriction of SiLU where the input {{formula:cc0b810d-d33d-4016-b7fe-7ccc731307d6}}  should simultaneously play two roles of features and modulation weight. Thus, our SM-FFN can obtain the advantages of SiLU, , training stability and implicit regularization while maintaining more representation capability. Consequently, it leads to better performance than conventional FFN.
Results on the task of compressive sensing.
graycolor Method
PSNR
SSIM
2DTV
25.26
0.863
3DTV
28.46
0.910
DeSCI {{cite:4cf1df1}}
26.62
0.912
SCI-TV-FFDNet {{cite:375ae9d}}
29.35
0.925
DPHSIR {{cite:1cebc3f}}
30.56
0.945
PnP-HSDT (ours)
31.64
0.948
{{table:d64a1b3d-8349-4f7b-946d-c0d071ecf6f5}}graycolor
2x
4x
graycolor Method
PSNR
SSIM
PSNR
SSIM
Bicubic
35.13
0.9575
35.12
0.954
SSPSR {{cite:3c214aa}}
47.55
0.995
39.19
0.979
Bi-3DQRNN {{cite:982ef94}}
42.53
0.989
39.56
0.979
DPHSIR {{cite:1cebc3f}}
48.75
0.996
40.95
0.980
PnP-HSDT (ours)
49.76
0.996
41.56
0.982
{{table:447c9e20-b6db-47c7-906b-eee9495cfef9}}[h]0.4

[h]0.55

Results on the task of super-resolution.

Experimental results of our PnP extension on the task of compressive sensing and super-resolution.{{table:170d861f-f23e-4979-809e-dbbbd9ff72a2}}
Extension as Plug-and-Play Prior
Considering the superior performance of our method on the Gaussian denoising task,
we demonstrate that HSDT can be used a plug-and-play (PnP) prior {{cite:47113e3}} to solve general HSI restoration tasks with proximal algorithms, , ADMM and HQS.

Experimental Setup.
We adopt PnP-ADMM {{cite:1cebc3f}} to extend our method to the tasks of compressive sensing, and super-resolution. To meet the requirements of PnP algorithms, , Gaussian denoiser for continuous noise strengths, we retrain our model, , HSDT-M, with an additional noise level map {{cite:f7ac43e}} on simulated Gaussian noise ranged from 0 to 70. We run 40 iterations for compressive sensing and 24 iterations for super-resolution. The hyperparameters of the algorithms are manually tuned to achieve the best performance.

Compressive Sensing.
We conduct the simulated experiments on CASSI {{cite:b2ab249}} system. Following {{cite:375ae9d}}, the shifting random binary mask {{cite:61e1aff}} is used in our simulation. We provide the results on CAVE Toy, which is obtained from {{cite:4cf1df1}}. We compare several recent methods, including DPHSIR {{cite:1cebc3f}}, SCI-TV-FFDNet {{cite:375ae9d}}, DeSCI {{cite:4cf1df1}}, and traditional methods, , 2DTV and 3DTV. The quantitative results are shown in Tab. linkcolor1tab:cs. It can be seen that our method obtains the best performance with over 1 dB improvement on PSNR. Specifically, the improvement is purely obtained through the superior denoising ability of our model, which means our model can also be integrated into other more advanced PnP methods for further improvement, , {{cite:375ae9d}}.

Super-Resolution.
We also provide results on the task of HSI super-resolution. Following {{cite:1cebc3f}}, we first blur the high-resolution HSI via an {{formula:1d2c1506-ee1e-456a-ab06-003e2f6ecb4f}}  Gaussian blur kernel with {{formula:475efa72-acaf-4e11-b21b-0054039dd816}} , and then downsample the image to obtain the low-resolution HSI. We provide the results on ICVL with a scale factor of 2 and 4. The competing methods include several recently developed methods, , SSPSR {{cite:3c214aa}}, Bi3DQRNN {{cite:982ef94}}, and DPHSIR {{cite:1cebc3f}} . As shown in Tab. linkcolor1tab:sr, our method achieves the best performance. In particular, our method only needs the pretrained Gaussian denoising model, which is the same as {{cite:1cebc3f}}. The improvement against {{cite:1cebc3f}} comes from the better PnP denoising prior, which further demonstrates the stronger denoising ability of our method.

More Implementation Details
Our multi-step learning rate scheduler.
graycolor Stage 1
Gaussian Noise {{formula:83f16c69-d1b2-496a-94c9-f57bbee42782}}
Epoch
0 - 20
20 - 30

LR
{{formula:60b6ca4e-6656-4742-9968-c0364d830f99}}
{{formula:efe771e2-108b-461b-9f20-03e6b6cb77a8}}

graycolorStage 2
Gaussian Noise {{formula:90e7f979-e9d9-4bcc-86c0-07e1df544bf1}}
Epoch
30 - 45
45 - 55
55 - 60
60 - 65
65 - 75
75 - 80
LR
{{formula:113b52a8-f46b-45df-9797-0ca7adec0643}}
{{formula:4207eb8d-b4ae-4530-aca9-07f1d2d5e38b}}
{{formula:70342053-253b-4e54-bd99-dbd7669c2d8b}}
{{formula:f34af899-10ed-4ff0-8ba5-f86b968c45b9}}
{{formula:d8f917aa-802a-4aac-b986-87e2f9cd41f0}}
{{formula:c6e0bafa-2d1d-48d9-a2a4-20bde4c77591}}
graycolorStage 3
Complex Noise
Epoch
80 - 90
90 - 95
95 - 100
100 - 105
105 - 110

LR
{{formula:15a87b8d-b28b-484f-821d-5fdb87b6738e}}
{{formula:ff371e81-37f0-414a-90bf-d17efd6201f8}}
{{formula:7de847b5-d85b-4a10-840f-088b1058b59e}}
{{formula:549ebe06-c650-4a3b-9cdf-991c88a50301}}
{{formula:097ac8dc-1d52-4c0c-b253-01aa8fd1974e}}

{{table:e3e3a63b-c264-41c8-ae97-678dbaccd2a2}}graycolorSystem
Ubuntu 20.04.1 LTS
graycolorGPU
Nvidia GeForce RTX 3090
graycolorCPU
Intel(R) Core(TM) i9-10850K CPU
graycolorFramework
PyTorch 1.7.1
graycolorDriver
Cuda 11.2
graycolorSoftware
Matlab 2020
graycolorDataset
ICVL
graycolorImage Size
{{formula:c3b36c1f-5c80-40e6-a31b-e3377716533b}}
graycolorRepeat times
10
{{table:c2fe1af9-64fa-4e2b-9fac-c5d7d1fc8e10}}[h]0.56

[h]0.4

System configuration for the speed test.

More implementation details. (a) We adopt a multi-stage training strategy with the learning warmup setup for the first epoch. (b) We provide the system configuration as the results of the speed test are strongly correlated with the configuration.{{table:7694bbd9-368f-46ff-ab33-81b07980b635}}Setup of the Learning Rate.
In this part, we provide more details about the multi-step learning rate scheduler that we used for training our simulated Gaussian and complex denoising models. Specifically, we use a multi-stage training strategy to train the models for Gaussian noise and complex noise. The learning rate is set up as shown in Tab. linkcolor3tab:lr. We use learning rate warmup to gradually increase the learning rate from 0 to {{formula:9f5a4e6b-ced9-48b8-aed6-890470392f82}}  for the first epoch of the second stage.

Details of the Simulated Complex Noise.
We follow {{cite:e2c0164}} for constructing simulated complex noise. In details, we consider the non-independent and non-identically distributed (non-i.i.d) Gaussian noise, stripe noise, deadline noise, impulse noise, and the combination of the aforementioned noise (denoted as mixture noise). The details about these five cases of noise are listed as follows,
[noitemsep, leftmargin=*]
Non-i.i.d noise. The non-independent and non-identically distributed Gaussian is added to every pixel of each HSI. The noise strength is randomly selected from 10, 30, 50, and 70.

Stripe noise. Stripe noise (5% to 15% percentages of columns) is added to randomly selected one-third of bands. Non-i.i.d. Gaussian noise is added to All bands.

Deadline noise. Deadline noise is added to randomly selected one-third of bands. Non-i.i.d. Gaussian noise is added to All bands.

Impulse noise. Impulse noise with intensity ranging from 10% to 70% is added to randomly selected one-third of bands. Non-i.i.d. Gaussian noise is added to All bands.

Mixture noise. Each band is randomly corrupted by at least one kind of noise mentioned above.

System Configuration.
In the main paper, we compare the running time of different methods. All the comparisons are performed with an Nvidia GeForce RTX 3090, and an Intel(R) Core(TM) i9-10850K CPU @ 3.60GHz on Ubuntu 20.04.1 LTS. All the CNN-based methods are implemented and tested with PyTorch 1.7.1. All the optimization-based methods are implemented and tested with Matlab. We test the running time on ICVL with an image size of {{formula:f0633a79-06e9-4211-917e-7efc74ea1feb}}  by repeating the test 10 times and averaging the results.

More Visual Results
Simulated Gaussian Noise.
In the main paper, we provide a visual comparison of our method against the competing ones under 50 noise strength. Here, we provide more visual results with other noise strengths used in our experiments. The results are shown in fig:gaussian30,fig:gaussian50,fig:gaussian70,fig:gaussianblind.

Simulated Complex Noise.
For complex noise, we provide the visual results under stripe noise in the main paper. Here, we provide additional results on other types of complex noise, , non-i.i.d, deadline, impulse, and mixture. The results are shown in fig:complexnoniid,fig:complexstripe,fig:complexdeadline,fig:compleximpulse,fig:complexmixture.

Real-world Noise.
More visual results on the RealHSI dataset are provided in Fig. {{figure:f7cc2824-f65e-4901-8f39-4565cee24f0e}} .

Future work.
In this work, we propose a transformer architecture, , HSDT for hyperspectral image denoising. We introduce several effective and generalizable components to better explore the spatial-spectral and global spectral correlations of HSI. Specifically, it is worthwhile to explore the applications of the proposed S3Conv and HSDT for more network architectures and tasks. Furthermore, our learnable queries could also be extended to condition on some external information for more explicit guidance. For example, we might be able to inject the Gaussian noise strength into the network with learnable queries, through an embedding layer. This is helpful for a PnP Gaussian denoiser, where the noise strength is known.

Broader Impacts
Our work has no ethical issues or broader impacts.
HSIDCNN{{cite:cb9021e}}
0.1380.1380.1380.1380.1380.1380.138(a) b(b) b(c) b(d) b(e) b(f) b(g) b{{table:0cf775df-97fd-4121-aa68-35422b283731}}{{figure:483eed14-8366-498f-8ce8-0dfada1e0450}}{{figure:561fbc33-e6cc-4e3b-b249-291a1b8c1661}}QRNN3D{{cite:e2c0164}}{{figure:567ca86b-f894-43dd-9455-50035901f1b6}}T3SC{{cite:27e8eef}}{{figure:5b027539-3f21-45ff-a8a4-84601a0bbe29}}GRUNet{{cite:1cebc3f}}{{figure:9f5682ca-ef87-4c5b-a728-ed358a908bef}}Restormer{{cite:e4b9502}}{{figure:c5177f87-5704-4aee-bb45-483979cc1de4}}TRQ3D{{cite:0d1acf1}}{{figure:97a3c726-77aa-4c20-a42f-e486d133ca3a}}HSDT-L(ours)Simulated Gaussian noise removal results at 20th band of image under noise level {{formula:25ba2a70-c708-47ae-888a-2c198eb61b16}}  on ICVL dataset.{{figure:b8e5d6e3-3c32-4c72-92e8-f4006eb9c8d4}}HSIDCNN{{cite:cb9021e}}
0.1380.1380.1380.1380.1380.1380.138(a) b(b) b(c) b(d) b(e) b(f) b(g) b{{table:5e4e378f-2537-48aa-a8af-ffd299cde39e}}{{figure:5f32eb8b-f186-4512-8e42-86dc50502002}}{{figure:06e75020-6e48-41bf-8f62-f6f0336cfbd0}}QRNN3D{{cite:e2c0164}}{{figure:24176bee-7a01-425f-bb32-7b4ddec7f6ae}}T3SC{{cite:27e8eef}}{{figure:6da560ed-e2d8-4778-9c1b-a6994a4004ef}}GRUNet{{cite:1cebc3f}}{{figure:b0ed9cdc-e4f3-4a0f-b23d-d5cecf22caa7}}Restormer{{cite:e4b9502}}{{figure:8bd744e9-b90c-4672-bb55-f0d2dc1db64a}}TRQ3D{{cite:0d1acf1}}{{figure:0238a188-76c1-42b6-a7dc-d6cd29aba198}}HSDT-L(ours)Simulated Gaussian noise removal results at 20th band of image under noise level {{formula:e6f4a44f-b4db-4b1b-90b8-ff5489b33e41}}  on ICVL dataset.{{figure:937bac33-650b-458e-89c9-d0251759e2ed}}HSIDCNN{{cite:cb9021e}}
0.1380.1380.1380.1380.1380.1380.138(a) b(b) b(c) b(d) b(e) b(f) b(g) b{{table:3a575ba3-951f-463d-8e20-0d7d4f914d47}}{{figure:da9b9446-1d49-4862-bf4d-71f7646c2c25}}{{figure:d3c38172-fb9c-4713-a3e2-d57e62c27f38}}QRNN3D{{cite:e2c0164}}{{figure:967c3f70-002b-4b24-a6f5-b028045f61a9}}T3SC{{cite:27e8eef}}{{figure:21a43c50-2afd-4711-8f99-8182a7d5dbdc}}GRUNet{{cite:1cebc3f}}{{figure:eb1f0437-7aa1-4277-9a33-46173a742c6c}}Restormer{{cite:e4b9502}}{{figure:33565669-3fd7-468f-be15-28df4cbfcdb9}}TRQ3D{{cite:0d1acf1}}{{figure:f6176dd6-d078-426a-856d-fb08aba03162}}HSDT-L(ours)Simulated Gaussian noise removal results at 20th band of image under noise level {{formula:afa67166-e93c-4efe-9f83-ba382c1074c0}}  on ICVL dataset.{{figure:eeb8d181-9027-4ca2-ba90-181ae93e5516}}HSIDCNN{{cite:cb9021e}}
0.1380.1380.1380.1380.1380.1380.138(a) b(b) b(c) b(d) b(e) b(f) b(g) b{{table:4cfe1d40-d1a9-4b6d-8cca-77e80823a64c}}{{figure:80dfd2cb-31af-40ad-95ba-19618a6123da}}{{figure:beb73265-664d-497a-a5cb-da73d92210eb}}QRNN3D{{cite:e2c0164}}{{figure:6ead4807-1df1-4aea-883d-429681e9f060}}T3SC{{cite:27e8eef}}{{figure:bea8974a-83e6-4402-bba5-756d559ffb4e}}GRUNet{{cite:1cebc3f}}{{figure:1952f538-39a3-4271-9423-dd88a2ac298c}}Restormer{{cite:e4b9502}}{{figure:085ff012-b0e4-489c-8012-65c24350b0b2}}TRQ3D{{cite:0d1acf1}}{{figure:5b7ccae4-1c94-424b-bd39-88dcb3e4fc67}}HSDT-L(ours)Simulated Gaussian noise removal results at 20th band of image under noise level {{formula:af50ddab-4462-48b7-9633-498d226ca398}}  on ICVL dataset.{{figure:0c1c06b0-d422-4f20-b70a-ee811fdc98f9}}HSID-CNN {{cite:cb9021e}}
0.150.150.150.150.150.15(a) b(b) b(c) b(d) b(e) b(f) b{{table:13e567b4-30bc-426f-8715-8d357ecd3487}}{{figure:d3240b64-ec67-4e41-93da-e70aa5a72f17}}{{figure:3bc03ae1-5ee3-4b60-ad58-ca32ddeb80f9}}QRNN3D {{cite:e2c0164}}{{figure:0ed32efd-98d0-4f08-bbda-cd29ba48c986}}GRUNet {{cite:1cebc3f}}{{figure:9485c642-de18-4d66-bfbc-ca2676e5eee2}}Restormer {{cite:e4b9502}}{{figure:7a0c6a92-0b4b-4147-8c70-232c065f624d}}TRQ3D {{cite:0d1acf1}}{{figure:2df6da17-18b3-4c67-a623-40c3a7251183}}HSDT-L(ours)Simulated complex noise removal results at 20th band of an image on ICVL dataset. (noniid case){{figure:8ab74684-3830-4b8b-8462-dcd2bf51cb74}}HSID-CNN {{cite:cb9021e}}
0.150.150.150.150.150.15(a) b(b) b(c) b(d) b(e) b(f) b{{table:8d390a84-6fa0-49a0-9fb2-fc3186e05255}}{{figure:a27525fb-c81f-4da4-903b-fdb298bfb1e7}}{{figure:e8be6bde-b7b7-4ac1-94cd-e29528d8fada}}QRNN3D {{cite:e2c0164}}{{figure:32f16799-1892-441b-9675-54f53fbb1b2a}}GRUNet {{cite:1cebc3f}}{{figure:bf274053-d226-4929-9b7a-2fc702e337b4}}Restormer {{cite:e4b9502}}{{figure:e72d5e90-0055-49b6-b549-1ed41c05738a}}TRQ3D {{cite:0d1acf1}}{{figure:346dad78-b7e3-4eb1-9fd0-93ad5409c9ad}}HSDT-L(ours)Simulated complex noise removal results at 20th band of an image on ICVL dataset. (stripe case){{figure:fee14217-8c48-4176-82e1-8ac31dc8db2e}}HSID-CNN {{cite:cb9021e}}
0.150.150.150.150.150.15(a) b(b) b(c) b(d) b(e) b(f) b{{table:a6876edc-4c67-4ae3-b594-8b63aa19d001}}{{figure:1ca7a11b-6345-42c2-a5d0-8347451aa111}}{{figure:c019d36c-996d-4a5e-915a-a221e6f7cd46}}QRNN3D {{cite:e2c0164}}{{figure:fa48787e-9175-4a11-941e-2e8456d9e71b}}GRUNet {{cite:1cebc3f}}{{figure:db4dc394-1600-4f42-884a-121b182931ce}}Restormer {{cite:e4b9502}}{{figure:73bba054-dd9a-4509-9e40-00130405a390}}TRQ3D {{cite:0d1acf1}}{{figure:89246eba-9c31-4002-a42b-510fd7f1d50f}}HSDT-L(ours)Simulated complex noise removal results at 20th band of an image on ICVL dataset. (deadline case){{figure:7cbafbc4-b0f9-4da4-8001-967cca7eb327}}HSID-CNN {{cite:cb9021e}}
0.150.150.150.150.150.15(a) b(b) b(c) b(d) b(e) b(f) b{{table:f1b5e653-b615-4c32-a8eb-2931fba76f03}}{{figure:946008da-5e64-4ece-810b-184c0e37c29e}}{{figure:8f2a2cc9-fee6-4bd1-81a8-0716fc8f82ec}}QRNN3D {{cite:e2c0164}}{{figure:2e513bb2-e0f1-4beb-8aa4-f9067017a67f}}GRUNet {{cite:1cebc3f}}{{figure:0751bf84-9169-4501-9789-df2fe7bc2089}}Restormer {{cite:e4b9502}}{{figure:3f4807c6-fa7f-4f82-84d0-4c1edcbebb96}}TRQ3D {{cite:0d1acf1}}{{figure:b888b48f-a5b9-4324-9b3a-7ffb897d4a92}}HSDT-L(ours)Simulated complex noise removal results at 20th band of an image on ICVL dataset. (impulse case){{figure:8d5f57ac-9693-4813-ab3a-2ddc843b5681}}HSID-CNN {{cite:cb9021e}}
0.150.150.150.150.150.15(a) b(b) b(c) b(d) b(e) b(f) b{{table:93d09221-f4c3-4044-9860-8539fba35e0e}}{{figure:d280e684-8b7a-4d29-910d-a8d2af1a739a}}{{figure:7e23b446-3fed-421e-86d2-674a3a532391}}QRNN3D {{cite:e2c0164}}{{figure:26b6fbe7-bb1a-4e4e-b102-6e0cf54271a3}}GRUNet {{cite:1cebc3f}}{{figure:bd1445f0-41af-4646-91e8-c1c3cad716d8}}Restormer {{cite:e4b9502}}{{figure:84879102-0971-46b0-a69f-ee9aa5ea2f6b}}TRQ3D {{cite:0d1acf1}}{{figure:4a5ecae0-f8a3-4c9e-bcae-d96a69db75db}}HSDT-L(ours)Simulated complex noise removal results at 20th band of an image on ICVL dataset. (mixture case){{figure:7115fbaf-213b-47cb-a51a-43478a016ab9}}Noisy
0.150.150.150.150.150.15(a) b(b) b(c) b(d) b(e) b(f) b{{table:a45492d3-7d65-44a8-9180-040f4a86ada1}}{{figure:cea5649f-ab2b-4692-b76c-af057c762736}}{{figure:287e8432-e962-4a75-b3ec-0d6614dc65a7}}NMoG {{cite:21c843b}}{{figure:331daf86-bff2-4165-8c59-5ad359e697cd}}QRNN3D {{cite:e2c0164}}{{figure:4958a6d8-fe70-4a25-9eb7-e09bd62922e4}}GRUNet {{cite:1cebc3f}}{{figure:2c38ec64-cc8c-469e-a326-f0e1cc19ff3d}}Restormer {{cite:e4b9502}}{{figure:5aadd928-ddad-49ab-bfc5-ccacec656410}}HSDT(ours)Visual real noise removal results at 20th band on RealHSI dataset.{{figure:e1150fb3-4520-48db-b4ef-348c371bc3d8}}