referential equality, we cannot infer the exact question

Please read carefully to get accepted!
(1) You're not required to answer the question. The answer is already provided and marked in red. Read examples below carefully to learn about what we want!
(2) Identify important short phrases that appear both in the question and in the context.
Important: The two appearances of the phrase should be exactly the same (trivial differences like plural form or past tense are still acceptable).
Important: Write sentences like Y is "Switzerland". Make sure there is no typo in what you quote.
(3) Explain how you locate the answer with the phrases you marked; Only use the suggested expressions in the table in the bottom.

1) Creative question writing: Given a term (e.g., silk), a description of the term, and an expected answer (yes or no), the task is to write a strategy question about the term with the expected answer, and the facts required to answer the question. 2) Strategy question decomposition: Given a strategy question, a yes/no answer, and a set of facts, the task is to write the steps needed to answer the question. 3) Evidence matching: Given a question and its de- composition (a list of single-step questions), the task is to find evidence paragraphs on Wikipedia for each retrieval step. Operation steps that do not require retrieval are marked as operation.

QED: A Framework and Dataset for Explanations in Question Answering

Abstract: A question answering system that in addition to providing an answer provides an explanation of the reasoning that leads to that answer has potential advantages in terms of debuggability, extensibility and trust. To this end, we propose QED, a linguistically informed, extensible framework for explanations in question answering. A QED explanation specifies the relationship between a question and answer according to formal semantic notions such as referential equality, sentencehood, and entailment. We describe and publicly release an expert-annotated dataset of QED explanations built upon a subset of the Google Natural Questions dataset, and report baseline models on two tasks -- post-hoc explanation generation given an answer, and joint question answering and explanation generation. In the joint setting, a promising result suggests that training on a relatively small amount of QED data can improve question answering. In addition to describing the formal, language-theoretic motivations for the QED approach, we describe a large user study showing that the presence of QED explanations significantly improves the ability of untrained raters to spot errors made by a strong neural QA baseline.

Teaching Machine Comprehension with Compositional Explanations

Abstract: Advances in machine reading comprehension (MRC) rely heavily on the collection of large scale human-annotated examples in the form of (question, paragraph, answer) triples. In contrast, humans are typically able to generalize with only a few examples, relying on deeper underlying world knowledge, linguistic sophistication, and/or simply superior deductive powers. In this paper, we focus on "teaching" machines reading comprehension, using a small number of semi-structured explanations that explicitly inform machines why answer spans are correct. We extract structured variables and rules from explanations and compose neural module teachers that annotate instances for training downstream MRC models. We use learnable neural modules and soft logic to handle linguistic variation and overcome sparse coverage; the modules are jointly optimized with the MRC model to improve final performance. On the SQuAD dataset, our proposed method achieves 70.14% F1 score with supervision from 26 explanations, comparable to plain supervised learning using 1,100 labeled instances, yielding a 12x speed up.

Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies

Abstract: A key limitation in current datasets for multi-hop reasoning is that the required steps for answering the question are mentioned in it explicitly. In this work, we introduce StrategyQA, a question answering (QA) benchmark where the required reasoning steps are implicit in the question, and should be inferred using a strategy. A fundamental challenge in this setup is how to elicit such creative questions from crowdsourcing workers, while covering a broad range of potential strategies. We propose a data collection procedure that combines term-based priming to inspire annotators, careful control over the annotator population, and adversarial filtering for eliminating reasoning shortcuts. Moreover, we annotate each question with (1) a decomposition into reasoning steps for answering it, and (2) Wikipedia paragraphs that contain the answers to each step. Overall, StrategyQA includes 2,780 examples, each consisting of a strategy question, its decomposition, and evidence paragraphs. Analysis shows that questions in StrategyQA are short, topic-diverse, and cover a wide range of strategies. Empirically, we show that humans perform well (87%) on this task, while our best baseline reaches an accuracy of $\sim$66%.

Questions that prior work uses to solicit human explanations redfor procedure (in structured explanations).

Proxy Questions for Proximal Mechanisms and Procedure

Procedures.
We cannot identify the exact questions in three of five papers for explicitly step-by-step procedures, which reflects the importance of reporting detailed annotation guidelines.
As researchers collect step-by-step guidelines, {{cite:2632bed}} and {{cite:fffd2a4}} adopt very different decomposition for their problems (see Table {{table:af09f5ef-07ec-4885-ae8b-09da296e35ad}} ).


Features: RefineNet {{cite:d70ee4b}} Descriptor: LosT (semantic label scores + conv5 layer feature maps ) + normalization BG: cosine distance

Features: RefineNet(Resnet101) {{cite:d70ee4b}}Descriptor: conv 5 feature maps BG: cosine distance

Top Candidates:Features: MobileNetV2 {{cite:e97f286}} Descriptor: final average pooling layer BG: nearest neighbors + normalized scalar product {{cite:daa71e3}}

1st Tier: Features: HybridNet(AlexNet) {{cite:30873b9}} and Gist {{cite:4717847}} BG: Difference Scores + normalization 2nd Tier:Features: KAZE {{cite:569774a}} and Only Look Once {{cite:549232b}} BG: (KAZE) sum of the residual distances and difference scores + normalization

Features: RefineNet {{cite:d70ee4b}} Descriptor: keypoint extracted from CNN layer activations GB: spatial layout Verification ( Semantic Label Consistency + weighted Euclidean distance)

Filtering: out-of-range keypoins based on depth maps Descriptor: same as coarse stage (filtered) BG: cosine distance

Features: SURF Descriptor: hash codes BG: Hamming Distance between top candidates and SURF-based descriptor + ratio test {{cite:f367b4c}} + RANSAC

3 Tier: Features: NetVLAD {{cite:9c50cad}} and HOG {{cite:5e47563}} BG: max(Average of Difference Scores)

Coarse: HNSW graphs Fine: Top candidates

1st tier: Database 2nd tier: Top candidates of the 1st tier 3th tier: Top candidates of the 2nd tier

Oxford Robotcar {{cite:ef1fbf9}}; Synthia Dataset {{cite:2ef6f6e}};

Oxford Robotcar {{cite:ef1fbf9}} Synthia {{cite:2ef6f6e}} (for depth evaluation)

KITTI {{cite:d6826ef}} Malaga 2009 Parking 6L {{cite:915a1ec}} New College {{cite:344948c}}

Nordland {{cite:561f4f0}} Berlin Kurfurstendamm {{cite:51679d5}}

LoST? Appearance-Invariant Place Recognition for Opposite Viewpoints using Visual Semantics

Abstract: Human visual scene understanding is so remarkable that we are able to recognize a revisited place when entering it from the opposite direction it was first visited, even in the presence of extreme variations in appearance. This capability is especially apparent during driving: a human driver can recognize where they are when travelling in the reverse direction along a route for the first time, without having to turn back and look. The difficulty of this problem exceeds any addressed in past appearance- and viewpoint-invariant visual place recognition (VPR) research, in part because large parts of the scene are not commonly observable from opposite directions. Consequently, as shown in this paper, the precision-recall performance of current state-of-the-art viewpoint- and appearance-invariant VPR techniques is orders of magnitude below what would be usable in a closed-loop system. Current engineered solutions predominantly rely on panoramic camera or LIDAR sensing setups; an eminently suitable engineering solution but one that is clearly very different to how humans navigate, which also has implications for how naturally humans could interact and communicate with the navigation system. In this paper we develop a suite of novel semantic- and appearance-based techniques to enable for the first time high performance place recognition in this challenging scenario. We first propose a novel Local Semantic Tensor (LoST) descriptor of images using the convolutional feature maps from a state-of-the-art dense semantic segmentation network. Then, to verify the spatial semantic arrangement of the top matching candidates, we develop a novel approach for mining semantically-salient keypoint correspondences.

Look No Deeper: Recognizing Places from Opposing Viewpoints under Varying Scene Appearance using Single-View Depth Estimation

Abstract: Visual place recognition (VPR) - the act of recognizing a familiar visual place - becomes difficult when there is extreme environmental appearance change or viewpoint change. Particularly challenging is the scenario where both phenomena occur simultaneously, such as when returning for the first time along a road at night that was previously traversed during the day in the opposite direction. While such problems can be solved with panoramic sensors, humans solve this problem regularly with limited field-of-view vision and without needing to constantly turn around. In this paper, we present a new depth- and temporal-aware visual place recognition system that solves the opposing viewpoint, extreme appearance-change visual place recognition problem. Our system performs sequence-to-single frame matching by extracting depth-filtered keypoints using a state-of-the-art depth estimation pipeline, constructing a keypoint sequence over multiple frames from the reference dataset, and comparing these keypoints to the keypoints extracted from a single query image. We evaluate the system on a challenging benchmark dataset and show that it consistently outperforms state-of-the-art techniques. We also develop a range of diagnostic simulation experiments that characterize the contribution of depth-filtered keypoint sequences with respect to key domain parameters including the degree of appearance change and camera motion.

Fast and Incremental Loop Closure Detection Using Proximity Graphs

Abstract: Visual loop closure detection, which can be considered as an image retrieval task, is an important problem in SLAM (Simultaneous Localization and Mapping) systems. The frequently used bag-of-words (BoW) models can achieve high precision and moderate recall. However, the requirement for lower time costs and fewer memory costs for mobile robot applications is not well satisfied. In this paper, we propose a novel loop closure detection framework titled FILD’ (Fast and Incremental Loop closure Detection), which focuses on an on-line and incremental graph vocabulary construction for fast loop closure detection. The global and local features of frames are extracted using the Convolutional Neural Networks (CNN) and SURF on the GPU, which guarantee extremely fast extraction speeds. The graph vocabulary construction is based on one type of proximity graph, named Hierarchical Navigable Small World (HNSW) graphs, which is modified to adapt to this specific application. In addition, this process is coupled with a novel strategy for real-time geometrical verification, which only keeps binary hash codes and significantly saves on memory usage. Extensive experiments on several publicly available datasets show that the proposed approach can achieve fairly good recall at 100% precision compared to other state-of-the-art methods. The source code can be downloaded at https://github.comlAnshanTJU/FILD for further studies.

Hierarchical Multi-Process Fusion for Visual Place Recognition

Abstract: Combining multiple complementary techniques together has long been regarded as a way to improve performance. In visual localization, multi-sensor fusion, multi-process fusion of a single sensing modality, and even combinations of different localization techniques have been shown to result in improved performance. However, merely fusing together different localization techniques does not account for the varying performance characteristics of different localization techniques. In this paper we present a novel, hierarchical localization system that explicitly benefits from three varying characteristics of localization techniques: the distribution of their localization hypotheses, their appearance- and viewpointinvariant properties, and the resulting differences in where in an environment each system works well and fails. We show how two techniques deployed hierarchically work better than in parallel fusion, how combining two different techniques works better than two levels of a single technique, even when the single technique has superior individual performance, and develop two and three-tier hierarchical structures that progressively improve localization performance. Finally, we develop a stacked hierarchical framework where localization hypotheses from techniques with complementary characteristics are concatenated at each layer, significantly improving retention of the correct hypothesis through to the final localization stage. Using two challenging datasets, we show the proposed system outperforming state-of-the-art techniques.

Summary of recent works using hierarchical frameworks. BG = Belief Generation and PM = Place mapping.

In this work, hierarchical frameworks refer to place recognition approaches that, similarly to parallel frameworks, rely on multiple methods; however, instead of having as main framework a parallel architecture, the architecture is formed by various stacked tiers. The hierarchical architectures find the best loop candidate by filter candidates progressively in each tier. An example of such a framework is the coarse-to-fine architecture, which has a coarse and a fine tier. The coarse tier is mostly dedicated to retrieving top candidates utilizing methods that rather are computer efficient than accurate. These top candidates are feed to the fine tier, which can use more computer demanding methods to find the best loop candidate. The coarse-to-fine architecture, while being the most common, is not the only. Other architectures exist, for example Fig. {{figure:f459c73b-432b-4061-87a0-113c0a93b5be}}  illustrates a framework proposed in {{cite:725382c}} and Table {{table:8003c93a-80e9-4315-8950-bda1eebcf6f8}}  presents a summary of recent works.


WaveNet: A Generative Model for Raw Audio

Abstract: This paper introduces WaveNet, a deep neural network for generating raw audio waveforms. The model is fully probabilistic and autoregressive, with the predictive distribution for each audio sample conditioned on all previous ones; nonetheless we show that it can be efficiently trained on data with tens of thousands of samples per second of audio. When applied to text-to-speech, it yields state-of-the-art performance, with human listeners rating it as significantly more natural sounding than the best parametric and concatenative systems for both English and Mandarin. A single WaveNet can capture the characteristics of many different speakers with equal fidelity, and can switch between them by conditioning on the speaker identity. When trained to model music, we find that it generates novel and often highly realistic musical fragments. We also show that it can be employed as a discriminative model, returning promising results for phoneme recognition.

SampleRNN: An Unconditional End-to-End Neural Audio Generation Model

Abstract: In this paper we propose a novel model for unconditional audio generation based on generating one audio sample at a time. We show that our model, which profits from combining memory-less modules, namely autoregressive multilayer perceptrons, and stateful recurrent neural networks in a hierarchical structure is able to capture underlying sources of variations in the temporal sequences over very long time spans, on three datasets of different nature. Human evaluation on the generated samples indicate that our model is preferred over competing models. We also show how each component of the model contributes to the exhibited performance.

Efficient Neural Audio Synthesis

Abstract: Sequential models achieve state-of-the-art results in audio, visual and textual domains with respect to both estimating the data distribution and generating high-quality samples. Efficient sampling for this class of models has however remained an elusive problem. With a focus on text-to-speech synthesis, we describe a set of general techniques for reducing sampling time while maintaining high output quality. We first describe a single-layer recurrent neural network, the WaveRNN, with a dual softmax layer that matches the quality of the state-of-the-art WaveNet model. The compact form of the network makes it possible to generate 24kHz 16-bit audio 4x faster than real time on a GPU. Second, we apply a weight pruning technique to reduce the number of weights in the WaveRNN. We find that, for a constant number of parameters, large sparse networks perform better than small dense networks and this relationship holds for sparsity levels beyond 96%. The small number of weights in a Sparse WaveRNN makes it possible to sample high-fidelity audio on a mobile CPU in real time. Finally, we propose a new generation scheme based on subscaling that folds a long sequence into a batch of shorter sequences and allows one to generate multiple samples at once. The Subscale WaveRNN produces 16 samples per step without loss of quality and offers an orthogonal method for increasing sampling efficiency.

LPCNet: Improving Neural Speech Synthesis Through Linear Prediction

Abstract: Neural speech synthesis models have recently demonstrated the ability to synthesize high quality speech for text-to-speech and compression applications. These new models often require powerful GPUs to achieve real-time operation, so being able to reduce their complexity would open the way for many new applications. We propose LPCNet, a WaveRNN variant that combines linear prediction with recurrent neural networks to significantly improve the efficiency of speech synthesis. We demonstrate that LPCNet can achieve significantly higher quality than WaveRNN for the same network size and that high quality LPCNet speech synthesis is achievable with a complexity under 3 GFLOPS. This makes it easier to deploy neural synthesis applications on lower-power devices, such as embedded systems and mobile phones.

Speaker Conditional WaveRNN: Towards Universal Neural Vocoder for Unseen Speaker and Recording Conditions

Abstract: Recent advancements in deep learning led to human-level performance in single-speaker speech synthesis. However, there are still limitations in terms of speech quality when generalizing those systems into multiple-speaker models especially for unseen speakers and unseen recording qualities. For instance, conventional neural vocoders are adjusted to the training speaker and have poor generalization capabilities to unseen speakers. In this work, we propose a variant of WaveRNN, referred to as speaker conditional WaveRNN (SC-WaveRNN). We target towards the development of an efficient universal vocoder even for unseen speakers and recording conditions. In contrast to standard WaveRNN, SC-WaveRNN exploits additional information given in the form of speaker embeddings. Using publicly-available data for training, SC-WaveRNN achieves significantly better performance over baseline WaveRNN on both subjective and objective metrics. In MOS, SC-WaveRNN achieves an improvement of about 23% for seen speaker and seen recording condition and up to 95% for unseen speaker and unseen condition. Finally, we extend our work by implementing a multi-speaker text-to-speech (TTS) synthesis similar to zero-shot speaker adaptation. In terms of performance, our system has been preferred over the baseline TTS system by 60% over 15.5% and by 60.9% over 32.6%, for seen and unseen speakers, respectively.

Parallel WaveNet: Fast High-Fidelity Speech Synthesis

Abstract: The recently-developed WaveNet architecture is the current state of the art in realistic speech synthesis, consistently rated as more natural sounding for many different languages than any previous system. However, because WaveNet relies on sequential generation of one audio sample at a time, it is poorly suited to today's massively parallel computers, and therefore hard to deploy in a real-time production setting. This paper introduces Probability Density Distillation, a new method for training a parallel feed-forward network from a trained WaveNet with no significant difference in quality. The resulting system is capable of generating high-fidelity speech samples at more than 20 times faster than real-time, and is deployed online by Google Assistant, including serving multiple English and Japanese voices.

WaveGlow: A Flow-based Generative Network for Speech Synthesis

Abstract: In this paper we propose WaveGlow: a flow-based network capable of generating high quality speech from mel-spectrograms. WaveGlow combines insights from Glow and WaveNet in order to provide fast, efficient and high-quality audio synthesis, without the need for auto-regression. WaveGlow is implemented using only a single network, trained using only a single cost function: maximizing the likelihood of the training data, which makes the training procedure simple and stable. Our PyTorch implementation produces audio samples at a rate of more than 500 kHz on an NVIDIA V100 GPU. Mean Opinion Scores show that it delivers audio quality as good as the best publicly available WaveNet implementation. All code will be made publicly available online.

FloWaveNet : A Generative Flow for Raw Audio

Abstract: Most modern text-to-speech architectures use a WaveNet vocoder for synthesizing high-fidelity waveform audio, but there have been limitations, such as high inference time, in its practical application due to its ancestral sampling scheme. The recently suggested Parallel WaveNet and ClariNet have achieved real-time audio synthesis capability by incorporating inverse autoregressive flow for parallel sampling. However, these approaches require a two-stage training pipeline with a well-trained teacher network and can only produce natural sound by using probability distillation along with auxiliary loss terms. We propose FloWaveNet, a flow-based generative model for raw audio synthesis. FloWaveNet requires only a single-stage training procedure and a single maximum likelihood loss, without any additional auxiliary terms, and it is inherently parallel due to the characteristics of generative flow. The model can efficiently sample raw audio in real-time, with clarity comparable to previous two-stage parallel models. The code and samples for all models, including our FloWaveNet, are publicly available.

WaveFlow: A Compact Flow-based Model for Raw Audio

Abstract: In this work, we propose WaveFlow, a small-footprint generative flow for raw audio, which is directly trained with maximum likelihood. It handles the long-range structure of 1-D waveform with a dilated 2-D convolutional architecture, while modeling the local variations using expressive autoregressive functions. WaveFlow provides a unified view of likelihood-based models for 1-D data, including WaveNet and WaveGlow as special cases. It generates high-fidelity speech as WaveNet, while synthesizing several orders of magnitude faster as it only requires a few sequential steps to generate very long waveforms with hundreds of thousands of time-steps. Furthermore, it can significantly reduce the likelihood gap that has existed between autoregressive models and flow-based models for efficient synthesis. Finally, our small-footprint WaveFlow has only 5.91M parameters, which is 15$\times$ smaller than WaveGlow. It can generate 22.05 kHz high-fidelity audio 42.6$\times$ faster than real-time (at a rate of 939.3 kHz) on a V100 GPU without engineered inference kernels.

SqueezeWave: Extremely Lightweight Vocoders for On-device Speech Synthesis

Abstract: Automatic speech synthesis is a challenging task that is becoming increasingly important as edge devices begin to interact with users through speech. Typical text-to-speech pipelines include a vocoder, which translates intermediate audio representations into an audio waveform. Most existing vocoders are difficult to parallelize since each generated sample is conditioned on previous samples. WaveGlow is a flow-based feed-forward alternative to these auto-regressive models (Prenger et al., 2019). However, while WaveGlow can be easily parallelized, the model is too expensive for real-time speech synthesis on the edge. This paper presents SqueezeWave, a family of lightweight vocoders based on WaveGlow that can generate audio of similar quality to WaveGlow with 61x - 214x fewer MACs. Code, trained models, and generated audio are publicly available at https://github.com/tianrengao/SqueezeWave.

Adversarial Audio Synthesis

Abstract: Audio signals are sampled at high temporal resolutions, and learning to synthesize audio requires capturing structure across a range of timescales. Generative adversarial networks (GANs) have seen wide success at generating images that are both locally and globally coherent, but they have seen little application to audio generation. In this paper we introduce WaveGAN, a first attempt at applying GANs to unsupervised synthesis of raw-waveform audio. WaveGAN is capable of synthesizing one second slices of audio waveforms with global coherence, suitable for sound effect generation. Our experiments demonstrate that, without labels, WaveGAN learns to produce intelligible words when trained on a small-vocabulary speech dataset, and can also synthesize audio from other domains such as drums, bird vocalizations, and piano. We compare WaveGAN to a method which applies GANs designed for image generation on image-like audio feature representations, finding both approaches to be promising.

High Fidelity Speech Synthesis with Adversarial Networks

Abstract: Generative adversarial networks have seen rapid development in recent years and have led to remarkable improvements in generative modelling of images. However, their application in the audio domain has received limited attention, and autoregressive models, such as WaveNet, remain the state of the art in generative modelling of audio signals such as human speech. To address this paucity, we introduce GAN-TTS, a Generative Adversarial Network for Text-to-Speech. Our architecture is composed of a conditional feed-forward generator producing raw speech audio, and an ensemble of discriminators which operate on random windows of different sizes. The discriminators analyse the audio both in terms of general realism, as well as how well the audio corresponds to the utterance that should be pronounced. To measure the performance of GAN-TTS, we employ both subjective human evaluation (MOS - Mean Opinion Score), as well as novel quantitative metrics (Fr\'echet DeepSpeech Distance and Kernel DeepSpeech Distance), which we find to be well correlated with MOS. We show that GAN-TTS is capable of generating high-fidelity speech with naturalness comparable to the state-of-the-art models, and unlike autoregressive models, it is highly parallelisable thanks to an efficient feed-forward generator. Listen to GAN-TTS reading this abstract at https://storage.googleapis.com/deepmind-media/research/abstract.wav.

Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram

Abstract: We propose Parallel WaveGAN, a distillation-free, fast, and small-footprint waveform generation method using a generative adversarial network. In the proposed method, a non-autoregressive WaveNet is trained by jointly optimizing multi-resolution spectrogram and adversarial loss functions, which can effectively capture the time-frequency distribution of the realistic speech waveform. As our method does not require density distillation used in the conventional teacher-student framework, the entire model can be easily trained. Furthermore, our model is able to generate high-fidelity speech even with its compact architecture. In particular, the proposed Parallel WaveGAN has only 1.44 M parameters and can generate 24 kHz speech waveform 28.68 times faster than real-time on a single GPU environment. Perceptual listening test results verify that our proposed method achieves 4.16 mean opinion score within a Transformer-based text-to-speech framework, which is comparative to the best distillation-based Parallel WaveNet system.

HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis

Abstract: Several recent work on speech synthesis have employed generative adversarial networks (GANs) to produce raw waveforms. Although such methods improve the sampling efficiency and memory usage, their sample quality has not yet reached that of autoregressive and flow-based generative models. In this work, we propose HiFi-GAN, which achieves both efficient and high-fidelity speech synthesis. As speech audio consists of sinusoidal signals with various periods, we demonstrate that modeling periodic patterns of an audio is crucial for enhancing sample quality. A subjective human evaluation (mean opinion score, MOS) of a single speaker dataset indicates that our proposed method demonstrates similarity to human quality while generating 22.05 kHz high-fidelity audio 167.9 times faster than real-time on a single V100 GPU. We further show the generality of HiFi-GAN to the mel-spectrogram inversion of unseen speakers and end-to-end speech synthesis. Finally, a small footprint version of HiFi-GAN generates samples 13.4 times faster than real-time on CPU with comparable quality to an autoregressive counterpart.

VocGAN: A High-Fidelity Real-time Vocoder with a Hierarchically-nested Adversarial Network

Abstract: We present a novel high-fidelity real-time neural vocoder called VocGAN. A recently developed GAN-based vocoder, MelGAN, produces speech waveforms in real-time. However, it often produces a waveform that is insufficient in quality or inconsistent with acoustic characteristics of the input mel spectrogram. VocGAN is nearly as fast as MelGAN, but it significantly improves the quality and consistency of the output waveform. VocGAN applies a multi-scale waveform generator and a hierarchically-nested discriminator to learn multiple levels of acoustic properties in a balanced way. It also applies the joint conditional and unconditional objective, which has shown successful results in high-resolution image synthesis. In experiments, VocGAN synthesizes speech waveforms 416.7x faster on a GTX 1080Ti GPU and 3.24x faster on a CPU than real-time. Compared with MelGAN, it also exhibits significantly improved quality in multiple evaluation metrics including mean opinion score (MOS) with minimal additional overhead. Additionally, compared with Parallel WaveGAN, another recently developed high-fidelity vocoder, VocGAN is 6.98x faster on a CPU and exhibits higher MOS.

A Spectral Energy Distance for Parallel Speech Synthesis

Abstract: Speech synthesis is an important practical generative modeling problem that has seen great progress over the last few years, with likelihood-based autoregressive neural models now outperforming traditional concatenative systems. A downside of such autoregressive models is that they require executing tens of thousands of sequential operations per second of generated audio, making them ill-suited for deployment on specialized deep learning hardware. Here, we propose a new learning method that allows us to train highly parallel models of speech, without requiring access to an analytical likelihood function. Our approach is based on a generalized energy distance between the distributions of the generated and real audio. This spectral energy distance is a proper scoring rule with respect to the distribution over magnitude-spectrograms of the generated waveform audio and offers statistical consistency guarantees. The distance can be calculated from minibatches without bias, and does not involve adversarial learning, yielding a stable and consistent method for training implicit generative models. Empirically, we achieve state-of-the-art generation quality among implicit generative models, as judged by the recently-proposed cFDSD metric. When combining our method with adversarial techniques, we also improve upon the recently-proposed GAN-TTS model in terms of Mean Opinion Score as judged by trained human evaluators.

Fre-GAN: Adversarial Frequency-consistent Audio Synthesis

Abstract: Although recent works on neural vocoder have improved the quality of synthesized audio, there still exists a gap between generated and ground-truth audio in frequency space. This difference leads to spectral artifacts such as hissing noise or reverberation, and thus degrades the sample quality. In this paper, we propose Fre-GAN which achieves frequency-consistent audio synthesis with highly improved generation quality. Specifically, we first present resolution-connected generator and resolution-wise discriminators, which help learn various scales of spectral distributions over multiple frequency bands. Additionally, to reproduce high-frequency components accurately, we leverage discrete wavelet transform in the discriminators. From our experiments, Fre-GAN achieves high-fidelity waveform generation with a gap of only 0.03 MOS compared to ground-truth audio while outperforming standard models in quality.

WaveGrad: Estimating Gradients for Waveform Generation

Abstract: This paper introduces WaveGrad, a conditional model for waveform generation which estimates gradients of the data density. The model is built on prior work on score matching and diffusion probabilistic models. It starts from a Gaussian white noise signal and iteratively refines the signal via a gradient-based sampler conditioned on the mel-spectrogram. WaveGrad offers a natural way to trade inference speed for sample quality by adjusting the number of refinement steps, and bridges the gap between non-autoregressive and autoregressive models in terms of audio quality. We find that it can generate high fidelity audio samples using as few as six iterations. Experiments reveal WaveGrad to generate high fidelity audio, outperforming adversarial non-autoregressive baselines and matching a strong likelihood-based autoregressive baseline using fewer sequential operations. Audio samples are available at this https URL.

DiffWave: A Versatile Diffusion Model for Audio Synthesis

Abstract: In this work, we propose DiffWave, a versatile Diffusion probabilistic model for conditional and unconditional Waveform generation. The model is non-autoregressive, and converts the white noise signal into structured waveform through a Markov chain with a constant number of steps at synthesis. It is efficiently trained by optimizing a variant of variational bound on the data likelihood. DiffWave produces high-fidelity audios in Different Waveform generation tasks, including neural vocoding conditioned on mel spectrogram, class-conditional generation, and unconditional generation. We demonstrate that DiffWave matches a strong WaveNet vocoder in terms of speech quality~(MOS: 4.44 versus 4.43), while synthesizing orders of magnitude faster. In particular, it significantly outperforms autoregressive and GAN-based waveform models in the challenging unconditional generation task in terms of audio quality and sample diversity from various automatic and human evaluations.

A list of vocoders and their corresponding characteristics.

Early neural vocoders such as WaveNet {{cite:a6c069d}}, {{cite:f83b996}}, Char2Wav {{cite:c0c2cab}}, WaveRNN {{cite:5a2781d}} directly take linguistic features as input and generate waveform. Later,  {{cite:87b3c74}}, {{cite:042df3b}}, {{cite:4267deb}}, {{cite:bc8f649}} take mel-spectrograms as input and generate waveform. Since speech waveform is very long, autoregressive waveform generation takes much inference time. Thus, generative models such as Flow {{cite:fe50488}}, {{cite:7e2642e}}, {{cite:a34a967}}, GAN {{cite:370e37b}}, VAE {{cite:bb3bc3d}}, and DDPM (Denoising Diffusion Probabilistic Model, Diffusion for short) {{cite:39c6603}}, {{cite:fdbbe5e}} are used in waveform generation. Accordingly, we divide the neural vocoders into different categories: 1) Autoregressive vocoders, 2) Flow-based vocoders, 3) GAN-based vocoders, 4) VAE-based vocoders, and 5) Diffusion-based vocoders. We list some representative vocoders in Table {{table:5c4dad86-5ca8-414f-83aa-c9baff7a2c62}}  and describe them as follows.


valence, arousal, latent affective features, blink rate, gaze direction, head pose, and hand pose

eye gaze, head pose, head
rotation, and AU

DAiSEE: Towards User Engagement Recognition in the Wild

Abstract: We introduce DAiSEE, the first multi-label video classification dataset comprising of 9068 video snippets captured from 112 users for recognizing the user affective states of boredom, confusion, engagement, and frustration in the wild. The dataset has four levels of labels namely - very low, low, high, and very high for each of the affective states, which are crowd annotated and correlated with a gold standard annotation created using a team of expert psychologists. We have also established benchmark results on this dataset using state-of-the-art video classification methods that are available today. We believe that DAiSEE will provide the research community with challenges in feature extraction, context-based inference, and development of suitable machine learning methods for related tasks, thus providing a springboard for further research. The dataset is available for download at https://people.iith.ac.in/vineethnb/resources/daisee/index.html.

Prediction and Localization of Student Engagement in the Wild

Abstract: Digital revolution has transformed the traditional teaching procedures, students are going online to access study materials. It is realised that analysis of student engagement in an e-learning environment would facilitate effective task accomplishment and learning. Well known social cues of engagement/disengagement can be inferred from facial expressions, body movements and gaze patterns. In this paper, student's response to various stimuli (educational videos) are recorded and cues are extracted to estimate variations in engagement level. We study the association of a subject's behavioral cues with his/her engagement level, as annotated by labelers. We have localized engaging/non-engaging parts in the stimuli videos using a deep multiple instance learning based framework, which can give useful insight into designing Massive Open Online Courses (MOOCs) video material. Recognizing the lack of any publicly available dataset in the domain of user engagement, a new ‘in the wild’ dataset is curated. The dataset: Engagement in the Wild contains 264 videos captured from 91 subjects, which is approximately 16.5 hours of recording. Detailed baseline results using different classifiers ranging from traditional machine learning to deep learning based approaches are evaluated on the database. Subject independent analysis is performed and the task of engagement prediction is modeled as a weakly supervised learning problem. The dataset is manually annotated by different labelers and the correlation studies between annotated and predicted labels of videos by different classifiers are reported. This dataset creation is an effort to facilitate research in various e-learning environments such as intelligent tutoring systems, MOOCs, and others.

(Upper section) End-to-end video-based engagement measurement approaches, their video-analysis classification (C) or regression (R) models, and the datasets they used. (Lower section) Feature-based video engagement measurement approaches, their features, feature fusion techniques, classification (C) or regression (R) models, and the datasets they used.

In end-to-end engagement measurement approaches, no hand-crafted features are extracted from videos. The consecutive raw frames of RGB videos are fed to a deep CNN (or its variant) classification, or regression model to output a class label, or a continuous value corresponding to the engagement level of the person in the video. The upper half of Table {{table:2526f858-4b5a-4852-9f23-1ef2fbf05800}}  summarizes the literature on the end-to-end video-based engagement measurement approaches focusing on their deep-learning video-analysis models. Most of the existing end-to-end approaches evaluated their method on the DAiSEE dataset {{cite:25c8f1f}} (described in Section ) for engagement level classification. Various techniques, such as using different loss functions or adding attention mechanisms to the neural networks, have been employed by researchers to enhance the results of their proposed methodologies {{cite:1fa4e44}}.

Different from the end-to-end approaches, in feature-based approaches, first, multi-modal handcrafted features are extracted from videos, and then the features are fed to a classification or regression model to output engagement {{cite:ba5e3f9}}, {{cite:97bdd04}}, {{cite:182050e}}, {{cite:5a16e17}}, {{cite:b28f0dd}}, {{cite:c3daa3d}}, {{cite:8697303}}, {{cite:edbd7ce}}, {{cite:ed8767c}}, {{cite:ec847b2}}, {{cite:3809e70}}, {{cite:eaa4d66}}. The lower half of Table {{table:2526f858-4b5a-4852-9f23-1ef2fbf05800}}  summarizes the literature on feature-based engagement measurement approaches focusing on their features, feature fusion techniques, machine-learning models, and datasets.

As can be seen in the lower section of Table {{table:2526f858-4b5a-4852-9f23-1ef2fbf05800}} , in some of the previous methods, conventional computer-vision features such as box filter, Gabor {{cite:5a16e17}}, or LBP-TOP {{cite:3809e70}} are used for engagement measurement. In some other methods, such as {{cite:ba5e3f9}}, for extracting facial embedding features, a CNN containing convolutional layers, followed by fully-connected layers, is trained on a face recognition or facial expression recognition dataset. Then, the output of the convolutional layers in the pretrained model is used as the facial embedding features. In most of the previous methods, facial action units (AUs), eye movement, gaze direction, and head pose features are extracted using OpenFace {{cite:b6850ed}}, or body pose features are extracted using OpenPose {{cite:07480fd}}. Various features are extracted and concatenated using these feature extraction techniques to construct a feature vector for each video frame or each video clip.


Performance Measures and a Data Set for Multi-Target, Multi-Camera Tracking

Abstract: To help accelerate progress in multi-target, multi-camera tracking systems, we present (i) a new pair of precision-recall measures of performance that treats errors of all types uniformly and emphasizes correct identification over sources of error; (ii) the largest fully-annotated and calibrated data set to date with more than 2 million frames of 1080p, 60fps video taken by 8 cameras observing more than 2,700 identities over 85 minutes; and (iii) a reference software system as a comparison baseline. We show that (i) our measures properly account for bottom-line identity match performance in the multi-camera setting; (ii) our data set poses realistic challenges to current trackers; and (iii) the performance of our system is comparable to the state of the art.

Person Transfer GAN to Bridge Domain Gap for Person Re-Identification

Abstract: Although the performance of person Re-Identification (ReID) has been significantly boosted, many challenging issues in real scenarios have not been fully investigated, e.g., the complex scenes and lighting variations, viewpoint and pose changes, and the large number of identities in a camera network. To facilitate the research towards conquering those issues, this paper contributes a new dataset called MSMT17 with many important features, e.g., 1) the raw videos are taken by an 15-camera network deployed in both indoor and outdoor scenes, 2) the videos cover a long period of time and present complex lighting variations, and 3) it contains currently the largest number of annotated identities, i.e., 4,101 identities and 126,441 bounding boxes. We also observe that, domain gap commonly exists between datasets, which essentially causes severe performance drop when training and testing on different datasets. This results in that available training data cannot be effectively leveraged for new testing domains. To relieve the expensive costs of annotating new training samples, we propose a Person Transfer Generative Adversarial Network (PTGAN) to bridge the domain gap. Comprehensive experiments show that the domain gap could be substantially narrowed-down by the PTGAN.

Key properties of the person Re-ID datasets. `Identities', `Images', and `Cameras' denotes the number of identities, images, and cameras of each dataset, respectively.

Three publicly available datasets, Market-1501 {{cite:861b797}}, DukeMTMC-Re-ID {{cite:a95abf5}}, and MSMT17 {{cite:05162e5}}, are exploited in our experiments. Key properties of the three dataset are described in Table {{table:3643b3e1-89e1-4d7d-82a1-9b4f30aac3da}} . In particular, MSMT17 suffers from substantial variations of scene and illumination conditions, and those issues can be regarded as a challenging problem compared with the other two datasets. Our experiments for unsupervised person Re-ID have conducted with the standard protocols {{cite:861b797}}, {{cite:a95abf5}}, {{cite:05162e5}}. Cumulative Matching Characteristics (CMC) and Mean Average Precision (mAP) are used as evaluation metrics.


nuScenes: A Multimodal Dataset for Autonomous Driving

Abstract: Robust detection and tracking of objects is crucial for the deployment of autonomous vehicle technology. Image based benchmark datasets have driven development in computer vision tasks such as object detection, tracking and segmentation of agents in the environment. Most autonomous vehicles, however, carry a combination of cameras and range sensors such as lidar and radar. As machine learning based methods for detection and tracking become more prevalent, there is a need to train and evaluate such methods on datasets containing range sensor data along with images. In this work we present nuTonomy scenes (nuScenes), the first dataset to carry the full autonomous vehicle sensor suite: 6 cameras, 5 radars and 1 lidar, all with full 360 degree field of view. nuScenes comprises 1000 scenes, each 20s long and fully annotated with 3D bounding boxes for 23 classes and 8 attributes. It has 7x as many annotations and 100x as many images as the pioneering KITTI dataset. We define novel 3D detection and tracking metrics. We also provide careful dataset analysis as well as baselines for lidar and image based detection and tracking. Data, development kit and more information are available online.

Scalability in Perception for Autonomous Driving: Waymo Open Dataset

Abstract: The research community has increasing interest in autonomous driving research, despite the resource intensity of obtaining representative real world data. Existing self-driving datasets are limited in the scale and variation of the environments they capture, even though generalization within and between operating regions is crucial to the overall viability of the technology. In an effort to help align the research community's contributions with real-world self-driving problems, we introduce a new large scale, high quality, diverse dataset. Our new dataset consists of 1150 scenes that each span 20 seconds, consisting of well synchronized and calibrated high quality LiDAR and camera data captured across a range of urban and suburban geographies. It is 15x more diverse than the largest camera+LiDAR dataset available based on our proposed diversity metric. We exhaustively annotated this data with 2D (camera image) and 3D (LiDAR) bounding boxes, with consistent identifiers across frames. Finally, we provide strong baselines for 2D as well as 3D detection and tracking tasks. We further study the effects of dataset size and generalization across geographies on 3D detection methods. Find data, code and more up-to-date information at http://www.waymo.com/open.

A*3D Dataset: Towards Autonomous Driving in Challenging Environments

Abstract: With the increasing global popularity of self-driving cars, there is an immediate need for challenging real-world datasets for benchmarking and training various computer vision tasks such as 3D object detection. Existing datasets either represent simple scenarios or provide only day-time data. In this paper, we introduce a new challenging A*3D dataset which consists of RGB images and LiDAR data with significant diversity of scene, time, and weather. The dataset consists of high-density images ($\approx~10$ times more than the pioneering KITTI dataset), heavy occlusions, a large number of night-time frames ($\approx~3$ times the nuScenes dataset), addressing the gaps in the existing datasets to push the boundaries of tasks in autonomous driving research to more challenging highly diverse environments. The dataset contains $39\text{K}$ frames, $7$ classes, and $230\text{K}$ 3D object annotations. An extensive 3D object detection benchmark evaluation on the A*3D dataset for various attributes such as high density, day-time/night-time, gives interesting insights into the advantages and limitations of training and testing 3D object detection in real-world setting.

Datasets used in this paper and their characteristics.

Our main focus is on evaluating the object detection architectures on data which is collected in real scenarios. We provide an overview of the datasets, which we use in our evaluations in Table {{table:1004dd4f-7829-4f59-8c64-a1d813e7da37}} . In Table {{table:1004dd4f-7829-4f59-8c64-a1d813e7da37}}  we see that different popular autonomous driving datasets consists only of scenes with limited variation in weather conditions. For example, the popular KITTI benchmark {{cite:6934640}} consists of us with data captured during clear weather conditions. Further, none of the other popular multi-modal benchmark datasets consists of scenes with snow or fog. Snow and fog can create problems for the object detection architectures. For example, for LiDAR based object detectors they can yield false positive predictions {{cite:4c73ecd}}, {{cite:3699cbc}}. Further, camera based object detectors operating in these adverse weather conditions tend to miss detections {{cite:0454944}}, {{cite:8ad329f}}. Consequently, it is not possible to assess the robustness of state-of-the-art detectors in adverse weather conditions on the KITTI benchmark. {{cite:6934640}}.

We use four datasets as illustrated in Table {{table:1004dd4f-7829-4f59-8c64-a1d813e7da37}} . They consist of 3 different data splits, which include: day, night and rain. We also evaluate the object detection architectures by augmenting fog and rain on the KITTI Dataset. For that purpose we use the open source implementation provided by Halder et al. {{cite:2969718}}. This method augments photo-realistic fog and rain on images for different intensities by using the depth maps provided by KITTI. We create different augmented versions of KITTI by varying the fog visiblity and rain intensity parameters. We name these datasets KITTI-FOG and KITTI-RAIN in our paper. We illustrate an example of our augmented dataset with different visibility parameters in Fig. {{figure:285f9959-d7c3-4efe-9a40-3461e612cf5a}} . We evaluate the degradation of object detectors on these datasets in Section REF .


Temporal Pyramid Network for Action Recognition

Abstract: Visual tempo characterizes the dynamics and the temporal scale of an action. Modeling such visual tempos of different actions facilitates their recognition. Previous works often capture the visual tempo through sampling raw videos at multiple rates and constructing an input-level frame pyramid, which usually requires a costly multi-branch network to handle. In this work we propose a generic Temporal Pyramid Network (TPN) at the feature-level, which can be flexibly integrated into 2D or 3D backbone networks in a plug-and-play manner. Two essential components of TPN, the source of features and the fusion of features, form a feature hierarchy for the backbone so that it can capture action instances at various tempos. TPN also shows consistent improvements over other challenging baselines on several action recognition datasets. Specifically, when equipped with TPN, the 3D ResNet-50 with dense sampling obtains a 2% gain on the validation set of Kinetics-400. A further analysis also reveals that TPN gains most of its improvements on action classes that have large variances in their visual tempos, validating the effectiveness of TPN.

GTA: Global Temporal Attention for Video Action Understanding

Abstract: Self-attention learns pairwise interactions to model long-range dependencies, yielding great improvements for video action recognition. In this paper, we seek a deeper understanding of self-attention for temporal modeling in videos. We first demonstrate that the entangled modeling of spatio-temporal information by flattening all pixels is sub-optimal, failing to capture temporal relationships among frames explicitly. To this end, we introduce Global Temporal Attention (GTA), which performs global temporal attention on top of spatial attention in a decoupled manner. We apply GTA on both pixels and semantically similar regions to capture temporal relationships at different levels of spatial granularity. Unlike conventional self-attention that computes an instance-specific attention matrix, GTA directly learns a global attention matrix that is intended to encode temporal structures that generalize across different samples. We further augment GTA with a cross-channel multi-head fashion to exploit channel interactions for better temporal modeling. Extensive experiments on 2D and 3D networks demonstrate that our approach consistently enhances temporal modeling and provides state-of-the-art performance on three video action recognition datasets.

Video Transformer Network

Abstract: This paper presents VTN, a transformer-based framework for video recognition. Inspired by recent developments in vision transformers, we ditch the standard approach in video action recognition that relies on 3D ConvNets and introduce a method that classifies actions by attending to the entire video sequence information. Our approach is generic and builds on top of any given 2D spatial network. In terms of wall runtime, it trains $16.1\times$ faster and runs $5.1\times$ faster during inference while maintaining competitive accuracy compared to other state-of-the-art methods. It enables whole video analysis, via a single end-to-end pass, while requiring $1.5\times$ fewer GFLOPs. We report competitive results on Kinetics-400 and present an ablation study of VTN properties and the trade-off between accuracy and inference speed. We hope our approach will serve as a new baseline and start a fresh line of research in the video recognition domain. Code and models are available at: https://github.com/bomri/SlowFast/blob/master/projects/vtn/README.md

Video action recognition models for extracting semantic features

For the feature extraction stage, Table {{table:98037c43-69a4-4fda-b11c-5464f45a6643}}  shows all the action recognition models we use with their main configurations.
These models are pretrained from various large-scale datasets, including Kinetics-400 (K400){{cite:b38003a}}, IG65M {{cite:f636d45}}, and Omnisource{{cite:bd065a7}}. All models are fine-tuned on SoccerNet-v2 snippets to reach a reasonable top-1 classification accuracy between {{formula:493ea839-44c0-4fa8-82d2-bdcc292309ce}}  and {{formula:252c53c3-d803-4b71-a3cb-30b0ea03fa92}}  (1-view test) on the test set. At test time, all models slide on the videos and produce features at 1fps.


Quantum Annealer; Universal Gate-Based Quantum Device

Polynomial and superpolynomial (e.g. Boltzmann machine)

Polynomial, problem specific, varies by measure (e.g. see {{cite:3c84b69}}, {{cite:21e92c0}})

On the construction of model Hamiltonians for adiabatic quantum computation and its application to finding low energy conformations of lattice protein models

Abstract: In this report, we explore the use of a quantum optimization algorithm for obtaining low energy conformations of protein models. We discuss mappings between protein models and optimization variables, which are in turn mapped to a system of coupled quantum bits. General strategies are given for constructing Hamiltonians to be used to solve optimization problems of physical/chemical/biological interest via quantum computation by adiabatic evolution. As an example, we implement the Hamiltonian corresponding to the Hydrophobic-Polar (HP) model for protein folding. Furthermore, we present an approach to reduce the resulting Hamiltonian to two-body terms gearing towards an experimental realization.

Genome assembly using quantum and quantum-inspired annealing

Abstract: Recent advances in DNA sequencing open prospects to make whole-genome analysis rapid and reliable, which is promising for various applications including personalized medicine. However, existing techniques for {\it de novo} genome assembly, which is used for the analysis of genomic rearrangements, chromosome phasing, and reconstructing genomes without a reference, require solving tasks of high computational complexity. Here we demonstrate a method for solving genome assembly tasks with the use of quantum and quantum-inspired optimization techniques. Within this method, we present experimental results on genome assembly using quantum annealers both for simulated data and the $\phi$X 174 bacteriophage. Our results pave a way for an increase in the efficiency of solving bioinformatics problems with the use of quantum computing and, in particular, quantum annealing. We expect that the new generation of quantum annealing devices would outperform existing techniques for {\it de novo} genome assembly. To the best of our knowledge, this is the first experimental study of de novo genome assembly problems both for real and synthetic data on quantum annealing devices and quantum-inspired techniques.

Fast quantum search algorithms in protein sequence comparisons: quantum bioinformatics.

Abstract: Quantum search algorithms are considered in the context of protein sequence comparison in bioinformatics. Given a sample protein sequence of length m (i.e., m residues), the problem considered is to find an optimal match in a large database containing N residues. Initially, Grover's quantum search algorithm is applied to a simple illustrative case-namely, where the database forms a complete set of states over the 2(m) basis states of a m qubit register, and thus is known to contain the exact sequence of interest. This example demonstrates explicitly the typical O(square root of [N]) speedup on the classical O(N) requirements. An algorithm is then presented for the (more realistic) case where the database may contain repeat sequences, and may not necessarily contain an exact match to the sample sequence. In terms of minimizing the Hamming distance between the sample sequence and the database subsequences the algorithm finds an optimal alignment, in O(square root of [N]) steps, by employing an extension of Grover's algorithm, due to Boyer et al. for the case when the number of matches is not a priori known.

Quantum pattern matching fast on average

Abstract: The $d$-dimensional pattern matching problem is to find an occurrence of a pattern of length $m \times \dots \times m$ within a text of length $n \times \dots \times n$, with $n \ge m$. This task models various problems in text and image processing, among other application areas. This work describes a quantum algorithm which solves the pattern matching problem for random patterns and texts in time $\widetilde{O}((n/m)^{d/2} 2^{O(d^{3/2}\sqrt{\log m})})$. For large $m$ this is super-polynomially faster than the best possible classical algorithm, which requires time $\widetilde{\Omega}( (n/m)^d + n^{d/2} )$. The algorithm is based on the use of a quantum subroutine for finding hidden shifts in $d$ dimensions, which is a variant of algorithms proposed by Kuperberg.

Quantum Simulation of Phylogenetic Trees

Abstract: Quantum simulations constructing probability tensors of biological multi-taxa in phylogenetic trees are proposed, in terms of positive trace preserving maps, describing evolving systems of quantum walks with multiple walkers. Basic phylogenetic models applying on trees of various topologies are simulated following appropriate decoherent quantum circuits. Quantum simulations of statistical inference for aligned sequences of biological characters are provided in terms of a quantum pruning map operating on likelihood operator observables, utilizing state-observable duality and measurement theory.

Quantum annealing versus classical machine learning applied to a simplified computational biology problem

Abstract: Transcription factors regulate gene expression, but how these proteins recognize and specifically bind to their DNA targets is still debated. Machine learning models are effective means to reveal interaction mechanisms. Here we studied the ability of a quantum machine learning approach to predict binding specificity. Using simplified datasets of a small number of DNA sequences derived from actual binding affinity experiments, we trained a commercially available quantum annealer to classify and rank transcription factor binding. The results were compared to state-of-the-art classical approaches for the same simplified datasets, including simulated annealing, simulated quantum annealing, multiple linear regression, LASSO, and extreme gradient boosting. Despite technological limitations, we find a slight advantage in classification performance and nearly equal ranking performance using the quantum annealer for these fairly small training data sets. Thus, we propose that quantum annealing might be an effective method to implement machine learning for certain computational biology problems.

The quest for a Quantum Neural Network

Abstract: With the overwhelming success in the field of quantum information in the last decades, the "quest" for a Quantum Neural Network (QNN) model began in order to combine quantum computing with the striking properties of neural computing. This article presents a systematic approach to QNN research, which so far consists of a conglomeration of ideas and proposals. It outlines the challenge of combining the nonlinear, dissipative dynamics of neural computing and the linear, unitary dynamics of quantum computing. It establishes requirements for a meaningful QNN and reviews existing literature against these requirements. It is found that none of the proposals for a potential QNN model fully exploits both the advantages of quantum physics and computing in neural networks. An outlook on possible ways forward is given, emphasizing the idea of Open Quantum Neural Networks based on dissipative quantum computing.

Selected quantum proof of principles relevant to biology and medicine. A variety of quantum approaches to addressing computational problems in biology and medicine exist, some of which have been experimentally demonstrated. Expected empirical advantages vary greatly. Among the most promising near term applications are ones that leverage quantum simulation and quantum machine learning techniques, such as quantum neural networks.

Future prospects in biology and medicine

In this section we describe a broad variety of potential applications for quantum algorithms. Our aim is to highlight the breadth of both existing quantum algorithms and the types of problems in biology and medicine that they may address. We leverage the quantum advantage framework described in Section  and note when a specific application may admit an empirical quantum advantage in the near- or medium-term (summarized in Table  {{table:fddec06c-95a5-42be-96a5-61450b09f923}}). While some of the applications described are not expected to be feasible in the near term – indeed, in some cases, even an FTQC may not be the most appropriate tool for the target problem – it is our intent for the breadth of potential applications and research directions covered to be valuable to an interdisciplinary audience. As such, when possible, we have sought to provide i) quantum scientists with relevant details and references to develop targeted quantum algorithms for applications in biology and medicine and ii) domain computationalists with information on quantum algorithms relevant to applications in the biology and medicine and their prospects for operational quantum advantages in the near- or medium-term.

Prospects for bioinformatics. A small number of quantum algorithms for problems in bioinformatics have been proposed (Table  {{table:fddec06c-95a5-42be-96a5-61450b09f923}}). These include theoretical algorithms developed for FTQC devices that target NP-hard problems, such as sequence alignment {{cite:d59e885}}, {{cite:bced555}}, {{cite:175227c}} and the inference of phylogenetic trees {{cite:27a5502}}, which leverage amplitude amplification and quantum walks {{cite:09ae5f8}}. To be made practical, these theoretical quantum algorithms are expected to require both significant refinement and effort in translation. In the near term, these refinements could include i) recasting them for NISQ devices using the VQA, QAOA, or QA frameworks and ii) integrating greater biological context. Already, examples of this type of work exist for de novo assembly {{cite:d377063}}, {{cite:20a58ff}}, sequence alignment {{cite:70d598f}}, and the inference of biological networks {{cite:a3ead65}}, {{cite:4580df6}}. Over the long term, operational advantages may be pursued by optimizing near term approaches and integrating fast quantum algorithm subroutines where possible. Known quantum algorithms that may be relevant to this work include ones for backtracking {{cite:b24b220}}, dynamic programming {{cite:8e46ab1}}, {{cite:2093d26}}, operating on strings {{cite:53f6430}}, {{cite:f4716ae}}, {{cite:34eccb5}}, and differential equations {{cite:5493987}}, {{cite:e8f1deb}}, {{cite:6b1e28a}}, {{cite:a9051f4}}, {{cite:ea32d74}}.


Randomly appends bytes to malware sample.

Appends a byte sequence generated by an RNN model trained on benign files

Exploring Adversarial Examples in Malware Detection

Abstract: The convolutional neural network (CNN) architecture is increasingly being applied to new domains, such as malware detection, where it is able to learn malicious behavior from raw bytes extracted from executables. These architectures reach impressive performance with no feature engineering effort involved, but their robustness against active attackers is yet to be understood. Such malware detectors could face a new attack vector in the form of adversarial interference with the classification model. Existing evasion attacks intended to cause misclassification on test-time instances, which have been extensively studied for image classifiers, are not applicable because of the input semantics that prevents arbitrary changes to the binaries. This paper explores the area of adversarial examples for malware detection. By training an existing model on a production-scale dataset, we show that some previous attacks are less effective than initially reported, while simultaneously highlighting architectural weaknesses that facilitate new attack strategies for malware classification. Finally, we explore how generalizable different attack strategies are, the trade-offs when aiming to increase their effectiveness, and the transferability of single-step attacks.

Binary Black-box Evasion Attacks Against Deep Learning-based Static Malware Detectors with Adversarial Byte-Level Language Model

Abstract: Anti-malware engines are the first line of defense against malicious software. While widely used, feature engineering-based anti-malware engines are vulnerable to unseen (zero-day) attacks. Recently, deep learning-based static anti-malware detectors have achieved success in identifying unseen attacks without requiring feature engineering and dynamic analysis. However, these detectors are susceptible to malware variants with slight perturbations, known as adversarial examples. Generating effective adversarial examples is useful to reveal the vulnerabilities of such systems. Current methods for launching such attacks require accessing either the specifications of the targeted anti-malware model, the confidence score of the anti-malware response, or dynamic malware analysis, which are either unrealistic or expensive. We propose MalRNN, a novel deep learning-based approach to automatically generate evasive malware variants without any of these restrictions. Our approach features an adversarial example generation process, which learns a language model via a generative sequence-to-sequence recurrent neural network to augment malware binaries. MalRNN effectively evades three recent deep learning-based malware detectors and outperforms current benchmark methods. Findings from applying our MalRNN on a real dataset with eight malware categories are discussed.

Overview of Benchmark Experiment Methods

where {{formula:864624cb-8e1f-4c6c-88d3-04efddec8b5b}}  and {{formula:af91d6bc-233b-40c9-8019-8e7152b11362}}  denote the sets of evasive and functional modified malware samples generated from the AMG method, respectively. {{formula:c487cc4c-aa07-495e-b066-945ff5f3e8cf}}  represents the total number of malware samples given as input to the AMG method. To evaluate MalGPT's performance, we conducted several benchmark experiments under the constraints of our threat model (i.e., single-shot evasion, black-box setting, and 10 KB maximum
append size). Table {{table:ab33e233-d8ef-4ba5-b6a1-4fb63dbd70b6}}  presents the description for each selected benchmark method.


Produce mutants syntactically similar to real faults

Derive syntactic patterns from bug-fixes

Deriving common fault syntactic patterns

DeepMutation: A Neural Mutation Tool

Abstract: Mutation testing can be used to assess the fault-detection capabilities of a given test suite. To this aim, two characteristics of mutation testing frameworks are of paramount importance: (i) they should generate mutants that are representative of real faults; and (ii) they should provide a complete tool chain able to automatically generate, inject, and test the mutants. To address the first point, we recently proposed an approach using a Recurrent Neural Network Encoder-Decoder architecture to learn mutants from ~787k faults mined from real programs. The empirical evaluation of this approach confirmed its ability to generate mutants representative of real faults. In this paper, we address the second point, presenting DEEPMUTATION, a tool wrapping our deep learning model into a fully automated tool chain able to generate, inject, and test mutants learned from real faults. Video: https://sites.google.com/view/learning-mutation/deepmutation

Learning How to Mutate Source Code from Bug-Fixes

Abstract: Mutation testing has been widely accepted as an approach to guide test case generation or to assess the effectiveness of test suites. Empirical studies have shown that mutants are representative of real faults; yet they also indicated a clear need for better, possibly customized, mutation operators and strategies. While methods to devise domain-specific or general-purpose mutation operators from real faults exist, they are effort-and error-prone, and do not help the tester to decide whether and how to mutate a given source code element. We propose a novel approach to automatically learn mutants from faults in real programs. First, our approach processes bug fixing changes using fine-grained differencing, code abstraction, and change clustering. Then, it learns mutation models using a deep learning strategy. We have trained and evaluated our technique on a set of ~787k bug fixes mined from GitHub. Our empirical evaluation showed that our models are able to predict mutants that resemble the actual fixed bugs in between 9% and 45% of the cases, and over 98% of the automatically generated mutants are lexically and syntactically correct.

What It Would Take to Use Mutation Testing in Industry—A Study at Facebook

Abstract: Traditionally, mutation testing generates an abundance of small deviations of a program, called mutants. At industrial systems the scale and size of Facebook's, doing this is infeasible. We should not create mutants that the test suite would likely fail on or that give no actionable signal to developers. To tackle this problem, in this paper, we semi-automatically learn error-inducing patterns from a corpus of common Java coding errors and from changes that caused operational anomalies at Facebook specifically. We combine the mutations with instrumentation that measures which tests exactly visited the mutated piece of code. Results on more than 15,000 generated mutants show that more than half of the generated mutants survive Facebook's rigorous test suite of unit, integration, and system tests. Moreover, in a case study with 26 developers, all but two expressed that the mutation exposed a lack of testing in principle. As such, almost half of the 26 would actually act on the mutant presented to them by adapting an existing or creating a new test. The others did not for a variety of reasons often outside the scope of mutation testing. It remains a practical challenge how we can include such external information to increase the true actionability rate on mutants.

Fault Mimicking Techniques

Table {{table:c41d749e-b0a6-4253-8ce5-14dea47faa50}}  lists a set of recent fault mimicking techniques that aim at generating (syntactically) realistic faults. By inspecting the table it becomes evident that the key objective of these techniques is to improve the realism of fault-seeding, which is defined and evaluated by some form of syntactic distance (number of tokens changed, Bleu score, etc.) from real faults. While such approaches may indeed succeed to generate some exact matches of real faults, they have never been evaluated with respect to their semantics and particularly their utility as fault-based testing techniques {{cite:c7f2d20}}, in a sense the typical use case of mutation testing {{cite:2f63a93}}, {{cite:c7f2d20}}.

Syntactic and Semantic similarity of Artificial and Real Faults

To evaluate fault seeding two types of metrics are used, the syntactic and semantic similarity. Table {{table:c41d749e-b0a6-4253-8ce5-14dea47faa50}}  lists studies using syntactic similarity, while some form of semantic similarity has been used by several studies {{cite:99eccc6}}, {{cite:27400e5}}, {{cite:9c9dbfd}}, {{cite:e28d341}}. Intuitively, syntactic similarity refers to the distance between the text representations of the mutant and the real faulty code, while semantic similarity to the program behavior similarities, between the seeded and the real fault.

The answer to this question will also provide evidence on the use of syntactic distance metrics in evaluating fault seeding methods, i.e., whether seeded faults with small syntactic distance from the real ones are indeed semantically close to them (at least closer than those that are not syntactically similar, i.e., dissimilar). As we discussed before, such evaluation metrics are followed by recent research (Table {{table:c41d749e-b0a6-4253-8ce5-14dea47faa50}} ) without any empirical evidence of their validity. This means that we want to check whether the choice of syntactic distance as an evaluation metric, as performed by previous studies in Table {{table:c41d749e-b0a6-4253-8ce5-14dea47faa50}} , is a valid choice.


FConn512, BatchNorm, FConn10, BatchNorm, SoftMax

ResBlk256, 3, 2, ResBlk256, 3, 2, ResBlk512, 3, 2, ResBlk512, 3, 2, FConn10, SoftMax

FINN: A Framework for Fast, Scalable Binarized Neural Network Inference

Abstract: Research has shown that convolutional neural networks contain significant redundancy, and high classification accuracy can be obtained even when weights and activations are reduced from floating point to binary values. In this paper, we present FINN, a framework for building fast and flexible FPGA accelerators using a flexible heterogeneous streaming architecture. By utilizing a novel set of optimizations that enable efficient mapping of binarized neural networks to hardware, we implement fully connected, convolutional and pooling layers, with per-layer compute resources being tailored to user-provided throughput requirements. On a ZC706 embedded FPGA platform drawing less than 25 W total system power, we demonstrate up to 12.3 million image classifications per second with 0.31 μs latency on the MNIST dataset with 95.8% accuracy, and 21906 image classifications per second with 283 μs latency on the CIFAR-10 and SVHN datasets with respectively 80.1% and 94.9% accuracy. To the best of our knowledge, ours are the fastest classification rates reported to date on these benchmarks.

Bi-Real Net: Enhancing the Performance of 1-bit CNNs With Improved Representational Capability and Advanced Training Algorithm

Abstract: In this work, we study the 1-bit convolutional neural networks (CNNs), of which both the weights and activations are binary. While being efficient, the classification accuracy of the current 1-bit CNNs is much worse compared to their counterpart real-valued CNN models on the large-scale dataset, like ImageNet. To minimize the performance gap between the 1-bit and real-valued CNN models, we propose a novel model, dubbed Bi-Real net, which connects the real activations (after the 1-bit convolution and/or BatchNorm layer, before the sign function) to activations of the consecutive block, through an identity shortcut. Consequently, compared to the standard 1-bit CNN, the representational capability of the Bi-Real net is significantly enhanced and the additional cost on computation is negligible. Moreover, we develop a specific training algorithm including three technical novelties for 1- bit CNNs. Firstly, we derive a tight approximation to the derivative of the non-differentiable sign function with respect to activation. Secondly, we propose a magnitude-aware gradient with respect to the weight for updating the weight parameters. Thirdly, we pre-train the real-valued CNN model with a clip function, rather than the ReLU function, to better initialize the Bi-Real net. Experiments on ImageNet show that the Bi-Real net with the proposed training algorithm achieves 56.4% and 62.2% top-1 accuracy with 18 layers and 34 layers, respectively. Compared to the state-of-the-arts (e.g., XNOR Net), Bi-Real net achieves up to 10% higher top-1 accuracy with more memory saving and lower computational cost. Keywords: binary neural network, 1-bit CNNs, 1-layer-per-block

Network architectures for evaluated benchmarks.Conv{{formula:d1c09764-8c63-4d58-9072-a98621cd05ee}} denotes a convolutional layer with {{formula:baf0432c-da24-4de9-9323-7a4ba8dae23c}}  outputs, kernel size {{formula:8e801c39-4af4-4c3d-ad90-4c2e670fe94c}}  and stride {{formula:d394efe4-db96-42e9-ab88-6f307f484e3c}} .FConn{{formula:582d6399-410a-409e-8d23-f9bcd25530fd}} is a fully connected layer with {{formula:db84aa4d-f7d3-44dd-9c3d-8934f50b0f12}}  outputs.MaxPool{{formula:34323ad6-3158-4531-8e4b-1e1bfe6117f3}} is an {{formula:a6c56d54-78b8-418c-911a-11416cd49cf6}}  maximum-pooling layer with stride {{formula:ea6befdd-bc7b-45e1-bbd2-536864ef9677}} , and BatchNorm and SoftMax are batch normalization and normalized exponential layers, respectively.ResBlk{{formula:b6214f87-903a-4571-8a95-71b5afafe8ce}} denotes a residual block with two Conv{{formula:59509007-de8f-4020-a22e-011ebd58e7ec}} layers, each followed by a BatchNorm.Layers in bold were unrolled and targeted for logic expansion (and shrinkage).For ImageNet, the residual block in bold had its first convolutional layer unrolled and targeted.

We evaluated our approach using the DNN model and dataset combinations detailed in Table {{table:f8e838d2-3eea-4bc7-8105-7e4119544ac3}} .
Hardware implementations for all datasets other than ImageNet targeted the Xilinx Kintex UltraScale XCKU115.
For ImageNet, we targeted the largest FPGA available to us: the Virtex UltraScale+ XCVU9P.
All implementations met timing at 200 MHz.
Our primary comparison point was LUTNet, trained as we described in its original publication {{cite:6b40a9b}}.
Where possible, we also maintained the BNN baseline, ReBNet {{cite:8ee7947}}, used as the starting point for LUTNet's logic expansion, and considered its test accuracy to be a performance floor.

For fairness of comparison to vanilla LUTNet (and ReBNet), we used identical experimental settings to those employed for its evaluation with MNIST, SVHN and CIFAR-10 {{cite:6b40a9b}}.
Implementations for these datasets included all layers: those selected for logic expansion (and subsequent shrinkage) were unrolled, with the remainder left identical to the BNN starting point.
For ImageNet, our design encompassed the target layer only due to the complexity of implementing the remaining layers.
In all cases, layers selected for logic expansion and shrinkage are marked in bold in Table {{table:f8e838d2-3eea-4bc7-8105-7e4119544ac3}} .


FEVER: a Large-scale Dataset for Fact Extraction and VERification

Abstract: In this paper we introduce a new publicly available dataset for verification against textual sources, FEVER: Fact Extraction and VERification. It consists of 185,445 claims generated by altering sentences extracted from Wikipedia and subsequently verified without knowledge of the sentence they were derived from. The claims are classified as Supported, Refuted or NotEnoughInfo by annotators achieving 0.6841 in Fleiss kappa. For the first two classes, the annotators also recorded the sentence(s) forming the necessary evidence for their judgment. To characterize the challenge of the dataset presented, we develop a pipeline approach and compare it to suitably designed oracles. The best accuracy we achieve on labeling a claim accompanied by the correct evidence is 31.87%, while if we ignore the evidence we achieve 50.91%. Thus we believe that FEVER is a challenging testbed that will help stimulate progress on claim verification against textual sources.

Zero-Shot Relation Extraction via Reading Comprehension

Abstract: We show that relation extraction can be reduced to answering simple reading comprehension questions, by associating one or more natural-language questions with each relation slot. This reduction has several advantages: we can (1) learn relation-extraction models by extending recent neural reading-comprehension techniques, (2) build very large training sets for those models by combining relation-specific crowd-sourced questions with distant supervision, and even (3) do zero-shot learning by extracting new relation types that are only specified at test-time, for which we have no labeled training examples. Experiments on a Wikipedia slot-filling task demonstrate that the approach can generalize to new questions for known relation types with high accuracy, and that zero-shot generalization to unseen relation types is possible, at lower accuracy levels, setting the bar for future work on this task.

HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering

Abstract: Existing question answering (QA) datasets fail to train QA systems to perform complex reasoning and provide explanations for answers. We introduce HotpotQA, a new dataset with 113k Wikipedia-based question-answer pairs with four key features: (1) the questions require finding and reasoning over multiple supporting documents to answer; (2) the questions are diverse and not constrained to any pre-existing knowledge bases or knowledge schemas; (3) we provide sentence-level supporting facts required for reasoning, allowing QA systems to reason with strong supervision and explain the predictions; (4) we offer a new type of factoid comparison questions to test QA systems' ability to extract relevant facts and perform necessary comparison. We show that HotpotQA is challenging for the latest QA systems, and the supporting facts enable models to improve performance and make explainable predictions.

TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension

Abstract: We present TriviaQA, a challenging reading comprehension dataset containing over 650K question-answer-evidence triples. TriviaQA includes 95K question-answer pairs authored by trivia enthusiasts and independently gathered evidence documents, six per question on average, that provide high quality distant supervision for answering the questions. We show that, in comparison to other recently introduced large-scale datasets, TriviaQA (1) has relatively complex, compositional questions, (2) has considerable syntactic and lexical variability between questions and corresponding answer-evidence sentences, and (3) requires more cross sentence reasoning to find answers. We also present two baseline algorithms: a feature-based classifier and a state-of-the-art neural network, that performs well on SQuAD reading comprehension. Neither approach comes close to human performance (23% and 40% vs. 80%), suggesting that TriviaQA is a challenging testbed that is worth significant future study. Data and code available at -- http://nlp.cs.washington.edu/triviaqa/

ELI5: Long Form Question Answering

Abstract: We introduce the first large-scale corpus for long form question answering, a task requiring elaborate and in-depth answers to open-ended questions. The dataset comprises 270K threads from the Reddit forum “Explain Like I’m Five” (ELI5) where an online community provides answers to questions which are comprehensible by five year olds. Compared to existing datasets, ELI5 comprises diverse questions requiring multi-sentence answers. We provide a large set of web documents to help answer the question. Automatic and human evaluations show that an abstractive model trained with a multi-task objective outperforms conventional Seq2Seq, language modeling, as well as a strong extractive baseline.However, our best model is still far from human performance since raters prefer gold responses in over 86% of cases, leaving ample opportunity for future improvement.

Wizard of Wikipedia: Knowledge-Powered Conversational agents

Abstract: In open-domain dialogue intelligent agents should exhibit the use of knowledge, however there are few convincing demonstrations of this to date. The most popular sequence to sequence models typically "generate and hope" generic utterances that can be memorized in the weights of the model when mapping from input utterance(s) to output, rather than employing recalled knowledge as context. Use of knowledge has so far proved difficult, in part because of the lack of a supervised learning benchmark task which exhibits knowledgeable open dialogue with clear grounding. To that end we collect and release a large dataset with conversations directly grounded with knowledge retrieved from Wikipedia. We then design architectures capable of retrieving knowledge, reading and conditioning on it, and finally generating natural responses. Our best performing dialogue models are able to conduct knowledgeable discussions on open-domain topics as evaluated by automatic metrics and human evaluations, while our new benchmark allows for measuring further improvements in this important research direction.

KILT tasks we consider.

We use the KILT benchmark {{cite:5a03e83}} as an evaluation suite for our work. KILT consists of 11 knowledge-intensive tasks, grouped into 5 categories: fact checking, entity linking, slot filling, open domain question answering and dialog. Given that entity linking tasks are intrinsically tied to the underlying Wikipedia snapshot, as entity labels come from Wikipedia titles of respective entities, we choose to omit those in our experiments. See Table {{table:e1b52f5f-a217-4122-84d6-475e53108b36}}  for the full list of datasets we consider and the shortcuts we use to refer to them.


our proposed method but with a limit to the treewidth of the
decomposable model of n

Estimation of KL Divergence: Optimal Minimax Rate

Abstract: The problem of estimating the Kullback-Leibler divergence $D(P\|Q)$ between two unknown distributions $P$ and $Q$ is studied, under the assumption that the alphabet size $k$ of the distributions can scale to infinity. The estimation is based on $m$ independent samples drawn from $P$ and $n$ independent samples drawn from $Q$. It is first shown that there does not exist any consistent estimator that guarantees asymptotically small worst-case quadratic risk over the set of all pairs of distributions. A restricted set that contains pairs of distributions, with density ratio bounded by a function $f(k)$ is further considered. {An augmented plug-in estimator is proposed, and its worst-case quadratic risk is shown to be within a constant factor of $(\frac{k}{m}+\frac{kf(k)}{n})^2+\frac{\log ^2 f(k)}{m}+\frac{f(k)}{n}$, if $m$ and $n$ exceed a constant factor of $k$ and $kf(k)$, respectively.} Moreover, the minimax quadratic risk is characterized to be within a constant factor of $(\frac{k}{m\log k}+\frac{kf(k)}{n\log k})^2+\frac{\log ^2 f(k)}{m}+\frac{f(k)}{n}$, if $m$ and $n$ exceed a constant factor of $k/\log(k)$ and $kf(k)/\log k$, respectively. The lower bound on the minimax quadratic risk is characterized by employing a generalized Le Cam's method. A minimax optimal estimator is then constructed by employing both the polynomial approximation and the plug-in approaches.

Computing Divergences between Discrete Decomposable Models

Abstract: There are many applications that benefit from computing the exact divergence between 2 discrete probability measures, including machine learning. Unfortunately, in the absence of any assumptions on the structure or independencies within these distributions, computing the divergence between them is an intractable problem in high dimensions. We show that we are able to compute a wide family of functionals and divergences, such as the alpha-beta divergence, between two decomposable models, i.e. chordal Markov networks, in time exponential to the treewidth of these models. The alpha-beta divergence is a family of divergences that include popular divergences such as the Kullback-Leibler divergence, the Hellinger distance, and the chi-squared divergence. Thus, we can accurately compute the exact values of any of this broad class of divergences to the extent to which we can accurately model the two distributions using decomposable models.

Divergence estimation methods that have existingimplementations that we will compare against.An implementation of our proposed approachand everything else needed to reproduce the experiments in thispapercan be found online athttps://gitlab.com/lklee/divergence-estimation-via-graphical-models.

However, since we will compare method against the
other methods in Table {{table:888b2b55-07b8-4dc0-aa7b-27ef912fd045}}  at estimating the
Symmetric KL Divergence, {{formula:f630f608-3bcd-46cb-a64a-7406e8c4b80d}} , we
will need the true Symmetric KL Divergence between the 2
population distributions of each dataset pair.
Therefore, during the data generation process, we measure and record
the exact Symmetric KL Divergence between the decomposable models from
which the dataset pairs are generated from.
We also measure the exact Hellinger Distance between the
decomposable models during generation as well for use in
Section REF .

In this section we will compare the method proposed in this
paper with various existing methods for
estimating the divergence between 2 distributions using
sample data.
Specifically we will compare method against the methods
in Table {{table:888b2b55-07b8-4dc0-aa7b-27ef912fd045}} , all of which has been described
in Section , in estimating the
symmetric Kullback-Leibler Divergence
between datasets in each dataset pair described in
Section REF .
We use the KL Divergence in this comparison due to some of the
methods in Table {{table:888b2b55-07b8-4dc0-aa7b-27ef912fd045}}  only having
implementations that estimate the KL Divergence.
We also use the symmetric version of the KL Divergence to
prevent any confusion on which of the 2 distributions is {{formula:4e3bea14-611b-4e53-90dc-003e0b420ee5}}
and which is {{formula:a39921b4-da8c-4113-b5af-267bb0383e59}} .

The results of the comparison between method and the
methods in Table {{table:888b2b55-07b8-4dc0-aa7b-27ef912fd045}}  can be found in
Figure {{figure:33790952-7669-4e9b-8cd5-5219d28457ba}} .
These results were obtained using all of the data available
in each dataset, that is to say each method is given 100,000
samples from each of the 2 distributions to estimate the
symmetric KL-divergence between them.
Furthermore, there are 2 plots in
Figure {{figure:33790952-7669-4e9b-8cd5-5219d28457ba}} , each representing results with
generator of different treewidths, specifically
treewidths of size 5 and 7.

On the surface, we can observe from Figure {{figure:33790952-7669-4e9b-8cd5-5219d28457ba}}  that
the method zg seems to severely underestimate the symmetric
KL divergence between the distributions, giving estimates close to 0, even
for datasets containing 10 binary variables.
On the other hand, bzlv gives estimates that outperforms
zg but still clearly underestimates the symmetric KL
divergence at 10 binary variables.
Out of all the existing methods in Table {{table:888b2b55-07b8-4dc0-aa7b-27ef912fd045}} ,
hjw performs the best, and also does not clearly
underestimate the KL divergence when the number of binary variables
is 10.
As the number of binary variables increases, these 3 existing methods
start to increasingly underestimate the symmetric KL divergence.
Specifically, these methods start returning estimates close of 0
when the number of binary variables is 30.


Multispectral Image Feature Points

Abstract: This paper presents a novel feature point descriptor for the multispectral image case Far-Infrared and Visible Spectrum images. It allows matching interest points on images of the same scene but acquired in different spectral bands. Initially, points of interest are detected on both images through a SIFT-like based scale space representation. Then, these points are characterized using an Edge Oriented Histogram (EOH) descriptor. Finally, points of interest from multispectral images are matched by finding nearest couples using the information from the descriptor. The provided experimental results and comparisons with similar methods show both the validity of the proposed approach as well as the improvements it offers with respect to the current state-of-the-art.

Two-View Matching with View Synthesis Revisited

Abstract: Wide-baseline matching focussing on problems with extreme viewpoint change is considered. We introduce the use of view synthesis with affine-covariant detectors to solve such problems and show that matching with the Hessian-Affine or MSER detectors outperforms the state-of-the-art ASIFT. To minimise the loss of speed caused by view synthesis, we propose the Matching On Demand with view Synthesis algorithm (MODS) that uses progressively more synthesized images and more (time-consuming) detectors until reliable estimation of geometry is possible. We show experimentally that the MODS algorithm solves problems beyond the state-of-the-art and yet is comparable in speed to standard wide-baseline matchers on simpler problems. Minor contributions include an improved method for tentative correspondence selection, applicable both with and without view synthesis and a view synthesis setup greatly improving MSER robustness to blur and scale change that increase its running time by 10% only.

Evaluation Datasets

Datasets with other than geometric changes

Despite being designed for (extreme) wide baseline stereo problems, MODS performance was evaluated on other datasets: GDB-ICP {{cite:7876ab5}} (modality, viewpoint and photometry changes), SymBench {{cite:6a5380f}} (photometrical changes and photo-vs-painting pairs), and MMS {{cite:c543af9}} (infrared-vs-visible pairs) – see Table {{table:1fbcbceb-9a72-4adb-9dcc-9bb1655eb165}} . The state-of-the-art matcher – ASIFT {{cite:6977b49}} and registration algorithm DualBootstrap {{cite:dda94ad}} as well, as the results of MSER, ORB and Hessian-Affine matchers without view synthesis were compared to MODS.