Style Transfer from Non-Parallel Text by Cross-Alignment

Abstract: This paper focuses on style transfer on the basis of non-parallel text. This is an instance of a broad family of problems including machine translation, decipherment, and sentiment modification. The key challenge is to separate the content from other aspects such as style. We assume a shared latent content distribution across different text corpora, and propose a method that leverages refined alignment of latent representations to perform style transfer. The transferred sentences from one style should match example sentences from the other style as a population. We demonstrate the effectiveness of this cross-alignment method on three tasks: sentiment modification, decipherment of word substitution ciphers, and recovery of word order.

Style Transformer: Unpaired Text Style Transfer without Disentangled Latent Representation

Abstract: Disentangling the content and style in the latent space is prevalent in unpaired text style transfer. However, two major issues exist in most of the current neural models. 1) It is difficult to completely strip the style information from the semantics for a sentence. 2) The recurrent neural network (RNN) based encoder and decoder, mediated by the latent representation, cannot well deal with the issue of the long-term dependency, resulting in poor preservation of non-stylistic semantic content. In this paper, we propose the Style Transformer, which makes no assumption about the latent representation of source sentence and equips the power of attention mechanism in Transformer to achieve better style transfer and better content preservation.

Ups and Downs: Modeling the Visual Evolution of Fashion Trends with One-Class Collaborative Filtering

Abstract: Building a successful recommender system depends on understanding both the dimensions of people's preferences as well as their dynamics. In certain domains, such as fashion, modeling such preferences can be incredibly difficult, due to the need to simultaneously model the visual appearance of products as well as their evolution over time. The subtle semantics and non-linear dynamics of fashion evolution raise unique challenges especially considering the sparsity and large scale of the underlying datasets. In this paper we build novel models for the One-Class Collaborative Filtering setting, where our goal is to estimate users' fashion-aware personalized ranking functions based on their past feedback. To uncover the complex and evolving visual factors that people consider when evaluating products, our method combines high-level visual features extracted from a deep convolutional neural network, users' past feedback, as well as evolving trends within the community. Experimentally we evaluate our method on two large real-world datasets from Amazon.com, where we show it to outperform state-of-the-art personalized ranking measures, and also use it to visualize the high-level fashion trends across the 11-year span of our dataset.

Dear Sir or Madam, May I introduce the GYAFC Dataset: Corpus, Benchmarks and Metrics for Formality Style Transfer

Abstract: Style transfer is the task of automatically transforming a piece of text in one particular style into another. A major barrier to progress in this field has been a lack of training and evaluation datasets, as well as benchmarks and automatic metrics. In this work, we create the largest corpus for a particular stylistic transfer (formality) and show that techniques from the machine translation community can serve as strong baselines for future work. We also discuss challenges of using automatic metrics.

Style Transfer in Text: Exploration and Evaluation

Abstract: Style transfer is an important problem in natural language processing (NLP). However, the progress in language style transfer is lagged behind other domains, such as computer vision, mainly because of the lack of parallel data and principle evaluation metrics. In this paper, we propose to learn style transfer with non-parallel data. We explore two models to achieve this goal, and the key idea behind the proposed models is to learn separate content representations and style representations using adversarial networks. We also propose novel evaluation metrics which measure two aspects of style transfer: transfer strength and content preservation. We access our models and the evaluation metrics on two tasks: paper-news title transfer, and positive-negative review transfer. Results show that the proposed content preservation metric is highly correlate to human judgments, and the proposed models are able to generate sentences with higher style transfer strength and similar content preservation score comparing to auto-encoder.

Delete, Retrieve, Generate: A Simple Approach to Sentiment and Style Transfer

Abstract: We consider the task of text attribute transfer: transforming a sentence to alter a specific attribute (e.g., sentiment) while preserving its attribute-independent content (e.g., changing "screen is just the right size" to "screen is too small"). Our training data includes only sentences labeled with their attribute (e.g., positive or negative), but not pairs of sentences that differ only in their attributes, so we must learn to disentangle attributes from attribute-independent content in an unsupervised way. Previous work using adversarial methods has struggled to produce high-quality outputs. In this paper, we propose simpler methods motivated by the observation that text attributes are often marked by distinctive phrases (e.g., "too small"). Our strongest method extracts content words by deleting phrases associated with the sentence's original attribute value, retrieves new phrases associated with the target attribute, and uses a neural model to fluently combine these into a final output. On human evaluation, our best method generates grammatical and appropriate responses on 22% more inputs than the best previous system, averaged over three attribute transfer datasets: altering sentiment of reviews on Yelp, altering sentiment of reviews on Amazon, and altering image captions to be more romantic or humorous.

Style Transfer Through Back-Translation

Abstract: Style transfer is the task of rephrasing the text to contain specific stylistic properties without changing the intent or affect within the context. This paper introduces a new method for automatic style transfer. We first learn a latent representation of the input sentence which is grounded in a language translation model in order to better preserve the meaning of the sentence while reducing stylistic properties. Then adversarial generation techniques are used to make the output match the desired style. We evaluate this technique on three different style transformations: sentiment, gender and political slant. Compared to two state-of-the-art style transfer modeling techniques we show improvements both in automatic evaluation of style transfer and in manual evaluation of meaning preservation and fluency.

Style Transfer Through Back-Translation

Style Transfer Through Back-Translation

Style Transfer Through Back-Translation

Fighting Offensive Language on Social Media with Unsupervised Text Style Transfer

Abstract: We introduce a new approach to tackle the problem of offensive language in online social media. Our approach uses unsupervised text style transfer to translate offensive sentences into non-offensive ones. We propose a new method for training encoder-decoders using non-parallel data that combines a collaborative classifier, attention and the cycle consistency loss. Experimental results on data from Twitter and Reddit show that our method outperforms a state-of-the-art text style transfer system in two out of three quantitative metrics and produces reliable non-offensive transferred sentences.

Dataset statistics for text style transfer.

Datasets. Table {{table:07aef32a-f118-4465-829c-8cf273322540}}  summarises the datasets used to existing studies to evaluate the performance of TST algorithms. These corpora often contain texts in labeled with two or more attributes. For example, the Yelp dataset contains review text records labeled with binary sentiment class (i.e., positive or negative), and the Caption dataset contain caption-text records labeled with romantic, humorous, and factual classes. Most of these datasets are non-parallel datasets, i.e., there are no matching text pairs in the different attribute classes, except for Shakespeare-Modern and GYAFC. It is also interesting to note that while the GYAFC corpus is a parallel dataset, most of the existing TST studies assume a non-parallel setting when training the TST models with this dataset.


Joint Trajectory and Passive Beamforming Design for Intelligent Reflecting Surface-Aided UAV Communications: A Deep Reinforcement Learning Approach

Abstract: In this paper, the intelligent reflecting surface (IRS)-aided unmanned aerial vehicle (UAV) communication system is studied, where the UAV is deployed to serve the user equipment (UE) with the assistance of multiple IRSs mounted on several buildings to enhance the communication quality between UAV and UE. We aim to maximize the energy efficiency of the system, including the data rate of UE and the energy consumption of UAV via jointly optimizing the UAV's trajectory and the phase shifts of reflecting elements of IRS, when the UE moves and the selection of IRSs is considered for the energy saving purpose. Since the system is complex and the environment is dynamic, it is challenging to derive low-complexity algorithms by using conventional optimization methods. To address this issue, we first propose a deep Q-network (DQN)-based algorithm by discretizing the trajectory, which has the advantage of training time. Furthermore, we propose a deep deterministic policy gradient (DDPG)-based algorithm to tackle the case with continuous trajectory for achieving better performance. The experimental results show that the proposed algorithms achieve considerable performance compared to other traditional solutions.

UAV-Assisted Intelligent Reflecting Surface Symbiotic Radio System

Abstract: This paper investigates a symbiotic unmanned aerial vehicle (UAV)-assisted intelligent reflecting surface (IRS) radio system, where the UAV is leveraged to help the IRS reflect its own signals to the base station, and meanwhile enhance the UAV transmission by passive beamforming at the IRS. First, we consider the weighted sum bit error rate (BER) minimization problem among all IRSs by jointly optimizing the UAV trajectory, IRS phase shift matrix, and IRS scheduling, subject to the minimum primary rate requirements. To tackle this complicated problem, a relaxation-based algorithm is proposed. We prove that the converged relaxation scheduling variables are binary, which means that no reconstruct strategy is needed, and thus the UAV rate constraints are automatically satisfied. Second, we consider the fairness BER optimization problem. We find that the relaxation-based method cannot solve this fairness BER problem since the minimum primary rate requirements may not be satisfied by the binary reconstruction operation. To address this issue, we first transform the binary constraints into a series of equivalent equality constraints. Then, a penalty-based algorithm is proposed to obtain a suboptimal solution. Numerical results are provided to evaluate the performance of the proposed designs under different setups, as compared with benchmarks.

Aerial Intelligent Reflecting Surface: Joint Placement and Passive Beamforming Design with 3D Beam Flattening

Abstract: This paper proposes a new three-dimensional (3D) wireless passive relaying system enabled by aerial IRS (AIRS). Compared to the conventional terrestrial IRS, AIRS enjoys more deployment flexibility as well as wider-range signal reflection, thanks to its high altitude and thus more likelihood of establishing line-of-sight (LoS) links with ground source/destination nodes. Specifically, we aim to maximize the worst-case signal-to-noise ratio (SNR) over all locations in a target area by jointly optimizing the transmit beamforming for the source node and the placement as well as 3D passive beamforming for the AIRS. The formulated problem is non-convex and thus difficult to solve. To gain useful insights, we first consider the special case of maximizing the SNR at a given target location, for which the optimal solution is obtained in closed-form. The result shows that the optimal horizontal AIRS placement only depends on the ratio between the source-destination distance and the AIRS altitude. Then for the general case of AIRS-enabled area coverage, we propose an efficient solution by decoupling the AIRS passive beamforming design to maximize the worst-case array gain, from its placement optimization by balancing the resulting angular span and the cascaded channel path loss. Our proposed solution is based on a novel 3D beam broadening and flattening technique, where the passive array of the AIRS is divided into sub-arrays of appropriate size, and their phase shifts are designed to form a flattened beam pattern with adjustable beamwidth catering to the size of the coverage area. Both the uniform linear array (ULA)-based and uniform planar array (UPA)-based AIRSs are considered in our design, which enable two-dimensional (2D) and 3D passive beamforming, respectively. Numerical results show that the proposed designs achieve significant performance gains over the benchmark schemes.

Reflections in the Sky: Millimeter Wave Communication with UAV-Carried Intelligent Reflectors

Abstract: In this paper, a novel approach that uses an unmanned aerial vehicle (UAV)-carried intelligent reflector (IR) is proposed to enhance the performance of millimeter wave (mmW) networks. In particular, the UAV-IR is used to intelligently reflect mmW beamforming signals from a base station towards a mobile outdoor user, while harvesting energy from mmW signals to power the IR. To maintain a line-of-sight (LOS) channel, a reinforcement learning (RL) approach, based on Q-learning and neural networks, is proposed to model the propagation environment, such that the location and reflection coefficient of the UAV-IR can be optimized to maximize the downlink transmission capacity. Simulation results show a significant advantage for using a UAV-IR over a static IR, in terms of the average data rate and the achievable downlink LOS probability. The results also show that the RL-based deployment of the UAV-IR further improves the network performance, relative to a scheme without learning.

Reflections in the Sky: Joint Trajectory and Passive Beamforming Design for Secure UAV Networks with Reconfigurable Intelligent Surface

Abstract: This paper investigates the problem of secure energy efficiency maximization for a reconfigurable intelligent surface (RIS) assisted uplink wireless communication system, where an unmanned aerial vehicle (UAV) equipped with an RIS works as a mobile relay between the base station (BS) and a group of users. We focus on maximizing the secure energy efficiency of the system via jointly optimizing the UAV's trajectory, the RIS's phase shift, users' association and transmit power. To tackle this problem, we divide the original problem into three sub-problems, and propose an efficient iterative algorithm. In particular, the successive convex approximation method (SCA) is applied to solve the nonconvex UAV trajectory, the RIS's phase shift, and transmit power optimization sub-problems. We further provide two schemes to simplify the solution of phase and trajectory sub-problem. Simulation results demonstrate that the proposed algorithm converges fast, and the proposed design can enhance the secure energy efficiency by up to 38\% gains, as compared to the traditional schemes without any RIS.

Optimization of Wireless Relaying With Flexible UAV-Borne Reflecting Surfaces

Abstract: This paper presents a theoretical framework to analyze the performance of an integrated unmanned aerial vehicle (UAV)-intelligent reflecting surface (IRS) relaying system in which the IRS provides an additional degree of freedom combined with the flexible deployment of full-duplex UAV to enhance communication between ground nodes. Our framework considers three different transmission modes: (i) UAV-only mode, (ii) IRS-only mode, and (iii) integrated UAV-IRS mode to achieve spectral and energy-efficient relaying. For the proposed modes, we provide exact and approximate expressions for the end-to-end outage probability, ergodic capacity, and energy efficiency (EE) in closed-form. We use the derived expressions to optimize key system parameters such as the UAV altitude and the number of elements on the IRS considering different modes. We formulate the problems in the form of fractional programming (e.g. single ratio, sum of multiple ratios or maximization-minimization of ratios) and devise optimal algorithms using quadratic transformations. Furthermore, we derive an analytic criterion to optimally select different transmission modes to maximize ergodic capacity and EE for a given number of IRS elements. Numerical results validate the derived expressions. The solutions obtained from the proposed optimization algorithms are compared with those obtained through exhaustive search. Insights are drawn related to the different communication modes, optimal number of IRS elements, and optimal UAV height.

Comparison of existing works on IRS and UAV.

Despite the promising advantages of M-MIMO and mmWave communications, their required high complexity and hardware cost as well as increased energy consumption are still crucial issues faced in practical implementation {{cite:c7a3c4e}}, {{cite:b783181}}.
As an alternative, IRS has recently emerged as a new and cost-effective solution to improve the received power and suppress air-ground interference in 3D space {{cite:dd44703}}, {{cite:9d2b6ca}}, {{cite:3ed4da3}}, {{cite:ea4bb95}}. An IRS is composed of a large number of passive reflecting elements, each of which is able to reflect the impinging electromagnetic wave with a tunable reflection coefficient (including an amplitude and a phase shift) {{cite:a9dcfb9}}, {{cite:aeff559}}, {{cite:91e87f0}}, {{cite:4ee4f0c}}. By smartly coordinating the reflections of all elements, an IRS is able to reconfigure the wireless channel with the desired signals added coherently and interference cancelled at designated receivers, thus significantly enhancing the communication throughput without the need for deploying new active BSs or relays.
Furthermore, IRSs possess appealing advantages in practice such as light weight, and thus can be easily mounted on walls and even the surface of high-speed moving vehicles to support numerous applications {{cite:a232879}}, {{cite:24878ce}}, {{cite:5873799}}, {{cite:3782b52}}, {{cite:1bf093f}}. As such, IRS has been considered as a disruptive technology for transforming our current “dumb” radio environment into an intelligent one, which potentially benefits a wide range of vertical industries such as transportation, manufacturing, and smart cities. Recently, IRS has also been recognized as a promising technology for the future sixth-generation (6G) ecosystem {{cite:e44adea}}, {{cite:c476154}}, {{cite:3ed4da3}} and studied in various system setups {{cite:ca45a79}}, {{cite:07d5e09}}, {{cite:7b18965}}, {{cite:c95d432}}, {{cite:b0aba1e}}, {{cite:70a9dcf}}, {{cite:3a43087}}, {{cite:f0a6729}}, {{cite:99dc856}}, {{cite:109b111}}. Generally speaking, as shown in Fig. {{figure:bceeb656-a615-459a-8bf8-ff19dfee9d7b}} , IRS can be either deployed on the ground to assist UAV communications or attached to UAVs to assist terrestrial communications {{cite:0e1c78a}}, as elaborated below. In addition, a brief comparison of existing works on IRS and UAV was summarized in Table {{table:5c0ed06b-d7f7-4430-907c-07d48ee29389}} .


Graph Partition Neural Networks for Semi-Supervised Classification

Abstract: We present graph partition neural networks (GPNN), an extension of graph neural networks (GNNs) able to handle extremely large graphs. GPNNs alternate between locally propagating information between nodes in small subgraphs and globally propagating information between the subgraphs. To efficiently partition graphs, we experiment with several partitioning algorithms and also propose a novel variant for fast processing of large scale graphs. We extensively test our model on a variety of semi-supervised node classification tasks. Experimental results indicate that GPNNs are either superior or comparable to state-of-the-art methods on a wide variety of datasets for graph-based semi-supervised classification. We also show that GPNNs can achieve similar performance as standard GNNs with fewer propagation steps.

Deep Graph Library: Towards Efficient and Scalable Deep Learning on Graphs

Abstract: Accelerating research in the emerging field of deep graph learning requires new tools. Such systems should support graph as the core abstraction and take care to maintain both forward (i.e. supporting new research ideas) and backward (i.e. integration with existing components) compatibility. In this paper, we present Deep Graph Library (DGL). DGL enables arbitrary message handling and mutation operators, flexible propagation rules, and is framework agnostic so as to leverage high-performance tensor, autograd operations, and other feature extraction modules already available in existing frameworks. DGL carefully handles the sparse and irregular graph structure, deals with graphs big and small which may change dynamically, fuses operations, and performs auto-batching, all to take advantages of modern hardware. DGL has been tested on a variety of models, including but not limited to the popular Graph Neural Networks (GNN) and its variants, with promising speed, memory footprint and scalability.

EnGN: A High-Throughput and Energy-Efficient Accelerator for Large Graph Neural Networks

Abstract: Graph neural networks (GNNs) emerge as a powerful approach to process non-euclidean data structures and have been proved powerful in various application domains such as social networks and e-commerce. While such graph data maintained in real-world systems can be extremely large and sparse, thus employing GNNs to deal with them requires substantial computational and memory overhead, which induces considerable energy and resource cost on CPUs and GPUs. In this work, we present a specialized accelerator architecture, EnGN, to enable high-throughput and energy-efficient processing of large-scale GNNs. The proposed EnGN is designed to accelerate the three key stages of GNN propagation, which is abstracted as common computing patterns shared by typical GNNs. To support the key stages simultaneously, we propose the ring-edge-reduce(RER) dataflow that tames the poor locality of sparsely-and-randomly connected vertices, and the RER PE-array to practice RER dataflow. In addition, we utilize a graph tiling strategy to fit large graphs into EnGN and make good use of the hierarchical on-chip buffers through adaptive computation reordering and tile scheduling. Overall, EnGN achieves performance speedup by 1802.9X, 19.75X, and 2.97X and energy efficiency by 1326.35X, 304.43X, and 6.2X on average compared to CPU, GPU, and a state-of-the-art GCN accelerator HyGCN, respectively.

HyGCN: A GCN Accelerator with Hybrid Architecture

Abstract: Inspired by the great success of neural networks, graph convolutional neural networks (GCNs) are proposed to analyze graph data. GCNs mainly include two phases with distinct execution patterns. The Aggregation phase, behaves as graph processing, showing a dynamic and irregular execution pattern. The Combination phase, acts more like the neural networks, presenting a static and regular execution pattern. The hybrid execution patterns of GCNs require a design that alleviates irregularity and exploits regularity. Moreover, to achieve higher performance and energy efficiency, the design needs to leverage the high intra-vertex parallelism in Aggregation phase, the highly reusable inter-vertex data in Combination phase, and the opportunity to fuse phase-by-phase execution introduced by the new features of GCNs. However, existing architectures fail to address these demands. In this work, we first characterize the hybrid execution patterns of GCNs on Intel Xeon CPU. Guided by the characterization, we design a GCN accelerator, HyGCN, using a hybrid architecture to efficiently perform GCNs. Specifically, first, we build a new programming model to exploit the fine-grained parallelism for our hardware design. Second, we propose a hardware design with two efficient processing engines to alleviate the irregularity of Aggregation phase and leverage the regularity of Combination phase. Besides, these engines can exploit various parallelism and reuse highly reusable data efficiently. Third, we optimize the overall system via inter-engine pipeline for inter-phase fusion and priority-based off-chip memory access coordination to improve off-chip bandwidth utilization. Compared to the state-of-the-art software framework running on Intel Xeon CPU and NVIDIA V100 GPU, our work achieves on average 1509× speedup with 2500× energy reduction and average 6.5× speedup with 10× energy reduction, respectively.

Homogenized nomenclature for aggregate and combine functions in the literature

We finally note that, due to the emergence of GNNs, aggregation and combination functions have taken different names in the literature. In an attempt to unify the notation, some equivalences are listed in Table {{table:4774b187-f415-46e0-9cfb-3cde6f36984c}} .


MacroBase: Prioritizing Attention in Fast Data

Abstract: As data volumes continue to rise, manual inspection is becoming increasingly untenable. In response, we present MacroBase, a data analytics engine that prioritizes end-user attention in high-volume fast data streams. MacroBase enables efficient, accurate, and modular analyses that highlight and aggregate important and unusual behavior, acting as a search engine for fast data. MacroBase is able to deliver order-of-magnitude speedups over alternatives by optimizing the combination of explanation and classification tasks and by leveraging a new reservoir sampler and heavy-hitters sketch specialized for fast data streams. As a result, MacroBase delivers accurate results at speeds of up to 2M events per second per query on a single core. The system has delivered meaningful results in production, including at a telematics company monitoring hundreds of thousands of vehicles.

A Unified Approach to Interpreting Model Predictions

Abstract: Understanding why a model makes a certain prediction can be as crucial as the prediction's accuracy in many applications. However, the highest accuracy for large modern datasets is often achieved by complex models that even experts struggle to interpret, such as ensemble or deep learning models, creating a tension between accuracy and interpretability. In response, various methods have recently been proposed to help users interpret the predictions of complex models, but it is often unclear how these methods are related and when one method is preferable over another. To address this problem, we present a unified framework for interpreting predictions, SHAP (SHapley Additive exPlanations). SHAP assigns each feature an importance value for a particular prediction. Its novel components include: (1) the identification of a new class of additive feature importance measures, and (2) theoretical results showing there is a unique solution in this class with a set of desirable properties. The new class unifies six existing methods, notable because several recent methods in the class lack the proposed desirable properties. Based on insights from this unification, we present new methods that show improved computational performance and/or better consistency with human intuition than previous approaches.

Beyond Sparsity: Tree Regularization of Deep Models for Interpretability

Abstract: The lack of interpretability remains a key barrier to the adoption of deep models in many applications. In this work, we explicitly regularize deep models so human users might step through the process behind their predictions in little time. Specifically, we train deep time-series models so their class-probability predictions have high accuracy while being closely modeled by decision trees with few nodes. Using intuitive toy examples as well as medical tasks for treating sepsis and HIV, we demonstrate that this new tree regularization yields models that are easier for humans to simulate than simpler L1 or L2 penalties without sacrificing predictive power.

Metrics used in other ED works and applicability of the Exathlon metrics

Applicability to Other ED Works.
While a user study may be the best way to evaluate the usefulness of explanations, it is not always available and may come at a high cost. Therefore, our benchmark aims to provide automated evaluation of ED methods based on intuitive metrics. For selecting those, we surveyed multiple other ED works and summarized the main metrics they used (see Table {{table:89a69e4b-7201-485e-9709-8bf4d2d56963}} ). Among these, we found conciseness and accuracy to be the most general and intuitive: human users typically prefer small sets of important factors as explanations, and if these were to be used as an AD tool, they should ideally achieve good accuracy. We also added the concept of consistency, given that accuracy is not always applicable or meaningful for a given ED method. As an example, running LIME to explain the outlier score assigned to a given test sample might not be directly usable for binary prediction. Another issue might be the impact of hard-coded constants in explanation predicates. For example, a valid explanation to a human user could be that “the CPU usage is too high”. However, this notion of “too high” might be a relative concept, meaning above 50% in a given context but above 80% in another. A lot of ED methods typically include such constants in their explanations, sometimes leading a “good” instance explanation to have poor accuracy when applied to other instances. Like conciseness, the concept of consistency has the benefit to rule out the effect of such constants in the evaluation of an ED method. Although the concept of these three metrics is common to all evaluated methods, the way we calculate them typically depends on how the method works. For instance, if the method takes a dataset as input to generate the explanation, like in {{cite:09a3ebb}}, {{cite:1beca84}}, {{cite:bae1914}}, {{cite:7e89b88}}, we consider the consistency implementation described in Section . For other methods, e.g.,  {{cite:f84e1ce}}, {{cite:5127772}}, {{cite:cde774e}}, the calculation of consistency is window-based, implemented for LIME {{cite:f84e1ce}} in our pipeline. If needed, other implementations can also be added by the user.


Fingerprint Spoof Buster, LOO + best subset

Feature encoding. Unknown PAs, cross sensor/DB

Generalizing Fingerprint Spoof Detector: Learning a One-Class Classifier

Abstract: Prevailing fingerprint recognition systems are vulnerable to spoof attacks. To mitigate these attacks, automated spoof detectors are trained to distinguish a set of live or bona fide fingerprints from a set of known spoof fingerprints. Despite their success, spoof detectors remain vulnerable when exposed to attacks from spoofs made with materials not seen during training of the detector. To alleviate this shortcoming, we approach spoof detection as a one-class classification problem. The goal is to train a spoof detector on only the live fingerprints such that once the concept of "live" has been learned, spoofs of any material can be rejected. We accomplish this through training multiple generative adversarial networks (GANS) on live fingerprint images acquired with the open source, dual-camera, 1900 ppi RaspiReader fingerprint reader. Our experimental results, conducted on 5.5K spoof images (from 12 materials) and 11.8K live images show that the proposed approach improves the cross-material spoof detection performance over state-of-the-art one-class and binary class spoof detectors on 11 of 12 testing materials and 7 of 12 testing materials, respectively.

On the Effectiveness of Laser Speckle Contrast Imaging and Deep Neural Networks for Detecting Known and Unknown Fingerprint Presentation Attacks

Abstract: Fingerprint presentation attack detection (FPAD) is becoming an increasingly challenging problem due to the continuous advancement of attack techniques, which generate `realistic-looking' fake fingerprint presentations. Recently, laser speckle contrast imaging (LSCI) has been introduced as a new sensing modality for FPAD. LSCI has the interesting characteristic of capturing the blood flow under the skin surface. Toward studying the importance and effectiveness of LSCI for FPAD, we conduct a comprehensive study using different patch-based deep neural network architectures. Our studied architectures include 2D and 3D convolutional networks as well as a recurrent network using long short-term memory (LSTM) units. The study demonstrates that strong FPAD performance can be achieved using LSCI. We evaluate the different models over a new large dataset. The dataset consists of 3743 bona fide samples, collected from 335 unique subjects, and 218 presentation attack samples, including six different types of attacks. To examine the effect of changing the training and testing sets, we conduct a 3-fold cross validation evaluation. To examine the effect of the presence of an unseen attack, we apply a leave-one-attack out strategy. The FPAD classification results of the networks, which are separately optimized and tuned for the temporal and spatial patch-sizes, indicate that the best performance is achieved by LSTM.

Fingerprint Presentation Attack Detection: Generalization and Efficiency

Abstract: We study the problem of fingerprint presentation attack detection (PAD) under unknown PA materials not seen during PAD training. A dataset of 5,743 bonafide and 4,912 PA images of 12 different materials is used to evaluate a state-of-the-art PAD, namely Fingerprint Spoof Buster. We utilize 3D t-SNE visualization and clustering of material characteristics to identify a representative set of PA materials that cover most of PA feature space. We observe that a set of six PA materials, namely Silicone, 2D Paper, Play Doh, Gelatin, Latex Body Paint and Monster Liquid Latex provide a good representative set that should be included in training to achieve generalization of PAD. We also propose an optimized Android app of Fingerprint Spoof Buster that can run on a commodity smartphone (Xiaomi Redmi Note 4) without a significant drop in PAD performance (from TDR = 95.7% to 95.3% @ FDR = 0.2%) which can make a PA prediction in less than 300ms.

Fingerprint Presentation Attack Detection Based on Local Features Encoding for Unknown Attacks

Abstract: Fingerprint-based biometric systems have experienced a large development in the last years. Despite their many advantages, they are still vulnerable to presentation attacks (PAs). Therefore, the task of determining whether a sample stems from a live subject (i.e., bona fide) or from an artificial replica is a mandatory issue which has received a lot of attention recently. Nowadays, when the materials for the fabrication of the Presentation Attack Instruments (PAIs) have been used to train the PA Detection (PAD) methods, the PAIs can be successfully identified. However, current PAD methods still face difficulties detecting PAIs built from unknown materials or captured using other sensors. Based on that fact, we propose a new PAD technique based on three image representation approaches combining local and global information of the fingerprint. By transforming these representations into a common feature space, we can correctly discriminate bona fide from attack presentations in the aforementioned scenarios. The experimental evaluation of our proposal over the LivDet 2011 to 2015 databases, yielded error rates outperforming the top state-of-the-art results by up to 50\% in the most challenging scenarios. In addition, the best configuration achieved the best results in the LivDet 2019 competition (overall accuracy of 96.17\%).

Anomaly Detection with Convolutional Autoencoders for Fingerprint Presentation Attack Detection

Abstract: In recent years, the popularity of fingerprint-based biometric authentication systems significantly increased. However, together with many advantages, biometric systems are still vulnerable to presentation attacks (PAs). In particular, this applies for unsupervised applications, where new attacks unknown to the system operator may occur. Therefore, presentation attack detection (PAD) methods are used to determine whether samples stem from a bona fide subject or from a presentation attack instrument (PAI). In this context, most works are dedicated to solve PAD as a two-class classification problem, which includes training a model on both bona fide and PA samples. In spite of the good detection rates reported, these methods still face difficulties detecting PAIs from unknown materials. To address this issue, we propose a new PAD technique based on autoencoders (AEs) trained only on bona fide samples (i.e. one-class), which are captured in the short wave infrared domain. On the experimental evaluation over a database of 19,711 bona fide and 4,339 PA images including 45 different PAI species, a detection equal error rate (D-EER) of 2.00% was achieved. Additionally, our best performing AE model is compared to further one-class classifiers (support vector machine, Gaussian mixture model). The results show the effectiveness of the AE model as it significantly outperforms the previously proposed methods.

Fingerprint PAD approaches on unknown attacks.

A detailed overview of hardware-based fingerprint PAD is presented in {{cite:ba825b3}}, hence we focus on unknown attacks {{cite:46bb6c6}} and generalisation approaches in this work. A summary of these works is given in Table {{table:b6972a6f-bf32-4b15-9e8c-44437633f62a}} . It should be noted that, most of the reviewed approaches use different datasets and present multiple experiments including leave-one-out (LOO) or cross database/sensor protocols. For LOO experiments, usually one PAI species is left out from training and only used for testing to evaluate the classification of unknown attacks. Hence, a comparison of performance metrics is not included due to its lack of fairness.


Channel estimation with lower transmission overhead

RIS-assisted network with all passive elements

RIS-assisted network with no labeled data

Estimate both direct channel and cascaded channel at the same time.

Reduce transmission overhead by 16X compared to centralized leaning approach.

Learn the mapping between a user's position and the optimal configuration of each RIS element with a fingerprinting database.

Introduce an Env-Twin framework to model RIS-assisted environment exploiting location attributes without channel estimation overhead.

Introduce unsupervised learning approach with no need of collecting labeled training data in advance.

Approach does not require any control from communication infrastructure. Validation showed achieving comparable performance as supervised learning.

Deep Channel Learning for Large Intelligent Surfaces Aided mm-Wave Massive MIMO Systems

Abstract: This letter presents the first work introducing a deep learning (DL) framework for channel estimation in large intelligent surface (LIS) assisted massive MIMO (multiple-input multiple-output) systems. A twin convolutional neural network (CNN) architecture is designed and it is fed with the received pilot signals to estimate both direct and cascaded channels. In a multi-user scenario, each user has access to the CNN to estimate its own channel. The performance of the proposed DL approach is evaluated and compared with state-of-the-art DL-based techniques and its superior performance is demonstrated.

Federated Learning for Channel Estimation in Conventional and RIS-Assisted Massive MIMO

Abstract: Machine learning (ML) has attracted a great research interest for physical layer design problems, such as channel estimation, thanks to its low complexity and robustness. Channel estimation via ML requires model training on a dataset, which usually includes the received pilot signals as input and channel data as output. In previous works, model training is mostly done via centralized learning (CL), where the whole training dataset is collected from the users at the base station (BS). This approach introduces huge communication overhead for data collection. In this paper, to address this challenge, we propose a federated learning (FL) framework for channel estimation. We design a convolutional neural network (CNN) trained on the local datasets of the users without sending them to the BS. We develop FL-based channel estimation schemes for both conventional and RIS (intelligent reflecting surface) assisted massive MIMO (multiple-input multiple-output) systems, where a single CNN is trained for two different datasets for both scenarios. We evaluate the performance for noisy and quantized model transmission and show that the proposed approach provides approximately 16 times lower overhead than CL, while maintaining satisfactory performance close to CL. Furthermore, the proposed architecture exhibits lower estimation error than the state-of-the-art ML-based schemes.

Indoor Signal Focusing with Deep Learning Designed Reconfigurable Intelligent Surfaces

Abstract: Reconfigurable Intelligent Surfaces (RISs) comprised of tunable unit elements have been recently considered in indoor communication environments for focusing signal reflections to intended user locations. However, the current proofs of concept require complex operations for the RIS configuration, which are mainly realized via wired control connections. In this paper, we present a deep learning method for efficient online wireless configuration of RISs when deployed in indoor communication environments. According to the proposed method, a database of coordinate fingerprints is implemented during an offline training phase. This fingerprinting database is used to train the weights and bias of a properly designed Deep Neural Network (DNN), whose role is to unveil the mapping between the measured coordinate information at a user location and the configuration of the RIS's unit cells that maximizes this user's received signal strength. During the online phase of the presented method, the trained DNN is fed with the measured position information at the target user to output the optimal phase configurations of the RIS for signal power focusing on this intended location. Our realistic simulation results using ray tracing on a three dimensional indoor environment demonstrate that the proposed DNN-based configuration method exhibits its merits for all considered cases, and effectively increases the achievable throughput at the target user location.

A Digital Twin for Reconfigurable Intelligent Surface Assisted Wireless Communication

Abstract: Reconfigurable Intelligent Surface (RIS) has emerged as one of the key technologies for 6G in recent years, which comprise a large number of low-cost passive elements that can smartly interact with the impinging electromagnetic waves for performance enhancement. However, optimally configuring massive number of RIS elements remains a challenge. In this paper, we present a novel digital-twin framework for RIS-assisted wireless networks which we name it Environment-Twin (Env-Twin). The goal of the Env-Twin framework is to enable automation of optimal control at various granularities. In this paper, we present one example of the Env-Twin models to learn the mapping function between the RIS configuration with measured attributes for the receiver location, and the corresponding achievable rate in an RIS-assisted wireless network without involving explicit channel estimation or beam training overhead. Once learned, our Env-Twin model can be used to predict optimal RIS configuration for any new receiver locations in the same wireless network. We leveraged deep learning (DL) techniques to build our model and studied its performance and robustness. Simulation results demonstrate that the proposed Env-Twin model can recommend near-optimal RIS configurations for test receiver locations which achieved close to an upper bound performance that assumes perfect channel knowledge. Our Env-Twin model was trained using less than 2% of the total receiver locations. This promising result represents great potential of the proposed Env-Twin framework for developing a practical RIS solution where the panel can automatically configure itself without requesting channel state information (CSI) from the wireless network infrastructure.

Unsupervised Learning for Passive Beamforming

Abstract: Reconfigurable intelligent surface (RIS) has recently emerged as a promising candidate to improve the energy and spectral efficiency of wireless communication systems. However, the unit modulus constraint on the phase shift of reflecting elements makes the design of optimal passive beamforming solution a challenging issue. The conventional approach is to find a suboptimal solution using the semi-definite relaxation (SDR) technique, yet the resultant suboptimal iterative algorithm usually incurs high complexity, hence is not amenable for real-time implementation. Motivated by this, we propose a deep learning approach for passive beamforming design in RIS-assisted systems. In particular, a customized deep neural network is trained offline using the unsupervised learning mechanism, which is able to make real-time prediction when deployed online. Simulation results show that the proposed approach maintains most of the performance while significantly reduces computation complexity when compared with SDR-based approach.

Deep Reinforcement Learning for Intelligent Reflecting Surfaces: Towards Standalone Operation

Abstract: The promising coverage and spectral efficiency gains of intelligent reflecting surfaces (IRSs) are attracting increasing interest. In order to realize these surfaces in practice, however, several challenges need to be addressed. One of these main challenges is how to configure the reflecting coefficients on these passive surfaces without requiring massive channel estimation or beam training overhead. Earlier work suggested leveraging supervised learning tools to design the IRS reflection matrices. While this approach has the potential of reducing the beam training overhead, it requires collecting large datasets for training the neural network models. In this paper, we propose a novel deep reinforcement learning framework for predicting the IRS reflection matrices with minimal training overhead. Simulation results show that the proposed online learning framework can converge to the optimal rate that assumes perfect channel knowledge. This represents an important step towards realizing a standalone IRS operation, where the surface configures itself without any control from the infrastructure.

Summary of existing ML-based RIS research works: (1) ML-based approach to assist channel state acquisition; (2) ML-based approach without explicit channel estimation; (3) Hybrid solution.

Hybrid solution combining both model based and model free approach

We summarize the above discussed AI/ML-based RIS solutions and experiments in Table  {{table:5685a7e8-9288-4de3-b876-ed4bb605ec37}}


Debiasing Word Embeddings Improves Multimodal Machine Translation

Abstract: In recent years, pretrained word embeddings have proved useful for multimodal neural machine translation (NMT) models to address the shortage of available datasets. However, the integration of pretrained word embeddings has not yet been explored extensively. Further, pretrained word embeddings in high dimensional spaces have been reported to suffer from the hubness problem. Although some debiasing techniques have been proposed to address this problem for other natural language processing tasks, they have seldom been studied for multimodal NMT models. In this study, we examine various kinds of word embeddings and introduce two debiasing techniques for three multimodal NMT models and two language pairs -- English-German translation and English-French translation. With our optimal settings, the overall performance of multimodal models was improved by up to +1.93 BLEU and +2.02 METEOR for English-German translation and +1.73 BLEU and +0.95 METEOR for English-French translation.

Visual Agreement Regularized Training for Multi-Modal Machine Translation

Abstract: Multi-modal machine translation aims at translating the source sentence into a different language in the presence of the paired image. Previous work suggests that additional visual information only provides dispensable help to translation, which is needed in several very special cases such as translating ambiguous words. To make better use of visual information, this work presents visual agreement regularized training. The proposed approach jointly trains the source-to-target and target-to-source translation models and encourages them to share the same focus on the visual information when generating semantically equivalent visual words (e.g. "ball" in English and "ballon" in French). Besides, a simple yet effective multi-head co-attention model is also introduced to capture interactions between visual and textual features. The results show that our approaches can outperform competitive baselines by a large margin on the Multi30k dataset. Further analysis demonstrates that the proposed regularized training can effectively improve the agreement of attention on the image, leading to better use of visual information.

Unsupervised Multimodal Neural Machine Translation with Pseudo Visual Pivoting

Abstract: Unsupervised machine translation (MT) has recently achieved impressive results with monolingual corpora only. However, it is still challenging to associate source-target sentences in the latent space. As people speak different languages biologically share similar visual systems, the potential of achieving better alignment through visual content is promising yet under-explored in unsupervised multimodal MT (MMT). In this paper, we investigate how to utilize visual content for disambiguation and promoting latent space alignment in unsupervised MMT. Our model employs multimodal back-translation and features pseudo visual pivoting in which we learn a shared multilingual visual-semantic embedding space and incorporate visually-pivoted captioning as additional weak supervision. The experimental results on the widely used Multi30K dataset show that the proposed model significantly improves over the state-of-the-art methods and generalizes well when the images are not available at the testing time.

Distilling Translations with Visual Awareness

Abstract: Previous work on multimodal machine translation has shown that visual information is only needed in very specific cases, for example in the presence of ambiguous words where the textual context is not sufficient. As a consequence, models tend to learn to ignore this information. We propose a translate-and-refine approach to this problem where images are only used by a second stage decoder. This approach is trained jointly to generate a good first draft translation and to improve over this draft by (i) making better use of the target language textual context (both left and right-side contexts) and (ii) making use of visual context. This approach leads to the state of the art results. Additionally, we show that it has the ability to recover from erroneous or missing words in the source language.

Unsupervised Multi-modal Neural Machine Translation

Abstract: Unsupervised neural machine translation (UNMT) has recently achieved remarkable results with only large monolingual corpora in each language. However, the uncertainty of associating target with source sentences makes UNMT theoretically an ill-posed problem. This work investigates the possibility of utilizing images for disambiguation to improve the performance of UNMT. Our assumption is intuitively based on the invariant property of image, i.e., the description of the same visual content by different languages should be approximately similar. We propose an unsupervised multi-modal machine translation (UMNMT) framework based on the language translation cycle consistency loss conditional on the image, targeting to learn the bidirectional multi-modal translation simultaneously. Through an alternate training between multi-modal and uni-modal, our inference model can translate with or without the image. On the widely used Multi30K dataset, the experimental results of our approach are significantly better than those of the text-only UNMT on the 2016 test dataset.

Multimodal Machine Translation with Embedding Prediction

Abstract: Multimodal machine translation is an attractive application of neural machine translation (NMT). It helps computers to deeply understand visual objects and their relations with natural languages. However, multimodal NMT systems suffer from a shortage of available training data, resulting in poor performance for translating rare words. In NMT, pretrained word embeddings have been shown to improve NMT of low-resource domains, and a search-based approach is proposed to address the rare word problem. In this study, we effectively combine these two approaches in the context of multimodal NMT and explore how we can take full advantage of pretrained word embeddings to better translate rare words. We report overall performance improvements of 1.24 METEOR and 2.49 BLEU and achieve an improvement of 7.67 F-score for rare word translation.

Latest research in Multimodal Machine Translation.

MMT is one of the most primitive VisLang tasks that has been a source of interest in the community for a long time, but it has gained escalation in several recent works focussing on this task (see Table {{table:f9ec1712-38b6-46a1-9f80-1a7296cd794b}} ). The emergence of generalizable BERT-based pre-trained latent spaces has led to a boost in the performance of MMT models over a wide range of global and regional languages. Similar to VC, former datasets like Multi30k  {{cite:6469c11}} are still the most popular benchmarks for this particular task. However, the growth of reasoning-oriented models has led to some newer datasets (such as VATEX  {{cite:65c45ee}} and Flickr30-Entities  {{cite:6fbdafe}}) that demand higher-level capabilities.


Words Can Shift: Dynamically Adjusting Word Representations Using Nonverbal Behaviors

Abstract: Humans convey their intentions through the usage of both verbal and nonverbal behaviors during face-to-face communication. Speaker intentions often vary dynamically depending on different nonverbal contexts, such as vocal patterns and facial expressions. As a result, when modeling human language, it is essential to not only consider the literal meaning of the words but also the nonverbal contexts in which these words appear. To better model human language, we first model expressive nonverbal representations by analyzing the fine-grained visual and acoustic patterns that occur during word segments. In addition, we seek to capture the dynamic nature of nonverbal intents by shifting word representations based on the accompanying nonverbal behaviors. To this end, we propose the Recurrent Attended Variation Embedding Network (RAVEN) that models the fine-grained structure of nonverbal subword sequences and dynamically shifts word representations based on nonverbal cues. Our proposed model achieves competitive performance on two publicly available datasets for multimodal sentiment analysis and emotion recognition. We also visualize the shifted word representations in different nonverbal contexts and summarize common patterns regarding multimodal variations of word representations.

UR-FUNNY: A Multimodal Language Dataset for Understanding Humor

Abstract: Humor is a unique and creative communicative behavior displayed during social interactions. It is produced in a multimodal manner, through the usage of words (text), gestures (vision) and prosodic cues (acoustic). Understanding humor from these three modalities falls within boundaries of multimodal language; a recent research trend in natural language processing that models natural language as it happens in face-to-face communication. Although humor detection is an established research area in NLP, in a multimodal context it is an understudied area. This paper presents a diverse multimodal dataset, called UR-FUNNY, to open the door to understanding multimodal language used in expressing humor. The dataset and accompanying studies, present a framework in multimodal humor detection for the natural language processing community. UR-FUNNY is publicly available for research.

Latest research in Multimodal Affective Computing (MAC). Note: FACET is available at https://imotions.com/platform/.

The latest trends in multimodal affective computing span several promising directions (see Table {{table:1cce3a99-d666-4e1b-bacc-376826d8d4a0}} ). One class of models improve upon tensor-based fusion methods and attempt to find efficient solutions to otherwise inefficient process {{cite:9f79b56}}, {{cite:d706775}}. Recent works, such as  {{cite:2559b21}} also address parallel, yet significant issues that include accounting for temporal imperfections in multimodal time-series data.


Sub-Instruction Aware Vision-and-Language Navigation

Abstract: Vision-and-language navigation requires an agent to navigate through a real 3D environment following a given natural language instruction. Despite significant advances, few previous works are able to fully utilize the strong correspondence between the visual and textual sequences. Meanwhile, due to the lack of intermediate supervision, the agent's performance at following each part of the instruction remains untrackable during navigation. In this work, we focus on the granularity of the visual and language sequences as well as the trackability of agents through the completion of instruction. We provide agents with fine-grained annotations during training and find that they are able to follow the instruction better and have a higher chance of reaching the target at test time. We enrich the previous dataset with sub-instructions and their corresponding paths. To make use of this data, we propose an effective sub-instruction attention and shifting modules that attend and select a single sub-instruction at each time-step. We implement our sub-instruction modules in four state-of-the-art agents, compare with their baseline model, and show that our proposed method improves the performance of all four agents.

Take the Scenic Route: Improving Generalization in Vision-and-Language Navigation

Abstract: In the Vision-and-Language Navigation (VLN) task, an agent with egocentric vision navigates to a destination given natural language instructions. The act of manually annotating these instructions is timely and expensive, such that many existing approaches automatically generate additional samples to improve agent performance. However, these approaches still have difficulty generalizing their performance to new environments. In this work, we investigate the popular Room-to-Room (R2R) VLN benchmark and discover that what is important is not only the amount of data you synthesize, but also how you do it. We find that shortest path sampling, which is used by both the R2R benchmark and existing augmentation methods, encode biases in the action space of the agent which we dub as action priors. We then show that these action priors offer one explanation toward the poor generalization of existing works. To mitigate such priors, we propose a path sampling method based on random walks to augment the data. By training with this augmentation strategy, our agent is able to generalize better to unknown environments compared to the baseline, significantly improving model performance in the process.

Vision-Dialog Navigation by Exploring Cross-modal Memory

Abstract: Vision-dialog navigation posed as a new holy-grail task in vision-language disciplinary targets at learning an agent endowed with the capability of constant conversation for help with natural language and navigating according to human responses. Besides the common challenges faced in visual language navigation, vision-dialog navigation also requires to handle well with the language intentions of a series of questions about the temporal context from dialogue history and co-reasoning both dialogs and visual scenes. In this paper, we propose the Cross-modal Memory Network (CMN) for remembering and understanding the rich information relevant to historical navigation actions. Our CMN consists of two memory modules, the language memory module (L-mem) and the visual memory module (V-mem). Specifically, L-mem learns latent relationships between the current language interaction and a dialog history by employing a multi-head attention mechanism. V-mem learns to associate the current visual views and the cross-modal memory about the previous navigation actions. The cross-modal memory is generated via a vision-to-language attention and a language-to-vision attention. Benefiting from the collaborative learning of the L-mem and the V-mem, our CMN is able to explore the memory about the decision making of historical navigation actions which is for the current step. Experiments on the CVDN dataset show that our CMN outperforms the previous state-of-the-art model by a significant margin on both seen and unseen environments.

Multi-View Learning for Vision-and-Language Navigation

Abstract: Learning to navigate in a visual environment following natural language instructions is a challenging task because natural language instructions are highly variable, ambiguous, and under-specified. In this paper, we present a novel training paradigm, Learn from EveryOne (LEO), which leverages multiple instructions (as different views) for the same trajectory to resolve language ambiguity and improve generalization. By sharing parameters across instructions, our approach learns more effectively from limited training data and generalizes better in unseen environments. On the recent Room-to-Room (R2R) benchmark dataset, LEO achieves 16% improvement (absolute) over a greedy agent as the base agent (25.3% $\rightarrow$ 41.4%) in Success Rate weighted by Path Length (SPL). Further, LEO is complementary to most existing models for vision-and-language navigation, allowing for easy integration with the existing techniques, leading to LEO+, which creates the new state of the art, pushing the R2R benchmark to 62% (9% absolute improvement).

Environment-agnostic Multitask Learning for Natural Language Grounded Navigation

Abstract: Recent research efforts enable study for natural language grounded navigation in photo-realistic environments, e.g., following natural language instructions or dialog. However, existing methods tend to overfit training data in seen environments and fail to generalize well in previously unseen environments. To close the gap between seen and unseen environments, we aim at learning a generalized navigation model from two novel perspectives: (1) we introduce a multitask navigation model that can be seamlessly trained on both Vision-Language Navigation (VLN) and Navigation from Dialog History (NDH) tasks, which benefits from richer natural language guidance and effectively transfers knowledge across tasks; (2) we propose to learn environment-agnostic representations for the navigation policy that are invariant among the environments seen during training, thus generalizing better on unseen environments. Extensive experiments show that environment-agnostic multitask learning significantly reduces the performance gap between seen and unseen environments, and the navigation agent trained so outperforms baselines on unseen environments by 16% (relative measure on success rate) on VLN and 120% (goal progress) on NDH. Our submission to the CVDN leaderboard establishes a new state-of-the-art for the NDH task on the holdout test set. Code is available at https://github.com/google-research/valan.

Just Ask:An Interactive Learning Framework for Vision and Language Navigation

Abstract: In the vision and language navigation task, the agent may encounter ambiguous situations that are hard to interpret by just relying on visual information and natural language instructions. We propose an interactive learning framework to endow the agent with the ability to ask for users' help in such situations. As part of this framework, we investigate multiple learning approaches for the agent with different levels of complexity. The simplest model-confusion-based method lets the agent ask questions based on its confusion, relying on the predefined confidence threshold of a next action prediction model. To build on this confusion-based method, the agent is expected to demonstrate more sophisticated reasoning such that it discovers the timing and locations to interact with a human. We achieve this goal using reinforcement learning (RL) with a proposed reward shaping term, which enables the agent to ask questions only when necessary. The success rate can be boosted by at least 15% with only one question asked on average during the navigation. Furthermore, we show that the RL agent is capable of adjusting dynamically to noisy human responses. Finally, we design a continual learning strategy, which can be viewed as a data augmentation method, for the agent to improve further utilizing its interaction history with a human. We demonstrate the proposed strategy is substantially more realistic and data-efficient compared to previously proposed pre-exploration techniques.

Vision-Language Navigation with Self-Supervised Auxiliary Reasoning Tasks

Abstract: Vision-Language Navigation (VLN) is a task where agents learn to navigate following natural language instructions. The key to this task is to perceive both the visual scene and natural language sequentially. Conventional approaches exploit the vision and language features in cross-modal grounding. However, the VLN task remains challenging, since previous works have neglected the rich semantic information contained in the environment (such as implicit navigation graphs or sub-trajectory semantics). In this paper, we introduce Auxiliary Reasoning Navigation (AuxRN), a framework with four self-supervised auxiliary reasoning tasks to take advantage of the additional training signals derived from the semantic information. The auxiliary tasks have four reasoning objectives: explaining the previous actions, estimating the navigation progress, predicting the next orientation, and evaluating the trajectory consistency. As a result, these additional training signals help the agent to acquire knowledge of semantic representations in order to reason about its activity and build a thorough perception of the environment. Our experiments indicate that auxiliary reasoning tasks improve both the performance of the main task and the model generalizability by a large margin. Empirically, we demonstrate that an agent trained with self-supervised auxiliary reasoning tasks substantially outperforms the previous state-of-the-art method, being the best existing approach on the standard benchmark.

Counterfactual Vision-and-Language Navigation via Adversarial Path Sampling

Abstract: Vision-and-Language Navigation (VLN) is a task where agents must decide how to move through a 3D environment to reach a goal by grounding natural language instructions to the visual surroundings. One of the problems of the VLN task is data scarcity since it is difficult to collect enough navigation paths with human-annotated instructions for interactive environments. In this paper, we explore the use of counterfactual thinking as a human-inspired data augmentation method that results in robust models. Counterfactual thinking is a concept that describes the human propensity to create possible alternatives to life events that have already occurred. We propose an adversarial-driven counterfactual reasoning model that can consider effective conditions instead of low-quality augmented data. In particular, we present a model-agnostic adversarial path sampler (APS) that learns to sample challenging paths that force the navigator to improve based on the navigation performance. APS also serves to do pre-exploration of unseen environments to strengthen the model's ability to generalize. We evaluate the influence of APS on the performance of different VLN baseline models using the room-to-room dataset (R2R). The results show that the adversarial training process with our proposed APS benefits VLN models under both seen and unseen environments. And the pre-exploration process can further gain additional improvements under unseen environments.

Robust Navigation with Language Pretraining and Stochastic Sampling

Abstract: Core to the vision-and-language navigation (VLN) challenge is building robust instruction representations and action decoding schemes, which can generalize well to previously unseen instructions and environments. In this paper, we report two simple but highly effective methods to address these challenges and lead to a new state-of-the-art performance. First, we adapt large-scale pretrained language models to learn text representations that generalize better to previously unseen instructions. Second, we propose a stochastic sampling scheme to reduce the considerable gap between the expert actions in training and sampled actions in test, so that the agent can learn to correct its own mistakes during long sequential action decoding. Combining the two techniques, we achieve a new state of the art on the Room-to-Room benchmark with 6% absolute gain over the previous best result (47% -> 53%) on the Success Rate weighted by Path Length metric.

Transferable Representation Learning in Vision-and-Language Navigation

Abstract: Vision-and-Language Navigation (VLN) tasks such as Room-to-Room (R2R) require machine agents to interpret natural language instructions and learn to act in visually realistic environments to achieve navigation goals. The overall task requires competence in several perception problems: successful agents combine spatio-temporal, vision and language understanding to produce appropriate action sequences. Our approach adapts pre-trained vision and language representations to relevant in-domain tasks making them more effective for VLN. Specifically, the representations are adapted to solve both a cross-modal sequence alignment and sequence coherence task. In the sequence alignment task, the model determines whether an instruction corresponds to a sequence of visual frames. In the sequence coherence task, the model determines whether the perceptual sequences are predictive sequentially in the instruction-conditioned latent space. By transferring the domain-adapted representations, we improve competitive agents in R2R as measured by the success rate weighted by path length (SPL) metric.

Embodied Vision-and-Language Navigation with Dynamic Convolutional Filters

Abstract: In Vision-and-Language Navigation (VLN), an embodied agent needs to reach a target destination with the only guidance of a natural language instruction. To explore the environment and progress towards the target location, the agent must perform a series of low-level actions, such as rotate, before stepping ahead. In this paper, we propose to exploit dynamic convolutional filters to encode the visual information and the lingual description in an efficient way. Differently from some previous works that abstract from the agent perspective and use high-level navigation spaces, we design a policy which decodes the information provided by dynamic convolution into a series of low-level, agent friendly actions. Results show that our model exploiting dynamic filters performs better than other architectures with traditional convolution, being the new state of the art for embodied VLN in the low-level action space. Additionally, we attempt to categorize recent work on VLN depending on their architectural choices and distinguish two main groups: we call them low-level actions and high-level actions models. To the best of our knowledge, we are the first to propose this analysis and categorization for VLN.

Learning to Navigate Unseen Environments: Back Translation with Environmental Dropout

Abstract: A grand goal in AI is to build a robot that can accurately navigate based on natural language instructions, which requires the agent to perceive the scene, understand and ground language, and act in the real-world environment. One key challenge here is to learn to navigate in new environments that are unseen during training. Most of the existing approaches perform dramatically worse in unseen environments as compared to seen ones. In this paper, we present a generalizable navigational agent. Our agent is trained in two stages. The first stage is training via mixed imitation and reinforcement learning, combining the benefits from both off-policy and on-policy optimization. The second stage is fine-tuning via newly-introduced 'unseen' triplets (environment, path, instruction). To generate these unseen triplets, we propose a simple but effective 'environmental dropout' method to mimic unseen environments, which overcomes the problem of limited seen environment variability. Next, we apply semi-supervised learning (via back-translation) on these dropped-out environments to generate new paths and instructions. Empirically, we show that our agent is substantially better at generalizability when fine-tuned with these triplets, outperforming the state-of-art approaches by a large margin on the private unseen test set of the Room-to-Room task, and achieving the top rank on the leaderboard.

Tactical Rewind: Self-Correction via Backtracking in Vision-and-Language Navigation

Abstract: We present the Frontier Aware Search with backTracking (FAST) Navigator, a general framework for action decoding, that achieves state-of-the-art results on the Room-to-Room (R2R) Vision-and-Language navigation challenge of Anderson et. al. (2018). Given a natural language instruction and photo-realistic image views of a previously unseen environment, the agent was tasked with navigating from source to target location as quickly as possible. While all current approaches make local action decisions or score entire trajectories using beam search, ours balances local and global signals when exploring an unobserved environment. Importantly, this lets us act greedily but use global signals to backtrack when necessary. Applying FAST framework to existing state-of-the-art models achieved a 17% relative gain, an absolute 6% gain on Success rate weighted by Path Length (SPL).

Self-Monitoring Navigation Agent via Auxiliary Progress Estimation

Abstract: The Vision-and-Language Navigation (VLN) task entails an agent following navigational instruction in photo-realistic unknown environments. This challenging task demands that the agent be aware of which instruction was completed, which instruction is needed next, which way to go, and its navigation progress towards the goal. In this paper, we introduce a self-monitoring agent with two complementary components: (1) visual-textual co-grounding module to locate the instruction completed in the past, the instruction required for the next action, and the next moving direction from surrounding images and (2) progress monitor to ensure the grounded instruction correctly reflects the navigation progress. We test our self-monitoring agent on a standard benchmark and analyze our proposed approach through a series of ablation studies that elucidate the contributions of the primary components. Using our proposed method, we set the new state of the art by a significant margin (8% absolute increase in success rate on the unseen test set). Code is available at https://github.com/chihyaoma/selfmonitoring-agent .

Latest research in Vision Language Navigation (VLN)

With the dynamic nature of the task of navigation using language instruction, a wide variety of works  {{cite:c47c235}}, {{cite:2c71360}} have resorted to utilizing (deep) reinforcement learning techniques for fabricating adaptive generalizable models for a range of environmental settings. In such approaches, the agent learns to generate a map of the environment alongside following the instructions to advance towards the goal by receiving rewards from the environment. Even contemporary learning-based techniques like imitation learning have been popularized to learn to navigate using instructions by mimicking an instructor's actions. Despite the rapid growth in VLN, the conventional Room-to-Room (R2R) dataset  {{cite:8d7d1da}} remains to be the most sought after benchmark in this task. Table {{table:d3bfbdd5-d9e1-49ea-9cb1-e0c2a6d948ed}}  shows the recent trends in the VLN research.


Automatic Spatially-aware Fashion Concept Discovery

Abstract: This paper proposes an automatic spatially-aware concept discovery approach using weakly labeled image-text data from shopping websites. We first fine-tune GoogleNet by jointly modeling clothing images and their corresponding descriptions in a visual-semantic embedding space. Then, for each attribute (word), we generate its spatially-aware representation by combining its semantic word vector representation with its spatial representation derived from the convolutional maps of the fine-tuned network. The resulting spatially-aware representations are further used to cluster attributes into multiple groups to form spatially-aware concepts (e.g., the neckline concept might consist of attributes like v-neck, round-neck, etc). Finally, we decompose the visual-semantic embedding space into multiple concept-specific subspaces, which facilitates structured browsing and attribute-feedback product retrieval by exploiting multimodal linguistic regularities. We conducted extensive experiments on our newly collected Fashion200K dataset, and results on clustering quality evaluation and attribute-feedback product retrieval task demonstrate the effectiveness of our automatically discovered spatially-aware concepts.

SoDeep: A Sorting Deep Net to Learn Ranking Loss Surrogates

Abstract: Several tasks in machine learning are evaluated using non-differentiable metrics such as mean average precision or Spearman correlation. However, their non-differentiability prevents from using them as objective functions in a learning framework. Surrogate and relaxation methods exist but tend to be specific to a given metric. In the present work, we introduce a new method to learn approximations of such non-differentiable objective functions. Our approach is based on a deep architecture that approximates the sorting of arbitrary sets of scores. It is trained virtually for free using synthetic data. This sorting deep (SoDeep) net can then be combined in a plug-and-play manner with existing deep architectures. We demonstrate the interest of our approach in three different tasks that require ranking: Cross-modal text-image retrieval, multi-label image classification and visual memorability ranking. Our approach yields very competitive results on these three tasks, which validates the merit and the flexibility of SoDeep as a proxy for sorting operation in ranking-based losses.

Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks

Abstract: Large-scale pre-training methods of learning cross-modal representations on image-text pairs are becoming popular for vision-language tasks. While existing methods simply concatenate image region features and text features as input to the model to be pre-trained and use self-attention to learn image-text semantic alignments in a brute force manner, in this paper, we propose a new learning method Oscar (Object-Semantics Aligned Pre-training), which uses object tags detected in images as anchor points to significantly ease the learning of alignments. Our method is motivated by the observation that the salient objects in an image can be accurately detected, and are often mentioned in the paired text. We pre-train an Oscar model on the public corpus of 6.5 million text-image pairs, and fine-tune it on downstream tasks, creating new state-of-the-arts on six well-established vision-language understanding and generation tasks.

Latest research in Visual Retrieval (VR)

Recent works in text-to-visual retrieval tasks have emphasized the learning of coherent VisLang representation spaces to obtain precise, meaningful matches (see Table {{table:c7548bd6-a932-48b0-80cf-91a6c7010d26}} ). Two major trends that have recently spanned this entire research area mainly focus on unbiased extraction and feedback based. As user-feedback has been widely used in product searches  {{cite:c7d20dd}}, it has played a vital role in improving the performance in a loop-wise manner. More recently, deep learning frameworks have shifted the focus more from classic ranking or matching algorithms to discovering semantic concepts and cues in both textual and visual spaces  {{cite:c7d20dd}}, {{cite:fadd0b7}}, either independently or combined.


Character-level Convolutional Networks for Text Classification

Abstract: This article offers an empirical exploration on the use of character-level convolutional networks (ConvNets) for text classification. We constructed several large-scale datasets to show that character-level convolutional networks could achieve state-of-the-art or competitive results. Comparisons are offered against traditional models such as bag of words, n-grams and their TFIDF variants, and deep learning models such as word-based ConvNets and recurrent neural networks.

Seeing Stars: Exploiting Class Relationships for Sentiment Categorization with Respect to Rating Scales

Abstract: We address the rating-inference problem, wherein rather than simply decide whether a review is "thumbs up" or "thumbs down", as in previous sentiment analysis work, one must determine an author's evaluation with respect to a multi-point scale (e.g., one to five "stars"). This task represents an interesting twist on standard multi-class text categorization because there are several different degrees of similarity between class labels; for example, "three stars" is intuitively closer to "four stars" than to "one star".We first evaluate human performance at the task. Then, we apply a meta-algorithm, based on a metric labeling formulation of the problem, that alters a given n-ary classifier's output in an explicit attempt to ensure that similar items receive similar labels. We show that the meta-algorithm can provide significant improvements over both multi-class and regression versions of SVMs when we employ a novel similarity measure appropriate to the problem.

Weight Poisoning Attacks on Pre-trained Models

Abstract: Recently, NLP has seen a surge in the usage of large pre-trained models. Users download weights of models pre-trained on large datasets, then fine-tune the weights on a task of their choice. This raises the question of whether downloading untrusted pre-trained weights can pose a security threat. In this paper, we show that it is possible to construct ``weight poisoning'' attacks where pre-trained weights are injected with vulnerabilities that expose ``backdoors'' after fine-tuning, enabling the attacker to manipulate the model prediction simply by injecting an arbitrary keyword. We show that by applying a regularization method, which we call RIPPLe, and an initialization procedure, which we call Embedding Surgery, such attacks are possible even with limited knowledge of the dataset and fine-tuning procedure. Our experiments on sentiment classification, toxicity detection, and spam detection show that this attack is widely applicable and poses a serious threat. Finally, we outline practical defenses against such attacks. Code to reproduce our experiments is available at https://github.com/neulab/RIPPLe.

Satirical News Detection and Analysis using Attention Mechanism and Linguistic Features

Abstract: Satirical news is considered to be entertainment, but it is potentially deceptive and harmful. Despite the embedded genre in the article, not everyone can recognize the satirical cues and therefore believe the news as true news. We observe that satirical cues are often reflected in certain paragraphs rather than the whole document. Existing works only consider document-level features to detect the satire, which could be limited. We consider paragraph-level linguistic features to unveil the satire by incorporating neural network and attention mechanism. We investigate the difference between paragraph-level features and document-level features, and analyze them on a large satirical news dataset. The evaluation shows that the proposed model detects satirical news effectively and reveals what features are important at which level.

Dataset details. #C means number of classes. Len is the average number of words in a sentence.

Datasets. We use 3 conventional text classification datasets: topic classification, sentiment classification, and question type classification. We also use 2 security-critical datasets: hate speech detection and fake news detection. Dataset details are given in Table {{table:5769e5af-89a0-424c-9f46-e5a6e0e2b8c2}} .


Synthetic, U.S. Bureau of Labor Statistics

Synthetic equity evaluation corpus (EEC)

BERT-based perturbations on natural sentences

Assessing Gender Bias in Machine Translation -- A Case Study with Google Translate

Abstract: Recently there has been a growing concern about machine bias, where trained statistical models grow to reflect controversial societal asymmetries, such as gender or racial bias. A significant number of AI tools have recently been suggested to be harmfully biased towards some minority, with reports of racist criminal behavior predictors, Iphone X failing to differentiate between two Asian people and Google photos' mistakenly classifying black people as gorillas. Although a systematic study of such biases can be difficult, we believe that automated translation tools can be exploited through gender neutral languages to yield a window into the phenomenon of gender bias in AI. In this paper, we start with a comprehensive list of job positions from the U.S. Bureau of Labor Statistics (BLS) and used it to build sentences in constructions like "He/She is an Engineer" in 12 different gender neutral languages such as Hungarian, Chinese, Yoruba, and several others. We translate these sentences into English using the Google Translate API, and collect statistics about the frequency of female, male and gender-neutral pronouns in the translated output. We show that GT exhibits a strong tendency towards male defaults, in particular for fields linked to unbalanced gender distribution such as STEM jobs. We ran these statistics against BLS' data for the frequency of female participation in each job position, showing that GT fails to reproduce a real-world distribution of female workers. We provide experimental evidence that even if one does not expect in principle a 50:50 pronominal gender distribution, GT yields male defaults much more frequently than what would be expected from demographic data alone. We are hopeful that this work will ignite a debate about the need to augment current statistical translation tools with debiasing techniques which can already be found in the scientific literature.

On Measuring Gender Bias in Translation of Gender-neutral Pronouns

Abstract: Ethics regarding social bias has recently thrown striking issues in natural language processing. Especially for gender-related topics, the need for a system that reduces the model bias has grown in areas such as image captioning, content recommendation, and automated employment. However, detection and evaluation of gender bias in the machine translation systems are not yet thoroughly investigated, for the task being cross-lingual and challenging to define. In this paper, we propose a scheme for making up a test set that evaluates the gender bias in a machine translation system, with Korean, a language with gender-neutral pronouns. Three word/phrase sets are primarily constructed, each incorporating positive/negative expressions or occupations; all the terms are gender-independent or at least not biased to one side severely. Then, additional sentence lists are constructed concerning formality of the pronouns and politeness of the sentences. With the generated sentence set of size 4,236 in total, we evaluate gender bias in conventional machine translation systems utilizing the proposed measure, which is termed here as translation gender bias index (TGBI). The corpus and the code for evaluation is available on-line.

Automatically Identifying Gender Issues in Machine Translation using Perturbations

Abstract: The successful application of neural methods to machine translation has realized huge quality advances for the community. With these improvements, many have noted outstanding challenges, including the modeling and treatment of gendered language. While previous studies have identified issues using synthetic examples, we develop a novel technique to mine examples from real world data to explore challenges for deployed systems. We use our method to compile an evaluation benchmark spanning examples for four languages from three language families, which we publicly release to facilitate research. The examples in our benchmark expose where model representations are gendered, and the unintended consequences these gendered representations can have in downstream application.

Evaluating Gender Bias in Machine Translation

Abstract: We present the first challenge set and evaluation protocol for the analysis of gender bias in machine translation (MT). Our approach uses two recent coreference resolution datasets composed of English sentences which cast participants into non-stereotypical gender roles (e.g., "The doctor asked the nurse to help her in the operation"). We devise an automatic gender bias evaluation method for eight target languages with grammatical gender, based on morphological analysis (e.g., the use of female inflection for the word "doctor"). Our analyses show that four popular industrial MT systems and two recent state-of-the-art academic MT models are significantly prone to gender-biased translation errors for all tested target languages. Our data and code are made publicly available.

Getting Gender Right in Neural Machine Translation

Abstract: Speakers of different languages must attend to and encode strikingly different aspects of the world in order to use their language correctly (Sapir, 1921; Slobin, 1996). One such difference is related to the way gender is expressed in a language. Saying "I am happy" in English, does not encode any additional knowledge of the speaker that uttered the sentence. However, many other languages do have grammatical gender systems and so such knowledge would be encoded. In order to correctly translate such a sentence into, say, French, the inherent gender information needs to be retained/recovered. The same sentence would become either "Je suis heureux", for a male speaker or "Je suis heureuse" for a female one. Apart from morphological agreement, demographic factors (gender, age, etc.) also influence our use of language in terms of word choices or even on the level of syntactic constructions (Tannen, 1991; Pennebaker et al., 2003). We integrate gender information into NMT systems. Our contribution is two-fold: (1) the compilation of large datasets with speaker information for 20 language pairs, and (2) a simple set of experiments that incorporate gender information into NMT for multiple language pairs. Our experiments show that adding a gender feature to an NMT system significantly improves the translation quality for some language pairs.

For eachStudy, the Table shows on which Benchmarkgender bias is assessed,how Gender is intended (here only in binary (b) terms).Finally, we indicate which (R)epresentational – under-representation and stereotyping– or (A)llocational Harm – as reduced quality of service – is addressed in the study.

To attest the existence and scale of gender bias across several languages, dedicated benchmarks, evaluations, and experiments have been designed.
We first discuss large scale analyses aimed at assessing gender bias in MT,
grouped
according to two main conceptualizations: i) works focusing on the weight of prejudices
and stereotypes in MT ({{formula:d1ed376e-084e-4992-b029-461bbf4367d7}}REF ); ii) studies assessing whether
gender is properly
preserved in translation ({{formula:2bb3f787-afbc-4209-907f-5f1f58533c7e}}REF ).
To keep the connection with the human-centered approach embraced in this survey, in Table {{table:5410a41c-9ee4-44ab-847b-5915b7348811}}  we map each work
to the harms (see {{formula:19fcd43b-24f8-4a8c-90da-4bbec1e42f92}}) ensuing from the biased behaviors they assess.
Finally, we review
existing benchmarks for comparing MT performance across genders
({{formula:6b4da7b4-7135-4e53-958f-57f8e2bbe423}}REF ).