Spatio-temporal Graph Convolutional Neural Network: A Deep Learning Framework for Traffic Forecasting

Abstract: The goal of traffic forecasting is to predict the future vital indicators (such as speed, volume and density) of the local traffic network in reasonable response time. Due to the dynamics and complexity of traffic network flow, typical simulation experiments and classic statistical methods cannot satisfy the requirements of mid-and-long term forecasting. In this work, we propose a novel deep learning framework, Spatio-Temporal Graph Convolutional Neural Network (ST-GCNN), to tackle this spatio-temporal sequence forecasting task. Instead of applying recurrent models to sequence learning, we build our model entirely on convolutional neural networks (CNNs) with gated linear units (GLU) and highway networks. The proposed architecture fully employs the graph structure of the road networks and enables faster training. Experiments show that our ST-GCNN network captures comprehensive spatio-temporal correlations throughout complex traffic network and consistently outperforms state-of-the-art baseline algorithms on several real-world traffic datasets.

GaAN: Gated Attention Networks for Learning on Large and Spatiotemporal Graphs

Abstract: We propose a new network architecture, Gated Attention Networks (GaAN), for learning on graphs. Unlike the traditional multi-head attention mechanism, which equally consumes all attention heads, GaAN uses a convolutional sub-network to control each attention head's importance. We demonstrate the effectiveness of GaAN on the inductive node classification problem. Moreover, with GaAN as a building block, we construct the Graph Gated Recurrent Unit (GGRU) to address the traffic speed forecasting problem. Extensive experiments on three real-world datasets show that our GaAN framework achieves state-of-the-art results on both tasks.

Graph WaveNet for Deep Spatial-Temporal Graph Modeling

Abstract: Spatial-temporal graph modeling is an important task to analyze the spatial relations and temporal trends of components in a system. Existing approaches mostly capture the spatial dependency on a fixed graph structure, assuming that the underlying relation between entities is pre-determined. However, the explicit graph structure (relation) does not necessarily reflect the true dependency and genuine relation may be missing due to the incomplete connections in the data. Furthermore, existing methods are ineffective to capture the temporal trends as the RNNs or CNNs employed in these methods cannot capture long-range temporal sequences. To overcome these limitations, we propose in this paper a novel graph neural network architecture, {Graph WaveNet}, for spatial-temporal graph modeling. By developing a novel adaptive dependency matrix and learn it through node embedding, our model can precisely capture the hidden spatial dependency in the data. With a stacked dilated 1D convolution component whose receptive field grows exponentially as the number of layers increases, Graph WaveNet is able to handle very long sequences. These two components are integrated seamlessly in a unified framework and the whole framework is learned in an end-to-end manner. Experimental results on two public traffic network datasets, METR-LA and PEMS-BAY, demonstrate the superior performance of our algorithm.

GMAN: A Graph Multi-Attention Network for Traffic Prediction

Abstract: Long-term traffic prediction is highly challenging due to the complexity of traffic systems and the constantly changing nature of many impacting factors. In this paper, we focus on the spatio-temporal factors, and propose a graph multi-attention network (GMAN) to predict traffic conditions for time steps ahead at different locations on a road network graph. GMAN adapts an encoder-decoder architecture, where both the encoder and the decoder consist of multiple spatio-temporal attention blocks to model the impact of the spatio-temporal factors on traffic conditions. The encoder encodes the input traffic features and the decoder predicts the output sequence. Between the encoder and the decoder, a transform attention layer is applied to convert the encoded traffic features to generate the sequence representations of future time steps as the input of the decoder. The transform attention mechanism models the direct relationships between historical and future time steps that helps to alleviate the error propagation problem among prediction time steps. Experimental results on two real-world traffic prediction tasks (i.e., traffic volume prediction and traffic speed prediction) demonstrate the superiority of GMAN. In particular, in the 1 hour ahead prediction, GMAN outperforms state-of-the-art methods by up to 4% improvement in MAE measure. The source code is available at https://github.com/zhengchuanpan/GMAN.

Connecting the Dots: Multivariate Time Series Forecasting with Graph Neural Networks

Abstract: Modeling multivariate time series has long been a subject that has attracted researchers from a diverse range of fields including economics, finance, and traffic. A basic assumption behind multivariate time series forecasting is that its variables depend on one another but, upon looking closely, it is fair to say that existing methods fail to fully exploit latent spatial dependencies between pairs of variables. In recent years, meanwhile, graph neural networks (GNNs) have shown high capability in handling relational dependencies. GNNs require well-defined graph structures for information propagation which means they cannot be applied directly for multivariate time series where the dependencies are not known in advance. In this paper, we propose a general graph neural network framework designed specifically for multivariate time series data. Our approach automatically extracts the uni-directed relations among variables through a graph learning module, into which external knowledge like variable attributes can be easily integrated. A novel mix-hop propagation layer and a dilated inception layer are further proposed to capture the spatial and temporal dependencies within the time series. The graph learning, graph convolution, and temporal convolution modules are jointly learned in an end-to-end framework. Experimental results show that our proposed model outperforms the state-of-the-art baseline methods on 3 of 4 benchmark datasets and achieves on-par performance with other approaches on two traffic datasets which provide extra structural information.

Adaptive Graph Convolutional Recurrent Network for Traffic Forecasting

Abstract: Modeling complex spatial and temporal correlations in the correlated time series data is indispensable for understanding the traffic dynamics and predicting the future status of an evolving traffic system. Recent works focus on designing complicated graph neural network architectures to capture shared patterns with the help of pre-defined graphs. In this paper, we argue that learning node-specific patterns is essential for traffic forecasting while the pre-defined graph is avoidable. To this end, we propose two adaptive modules for enhancing Graph Convolutional Network (GCN) with new capabilities: 1) a Node Adaptive Parameter Learning (NAPL) module to capture node-specific patterns; 2) a Data Adaptive Graph Generation (DAGG) module to infer the inter-dependencies among different traffic series automatically. We further propose an Adaptive Graph Convolutional Recurrent Network (AGCRN) to capture fine-grained spatial and temporal correlations in traffic series automatically based on the two modules and recurrent networks. Our experiments on two real-world traffic datasets show AGCRN outperforms state-of-the-art by a significant margin without pre-defined graphs about spatial connections.

Multi-Range Attentive Bicomponent Graph Convolutional Network for Traffic Forecasting

Abstract: Traffic forecasting is of great importance to transportation management and public safety, and very challenging due to the complicated spatial-temporal dependency and essential uncertainty brought about by the road network and traffic conditions. Latest studies mainly focus on modeling the spatial dependency by utilizing graph convolutional networks (GCNs) throughout a fixed weighted graph. However, edges, i.e., the correlations between pair-wise nodes, are much more complicated and interact with each other. In this paper, we propose the Multi-Range Attentive Bicomponent GCN (MRA-BGCN), a novel deep learning model for traffic forecasting. We first build the node-wise graph according to the road network distance and the edge-wise graph according to various edge interaction patterns. Then, we implement the interactions of both nodes and edges using bicomponent graph convolution. The multi-range attention mechanism is introduced to aggregate information in different neighborhood ranges and automatically learn the importance of different ranges. Extensive experiments on two real-world road network traffic datasets, METR-LA and PEMS-BAY, show that our MRA-BGCN achieves the state-of-the-art results.

Discrete Graph Structure Learning for Forecasting Multiple Time Series

Abstract: Time series forecasting is an extensively studied subject in statistics, economics, and computer science. Exploration of the correlation and causation among the variables in a multivariate time series shows promise in enhancing the performance of a time series model. When using deep neural networks as forecasting models, we hypothesize that exploiting the pairwise information among multiple (multivariate) time series also improves their forecast. If an explicit graph structure is known, graph neural networks (GNNs) have been demonstrated as powerful tools to exploit the structure. In this work, we propose learning the structure simultaneously with the GNN if the graph is unknown. We cast the problem as learning a probabilistic graph model through optimizing the mean performance over the graph distribution. The distribution is parameterized by a neural network so that discrete graphs can be sampled differentiably through reparameterization. Empirical evaluations show that our method is simpler, more efficient, and better performing than a recently proposed bilevel learning approach for graph structure learning, as well as a broad array of forecasting models, either deep or non-deep learning based, and graph or non-graph based.

Dynamic Graph Convolutional Recurrent Network for Traffic Prediction: Benchmark and Solution

Abstract: Traffic prediction is the cornerstone of intelligent transportation system. Accurate traffic forecasting is essential for the applications of smart cities, i.e., intelligent traffic management and urban planning. Although various methods are proposed for spatio-temporal modeling, they ignore the dynamic characteristics of correlations among locations on road network. Meanwhile, most Recurrent Neural Network based works are not efficient enough due to their recurrent operations. Additionally, there is a severe lack of fair comparison among different methods on the same datasets. To address the above challenges, in this article, we propose a novel traffic prediction framework, named Dynamic Graph Convolutional Recurrent Network (DGCRN). In DGCRN, hyper-networks are designed to leverage and extract dynamic characteristics from node attributes, while the parameters of dynamic filters are generated at each time step. We filter the node embeddings and then use them to generate dynamic graph, which is integrated with pre-defined static graph. As far as we know, we are first to employ a generation method to model fine topology of dynamic graph at each time step. Furthermore, to enhance efficiency and performance, we employ a training strategy for DGCRN by restricting the iteration number of decoder during forward and backward propagation. Finally, a reproducible standardized benchmark and a brand new representative traffic dataset are opened for fair comparison and further research. Extensive experiments on three datasets demonstrate that our model outperforms 15 baselines consistently. Source codes are available at https://github.com/tsinghua-fib-lab/Traffic-Benchmark.

Spatial-Temporal Fusion Graph Neural Networks for Traffic Flow Forecasting

Abstract: Spatial-temporal data forecasting of traffic flow is a challenging task because of complicated spatial dependencies and dynamical trends of temporal pattern between different roads. Existing frameworks usually utilize given spatial adjacency graph and sophisticated mechanisms for modeling spatial and temporal correlations. However, limited representations of given spatial graph structure with incomplete adjacent connections may restrict effective spatial-temporal dependencies learning of those models. Furthermore, existing methods were out at elbows when solving complicated spatial-temporal data: they usually utilize separate modules for spatial and temporal correlations, or they only use independent components capturing localized or global heterogeneous dependencies. To overcome those limitations, our paper proposes a novel Spatial-Temporal Fusion Graph Neural Networks (STFGNN) for traffic flow forecasting. First, a data-driven method of generating “temporal graph” is proposed to compensate several genuine correlations that spatial graph may not reflect. STFGNN could effectively learn hidden spatial-temporal dependencies by a novel fusion operation of various spatial and temporal graphs, treated for different time periods in parallel. Meanwhile, by integrating this fusion graph module and a novel gated convolution module into a unified layer parallelly, STFGNN could handle long sequences by learning more spatial-temporal dependencies with layers stacked. Experimental results on several public traffic datasets demonstrate that our method achieves state-of-the-art performance consistently than other baselines.

Coupled Layer-wise Graph Convolution for Transportation Demand Prediction

Abstract: Graph Convolutional Network (GCN) has been widely applied in transportation demand prediction due to its excellent ability to capture non-Euclidean spatial dependence among station-level or regional transportation demands. However, in most of the existing research, the graph convolution was implemented on a heuristically generated adjacency matrix, which could neither reflect the real spatial relationships of stations accurately, nor capture the multi-level spatial dependence of demands adaptively. To cope with the above problems, this paper provides a novel graph convolutional network for transportation demand prediction. Firstly, a novel graph convolution architecture is proposed, which has different adjacency matrices in different layers and all the adjacency matrices are self-learned during the training process. Secondly, a layer-wise coupling mechanism is provided, which associates the upper-level adjacency matrix with the lower-level one. It also reduces the scale of parameters in our model. Lastly, a unitary network is constructed to give the final prediction result by integrating the hidden spatial states with gated recurrent unit, which could capture the multi-level spatial dependence and temporal dynamics simultaneously. Experiments have been conducted on two real-world datasets, NYC Citi Bike and NYC Taxi, and the results demonstrate the superiority of our model over the state-of-the-art ones.

Spatial-Temporal Graph ODE Networks for Traffic Flow Forecasting

Abstract: Spatial-temporal forecasting has attracted tremendous attention in a wide range of applications, and traffic flow prediction is a canonical and typical example. The complex and long-range spatial-temporal correlations of traffic flow bring it to a most intractable challenge. Existing works typically utilize shallow graph convolution networks (GNNs) and temporal extracting modules to model spatial and temporal dependencies respectively. However, the representation ability of such models is limited due to: (1) shallow GNNs are incapable to capture long-range spatial correlations, (2) only spatial connections are considered and a mass of semantic connections are ignored, which are of great importance for a comprehensive understanding of traffic networks. To this end, we propose Spatial-Temporal Graph Ordinary Differential Equation Networks (STGODE).1 Specifically, we capture spatial-temporal dynamics through a tensor-based ordinary differential equation (ODE), as a result, deeper networks can be constructed and spatial-temporal features are utilized synchronously. To understand the network more comprehensively, semantical adjacency matrix is considered in our model, and a well-design temporal dialated convolution structure is used to capture long term temporal dependencies. We evaluate our model on multiple real-world traffic datasets and superior performance is achieved over state-of-the-art baselines.

Regularized Graph Structure Learning with Semantic Knowledge for Multi-variates Time-Series Forecasting

Abstract: Multivariate time-series forecasting is a critical task for many applications, and graph time-series network is widely studied due to its capability to capture the spatial-temporal correlation simultaneously. However, most existing works focus more on learning with the explicit prior graph structure, while ignoring potential information from the implicit graph structure, yielding incomplete structure modeling. Some recent works attempts to learn the intrinsic or implicit graph structure directly, while lacking a way to combine explicit prior structure with implicit structure together. In this paper, we propose Regularized Graph Structure Learning (RGSL) model to incorporate both explicit prior structure and implicit structure together, and learn the forecasting deep networks along with the graph structure. RGSL consists of two innovative modules. First, we derive an implicit dense similarity matrix through node embedding, and learn the sparse graph structure using the Regularized Graph Generation (RGG) based on the Gumbel Softmax trick. Second, we propose a Laplacian Matrix Mixed-up Module (LM3) to fuse the explicit graph and implicit graph together. We conduct experiments on three real-word datasets. Results show that the proposed RGSL model outperforms existing graph forecasting algorithms with a notable margin, while learning meaningful graph structure simultaneously. Our code and models are made publicly available at https://github.com/alipay/RGSL.git.

Learning the Evolutionary and Multi-scale Graph Structure for Multivariate Time Series Forecasting

Abstract: Recent studies have shown great promise in applying graph neural networks for multivariate time series forecasting, where the interactions of time series are described as a graph structure and the variables are represented as the graph nodes. Along this line, existing methods usually assume that the graph structure (or the adjacency matrix), which determines the aggregation manner of graph neural network, is fixed either by definition or self-learning. However, the interactions of variables can be dynamic and evolutionary in real-world scenarios. Furthermore, the interactions of time series are quite different if they are observed at different time scales. To equip the graph neural network with a flexible and practical graph structure, in this paper, we investigate how to model the evolutionary and multi-scale interactions of time series. In particular, we first provide a hierarchical graph structure cooperated with the dilated convolution to capture the scale-specific correlations among time series. Then, a series of adjacency matrices are constructed under a recurrent manner to represent the evolving correlations at each layer. Moreover, a unified neural network is provided to integrate the components above to get the final prediction. In this way, we can capture the pair-wise correlations and temporal dependency simultaneously. Finally, experiments on both single-step and multi-step forecasting tasks demonstrate the superiority of our method over the state-of-the-art approaches.

Pre-training Enhanced Spatial-temporal Graph Neural Network for Multivariate Time Series Forecasting

Abstract: Multivariate Time Series (MTS) forecasting plays a vital role in a wide range of applications. Recently, Spatial-Temporal Graph Neural Networks (STGNNs) have become increasingly popular MTS forecasting methods. STGNNs jointly model the spatial and temporal patterns of MTS through graph neural networks and sequential models, significantly improving the prediction accuracy. But limited by model complexity, most STGNNs only consider short-term historical MTS data, such as data over the past one hour. However, the patterns of time series and the dependencies between them (i.e., the temporal and spatial patterns) need to be analyzed based on long-term historical MTS data. To address this issue, we propose a novel framework, in which STGNN is Enhanced by a scalable time series Pre-training model (STEP). Specifically, we design a pre-training model to efficiently learn temporal patterns from very long-term history time series (e.g., the past two weeks) and generate segment-level representations. These representations provide contextual information for short-term time series input to STGNNs and facilitate modeling dependencies between time series. Experiments on three public real-world datasets demonstrate that our framework is capable of significantly enhancing downstream STGNNs, and our pre-training model aptly captures temporal patterns.

Graph Neural Controlled Differential Equations for Traffic Forecasting

Abstract: Traffic forecasting is one of the most popular spatio-temporal tasks in the field of machine learning. A prevalent approach in the field is to combine graph convolutional networks and recurrent neural networks for the spatio-temporal processing. There has been fierce competition and many novel methods have been proposed. In this paper, we present the method of spatio-temporal graph neural controlled differential equation (STG-NCDE). Neural controlled differential equations (NCDEs) are a breakthrough concept for processing sequential data. We extend the concept and design two NCDEs: one for the temporal processing and the other for the spatial processing. After that, we combine them into a single framework. We conduct experiments with 6 benchmark datasets and 20 baselines. STG-NCDE shows the best accuracy in all cases, outperforming all those 20 baselines by non-trivial margins.

Summary of Representative Models for Spatial and Temporal Modeling Techniques.

Just like the No-Free-Lunch theorem {{cite:88685fc}} in machine learning, using STGNNs to model the spatial-temporal correlation of traffic scenes requires selecting appropriate components according to specific problems and conditions. Here we summarize some representative research works, as shown in Table {{table:6213e269-ad0d-4e9e-a2f9-63ac463f3162}} . Some of these methods had reached the state-of-the-art in prediction tasks. STGCN {{cite:33602d0}} first combines gated temporal convolution with ChebNet operator for spatial-temporal prediction and achieves better performance than traditional time series models in all metrics. DCRNN {{cite:68a3908}} extends the diffusion graph convolution to a recurrent neural network of encoder-decoder to solve the directional problem of asymmetric traffic graph propagation. Graph WaveNet {{cite:97c1459}} innovatively proposes an adaptive graph dissipation of biases caused by human-defined spatial relations, and WaveNet based on causal convolution is used to learn temporal relations. Inspired by ST-ResNet {{cite:b6e378a}}, ASTGCN {{cite:3f1b11c}} proposes a set of temporal components called Clonesss, Period and Trend. Structurally, it utilizes a skip connection to connect the spatial-temporal attention layer to the convolutional layer to form model branchs, and finally fuses the three components together for prediction. These early studies of spatio-temporal graph neural networks significantly outperform traditional statistical methods and machine learning models in terms of predictive performance.


The images are from the Internet. Some images are collected from the scenarios of fighting against COVID-19.

Over half of the images are collected from WIDER Face, MAFA, RWMFD. “With mask" class requires faces with nose and mouth covered.

The images are from MAFA and Wider Face datasets. The annotations with a list of images publicly available are provided.

10471 faces without masks; 7695 correct masked faces; 366 incorrect masked faces

32012 faces without masks; 29532 correct masked faces; 1528 incorrect masked faces

Masked Face Recognition Dataset and Application

Abstract: In order to effectively prevent the spread of COVID-19 virus, almost everyone wears a mask during coronavirus epidemic. This almost makes conventional facial recognition technology ineffective in many cases, such as community access control, face access control, facial attendance, facial security checks at train stations, etc. Therefore, it is very urgent to improve the recognition performance of the existing face recognition technology on the masked faces. Most current advanced face recognition approaches are designed based on deep learning, which depend on a large number of face samples. However, at present, there are no publicly available masked face recognition datasets. To this end, this work proposes three types of masked face datasets, including Masked Face Detection Dataset (MFDD), Real-world Masked Face Recognition Dataset (RMFRD) and Simulated Masked Face Recognition Dataset (SMFRD). Among them, to the best of our knowledge, RMFRD is currently theworld's largest real-world masked face dataset. These datasets are freely available to industry and academia, based on which various applications on masked faces can be developed. The multi-granularity masked face recognition model we developed achieves 95% accuracy, exceeding the results reported by the industry. Our datasets are available at: https://github.com/X-zhangyang/Real-World-Masked-Face-Dataset.

Real-Time Face Mask Detection Method Based on YOLOv3

Abstract: The rapid outbreak of COVID-19 has caused serious harm and infected tens of millions of people worldwide. Since there is no specific treatment, wearing masks has become an effective method to prevent the transmission of COVID-19 and is required in most public areas, which has also led to a growing demand for automatic real-time mask detection services to replace manual reminding. However, few studies on face mask detection are being conducted. It is urgent to improve the performance of mask detectors. In this paper, we proposed the Properly Wearing Masked Face Detection Dataset (PWMFD), which included 9205 images of mask wearing samples with three categories. Moreover, we proposed Squeeze and Excitation (SE)-YOLOv3, a mask detector with relatively balanced effectiveness and efficiency. We integrated the attention mechanism by introducing the SE block into Darknet53 to obtain the relationships among channels so that the network can focus more on the important feature. We adopted GIoUloss, which can better describe the spatial difference between predicted and ground truth boxes to improve the stability of bounding box regression. Focal loss was utilized for solving the extreme foreground-background class imbalance. Besides, we performed corresponding image augmentation techniques to further improve the robustness of the model on the specific task. Experimental results showed that SE-YOLOv3 outperformed YOLOv3 and other state-of-the-art detectors on PWMFD and achieved a higher 8.6% mAP compared to YOLOv3 while having a comparable detection speed.

How to Correctly Detect Face-Masks for COVID-19 from Visual Information?

Abstract: The new Coronavirus disease (COVID-19) has seriously affected the world. By the end of November 2020, the global number of new coronavirus cases had already exceeded 60 million and the number of deaths 1,410,378 according to information from the World Health Organization (WHO). To limit the spread of the disease, mandatory face-mask rules are now becoming common in public settings around the world. Additionally, many public service providers require customers to wear face-masks in accordance with predefined rules (e.g., covering both mouth and nose) when using public services. These developments inspired research into automatic (computer-vision-based) techniques for face-mask detection that can help monitor public behavior and contribute towards constraining the COVID-19 pandemic. Although existing research in this area resulted in efficient techniques for face-mask detection, these usually operate under the assumption that modern face detectors provide perfect detection performance (even for masked faces) and that the main goal of the techniques is to detect the presence of face-masks only. In this study, we revisit these common assumptions and explore the following research questions: (i) How well do existing face detectors perform with masked-face images? (ii) Is it possible to detect a proper (regulation-compliant) placement of facial masks? and iii) How useful are existing face-mask detection techniques for monitoring applications during the COVID-19 pandemic? To answer these and related questions we conduct a comprehensive experimental evaluation of several recent face detectors for their performance with masked-face images. Furthermore, we investigate the usefulness of multiple off-the-shelf deep-learning models for recognizing correct face-mask placement. Finally, we design a complete pipeline for recognizing whether face-masks are worn correctly or not and compare the performance of the pipeline with standard face-mask detection models from the literature. To facilitate the study, we compile a large dataset of facial images from the publicly available MAFA and Wider Face datasets and annotate it with compliant and non-compliant labels. The annotation dataset, called Face-Mask-Label Dataset (FMLD), is made publicly available to the research community.

Detailed Descriptions of Some Open Datasets for Masked Facial Detection.

In summary, detailed information for above mentioned datasets is illustrated in Table {{table:d3f69b4a-d0e0-4c89-8f44-c9b4a1ef9b3e}} . The corresponding available links are also provided in Table {{table:8072f493-afb1-45e5-b5cf-6ead631c2b4b}} . All the links had been verified to be effective before May 10, 2021.

It's also notable from Table {{table:d3f69b4a-d0e0-4c89-8f44-c9b4a1ef9b3e}}  that nine of thirteen datasets are constructed by real-world images. MFDD {{cite:01dc871}} and Dey's dataset {{cite:1b0867f}} include both real and simulated images. For MFNID {{cite:0b5ae79}} and SMFD {{cite:8b77eb1}} datasets, the masked faces are created entirely by simulating images. Some samples are given in Fig. {{figure:c32c3391-f8cc-4aa8-bfba-2c68bde9d0cb}} . Only one type of mask is used to synthesize masked faces in MFNID or SMFD.

It can be clearly concluded from the Table {{table:ac5de3ac-7441-4b8b-8150-059cd0f7cbfa}}  that methods {{cite:c89287b}}, {{cite:1b0867f}} achieve higher {{formula:e4f0d1ca-9fcc-4060-ba0d-d69cfd605425}}  and {{formula:764992b1-bc58-4f14-83a1-c6ba94a58511}}  values on the given datasets. This also verifies the description in Table {{table:d3f69b4a-d0e0-4c89-8f44-c9b4a1ef9b3e}}  that scenes of UFMD and Dey's datasets are simple. The method {{cite:3384c94}} largely gets benefit from two deep neural networks: RetinaFace and ResNet-152. Other datasets such as MAFA, Moxa3K, PWMFD, Singh's, WMD are challenging in terms of quantitative results. As a consequence, these results can be treated as benchmarks for future comparison.


Leveraging Video Descriptions to Learn Video Question Answering

Abstract: We propose a scalable approach to learn video-based question answering (QA): to answer a free-form natural language question about the contents of a video. Our approach automatically harvests a large number of videos and descriptions freely available online. Then, a large number of candidate QA pairs are automatically generated from descriptions rather than manually annotated. Next, we use these candidate QA pairs to train a number of video-based QA methods extended from MN (Sukhbaatar et al. 2015), VQA (Antol et al. 2015), SA (Yao et al. 2015), and SS (Venugopalan et al. 2015). In order to handle non-perfect candidate QA pairs, we propose a self-paced learning procedure to iteratively identify them and mitigate their effects in training. Finally, we evaluate performance on manually generated video-based QA pairs. The results show that our self-paced learning procedure is effective, and the extended SS model outperforms various baselines.

TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answering

Abstract: Vision and language understanding has emerged as a subject undergoing intense study in Artificial Intelligence. Among many tasks in this line of research, visual question answering (VQA) has been one of the most successful ones, where the goal is to learn a model that understands visual content at region-level details and finds their associations with pairs of questions and answers in the natural language form. Despite the rapid progress in the past few years, most existing work in VQA have focused primarily on images. In this paper, we focus on extending VQA to the video domain and contribute to the literature in three important ways. First, we propose three new tasks designed specifically for video VQA, which require spatio-temporal reasoning from videos to answer questions correctly. Next, we introduce a new large-scale dataset for video VQA named TGIF-QA that extends existing VQA work with our new tasks. Finally, we propose a dual-LSTM based approach with both spatial and temporal attention, and show its effectiveness over conventional VQA techniques through empirical evaluations.

ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answering

Abstract: Recent developments in modeling language and vision have been successfully applied to image question answering. It is both crucial and natural to extend this research direction to the video domain for video question answering (VideoQA). Compared to the image domain where large scale and fully annotated benchmark datasets exists, VideoQA datasets are limited to small scale and are automatically generated, etc. These limitations restrict their applicability in practice. Here we introduce ActivityNet-QA, a fully annotated and large scale VideoQA dataset. The dataset consists of 58,000 QA pairs on 5,800 complex web videos derived from the popular ActivityNet dataset. We present a statistical analysis of our ActivityNet-QA dataset and conduct extensive experiments on it by comparing existing VideoQA baselines. Moreover, we explore various video representation strategies to improve VideoQA performance, especially for long videos.

MovieQA: Understanding Stories in Movies through Question-Answering

Abstract: We introduce the MovieQA dataset which aims to evaluate automatic story comprehension from both video and text. The dataset consists of 14,944 questions about 408 movies with high semantic diversity. The questions range from simpler "Who" did "What" to "Whom", to "Why" and "How" certain events occurred. Each question comes with a set of five possible answers, a correct one and four deceiving answers provided by human annotators. Our dataset is unique in that it contains multiple sources of information - video clips, plots, subtitles, scripts, and DVS. We analyze our data through various statistics and methods. We further extend existing QA techniques to show that question-answering with such open-ended semantics is hard. We make this data set public along with an evaluation benchmark to encourage inspiring work in this challenging domain.

A Dataset and Exploration of Models for Understanding Video Data through Fill-in-the-Blank Question-Answering

Abstract: While deep convolutional neural networks frequently approach or exceed human-level performance in benchmark tasks involving static images, extending this success to moving images is not straightforward. Video understanding is of interest for many applications, including content recommendation, prediction, summarization, event/object detection, and understanding human visual perception. However, many domains lack sufficient data to explore and perfect video models. In order to address the need for a simple, quantitative benchmark for developing and understanding video, we present MovieFIB, a fill-in-the-blank question-answering dataset with over 300,000 examples, based on descriptive video annotations for the visually impaired. In addition to presenting statistics and a description of the dataset, we perform a detailed analysis of 5 different models predictions, and compare these with human performance. We investigate the relative importance of language, static (2D) visual features, and moving (3D) visual features, the effects of increasing dataset size, the number of frames sampled, and of vocabulary size. We illustrate that: this task is not solvable by a language model alone, our model combining 2D and 3D visual information indeed provides the best result, all models perform significantly worse than human-level. We provide human evaluation for responses given by different models and find that accuracy on the MovieFIB evaluation corresponds well with human judgment. We suggest avenues for improving video models, and hope that the MovieFIB challenge can be useful for measuring and encouraging progress in this very interesting field.

TVQA: Localized, Compositional Video Question Answering

Abstract: Recent years have witnessed an increasing interest in image-based question-answering (QA) tasks. However, due to data limitations, there has been much less work on video-based QA. In this paper, we present TVQA, a large-scale video QA dataset based on 6 popular TV shows. TVQA consists of 152,545 QA pairs from 21,793 clips, spanning over 460 hours of video. Questions are designed to be compositional in nature, requiring systems to jointly localize relevant moments within a clip, comprehend subtitle-based dialogue, and recognize relevant visual concepts. We provide analyses of this new dataset as well as several baselines and a multi-stream end-to-end trainable neural network framework for the TVQA task. The dataset is publicly available at http://tvqa.cs.unc.edu.

Just Ask: Learning to Answer Questions from Millions of Narrated Videos

Abstract: Recent methods for visual question answering rely on large-scale annotated datasets. Manual annotation of questions and answers for videos, however, is tedious, expensive and prevents scalability. In this work, we propose to avoid manual annotation and generate a large-scale training dataset for video question answering making use of automatic cross-modal supervision. We leverage a question generation transformer trained on text data and use it to generate question-answer pairs from transcribed video narrations. Given narrated videos, we then automatically generate the HowToVQA69M dataset with 69M video-question-answer triplets. To handle the open vocabulary of diverse answers in this dataset, we propose a training procedure based on a contrastive loss between a video-question multi-modal transformer and an answer transformer. We introduce the zero-shot VideoQA task and show excellent results, in particular for rare answers. Furthermore, we demonstrate our method to significantly outperform the state of the art on MSRVTT-QA, MSVD-QA, ActivityNet-QA and How2QA. Finally, for a detailed evaluation we introduce iVQA, a new VideoQA dataset with reduced language biases and high-quality redundant manual annotations.

CLEVRER: CoLlision Events for Video REpresentation and Reasoning

Abstract: The ability to reason about temporal and causal events from videos lies at the core of human intelligence. Most video reasoning benchmarks, however, focus on pattern recognition from complex visual and language input, instead of on causal structure. We study the complementary problem, exploring the temporal and causal structures behind videos of objects with simple visual appearance. To this end, we introduce the CoLlision Events for Video REpresentation and Reasoning (CLEVRER), a diagnostic video dataset for systematic evaluation of computational models on a wide range of reasoning tasks. Motivated by the theory of human casual judgment, CLEVRER includes four types of questions: descriptive (e.g., "what color"), explanatory ("what is responsible for"), predictive ("what will happen next"), and counterfactual ("what if"). We evaluate various state-of-the-art models for visual reasoning on our benchmark. While these models thrive on the perception-based task (descriptive), they perform poorly on the causal tasks (explanatory, predictive and counterfactual), suggesting that a principled approach for causal reasoning should incorporate the capability of both perceiving complex visual and language inputs, and understanding the underlying dynamics and causal relations. We also study an oracle model that explicitly combines these components via symbolic representations.

AGQA: A Benchmark for Compositional Spatio-Temporal Reasoning

Abstract: Visual events are a composition of temporal actions involving actors spatially interacting with objects. When developing computer vision models that can reason about compositional spatio-temporal events, we need benchmarks that can analyze progress and uncover shortcomings. Existing video question answering benchmarks are useful, but they often conflate multiple sources of error into one accuracy metric and have strong biases that models can exploit, making it difficult to pinpoint model weaknesses. We present Action Genome Question Answering (AGQA), a new benchmark for compositional spatio-temporal reasoning. AGQA contains 192M unbalanced question answer pairs for 9.6K videos. We also provide a balanced subset of 3.9M question answer pairs, 3 orders of magnitude larger than existing benchmarks, that minimizes bias by balancing the answer distributions and types of question structures. Although human evaluators marked 86.02% of our question-answer pairs as correct, the best model achieves only 47.74% accuracy. In addition, AGQA introduces multiple training/test splits to test for various reasoning abilities, including generalization to novel compositions, to indirect references, and to more compositional steps. Using AGQA, we evaluate modern visual reasoning systems, demonstrating that the best models barely perform better than non-visual baselines exploiting linguistic biases and that none of the existing models generalize to novel compositions unseen during training.

NExT-QA: Next Phase of Question-Answering to Explaining Temporal Actions

Abstract: We introduce NExT-QA, a rigorously designed video question answering (VideoQA) benchmark to advance video understanding from describing to explaining the temporal actions. Based on the dataset, we set up multi-choice and open-ended QA tasks targeting causal action reasoning, temporal action reasoning, and common scene comprehension. Through extensive analysis of baselines and established VideoQA techniques, we find that top-performing methods excel at shallow scene descriptions but are weak in causal and temporal action reasoning. Furthermore, the models that are effective on multi-choice QA, when adapted to open-ended QA, still struggle in generalizing the answers. This raises doubt on the ability of these models to reason and highlights possibilities for improvement. With detailed results for different question types and heuristic observations for future works, we hope NExT-QA will guide the next generation of VQA research to go beyond superficial description towards a deeper understanding of videos. (The dataset and related resources are available at https://github.com/doc-doc/NExT-QA.git).

VideoQA Datasets. (MB: Modality-based Taxonomy, QB: Question-based Taxonomy, Vid: VideoQA, MM: Multi-modal VideoQA, KB: Knowledge-based VideoQA, F: Factoid VideoQA, I: Inference VideoQA, Auto: automatic annotation, Man: manually annotation, MC: multi-choice QA, OE: open-ended QA.)

Datasets Analysis. The timeline of some monumental VideoQA datasets are shown in Figure {{figure:64655e1b-af1c-44a5-ade9-f16f24b31427}} , and the details are listed in Table {{table:51a50c38-0f68-4255-a7f8-25187374d991}} . VideoQA and MM VideoQA almost appear simultaneously, and have been studied separately by the community. Despite the unique challenges of MM VideoQA in reasoning on multiple modalities {{cite:5b234ee}}, algorithms targeting VideoQA and MM videoQA can share similar spirits. Actually modality-based taxonomy stems from research preference for video domains. While question-based taxonomy, is affected more by the methodological considerations, since the recently proposed Inference VideoQA brings new technical challenges, which is driving artificial intelligence towards new heights, not just limited to learning the correlations in data.


A General Framework for Adversarial Examples with Objectives

Abstract: Images perturbed subtly to be misclassified by neural networks, called adversarial examples, have emerged as a technically deep challenge and an important concern for several application domains. Most research on adversarial examples takes as its only constraint that the perturbed images are similar to the originals. However, real-world application of these ideas often requires the examples to satisfy additional objectives, which are typically enforced through custom modifications of the perturbation process. In this article, we propose adversarial generative nets (AGNs), a general methodology to train a generator neural network to emit adversarial examples satisfying desired objectives. We demonstrate the ability of AGNs to accommodate a wide range of objectives, including imprecise ones difficult to model, in two application domains. In particular, we demonstrate physical adversarial examples—eyeglass frames designed to fool face recognition—with better robustness, inconspicuousness, and scalability than previous approaches, as well as a new attack to fool a handwritten-digit classifier.

On adversarial patches: real-world attack on ArcFace-100 face recognition system

Abstract: Recent works showed the vulnerability of image classifiers to adversarial attacks in the digital domain. However, the majority of attacks involve adding small perturbation to an image to fool the classifier. Unfortunately, such procedures can not be used to conduct a real-world attack, where adding an adversarial attribute to the photo is a more practical approach. In this paper, we study the problem of real-world attacks on face recognition systems. We examine security of one of the best public face recognition systems, LResNet100E-IR with ArcFace loss, and propose a simple method to attack it in the physical world. The method suggests creating an adversarial patch that can be printed, added as a face attribute and photographed; the photo of a person with such attribute is then passed to the classifier such that the classifier's recognized class changes from correct to the desired one. Proposed generating procedure allows projecting adversarial patches not only on different areas of the face, such as nose or forehead but also on some wearable accessory, such as eyeglasses.

AdvHat: Real-world adversarial attack on ArcFace Face ID system

Abstract: In this paper we propose a novel easily reproducible technique to attack the best public Face ID system ArcFace in different shooting conditions. To create an attack, we print the rectangular paper sticker on a common color printer and put it on the hat. The adversarial sticker is prepared with a novel algorithm for off-plane transformations of the image which imitates sticker location on the hat. Such an approach confuses the state-of-the-art public Face ID model LResNet100E-IR, ArcFace@ms1m-refine-v2 and is transferable to other Face ID models.

Adversarial Light Projection Attacks on Face Recognition Systems: A Feasibility Study

Abstract: Deep learning-based systems have been shown to be vulnerable to adversarial attacks in both digital and physical domains. While feasible, digital attacks have limited applicability in attacking deployed systems, including face recognition systems, where an adversary typically has access to the input and not the transmission channel. In such setting, physical attacks that directly provide a malicious input through the input channel pose a bigger threat. We investigate the feasibility of conducting real-time physical attacks on face recognition systems using adversarial light projections. A setup comprising a commercially available web camera and a projector is used to conduct the attack. The adversary uses a transformation-invariant adversarial pattern generation method to generate a digital adversarial pattern using one or more images of the target available to the adversary. The digital adversarial pattern is then projected onto the adversary's face in the physical domain to either impersonate a target (impersonation) or evade recognition (obfuscation). We conduct preliminary experiments using two open-source and one commercial face recognition system on a pool of 50 subjects. Our experimental results demonstrate the vulnerability of face recognition systems to light projection attacks in both white-box and black-box attack settings.

Improving Transferability of Adversarial Patches on Face Recognition with Generative Models

Abstract: Face recognition is greatly improved by deep convolutional neural networks (CNNs). Recently, these face recognition models have been used for identity authentication in security sensitive applications. However, deep CNNs are vulnerable to adversarial patches, which are physically realizable and stealthy, raising new security concerns on the real-world applications of these models. In this paper, we evaluate the robustness of face recognition models using adversarial patches based on transferability, where the attacker has limited accessibility to the target models. First, we extend the existing transfer-based attack techniques to generate transferable adversarial patches. However, we observe that the transferability is sensitive to initialization and degrades when the perturbation magnitude is large, indicating the overfitting to the substitute models. Second, we propose to regularize the adversarial patches on the low dimensional data manifold. The manifold is represented by generative models pre-trained on legitimate human face images. Using face-like features as adversarial perturbations through optimization on the manifold, we show that the gaps between the responses of substitute models and the target models dramatically decrease, exhibiting a better transferability. Extensive digital world experiments are conducted to demonstrate the superiority of the proposed method in the black-box setting. We apply the proposed method in the physical world as well.

Adv-Makeup: A New Imperceptible and Transferable Attack on Face Recognition

Abstract: Deep neural networks, particularly face recognition models, have been shown to be vulnerable to both digital and physical adversarial examples. However, existing adversarial examples against face recognition systems either lack transferability to black-box models, or fail to be implemented in practice. In this paper, we propose a unified adversarial face generation method - Adv-Makeup, which can realize imperceptible and transferable attack under black-box setting. Adv-Makeup develops a task-driven makeup generation method with the blending module to synthesize imperceptible eye shadow over the orbital region on faces. And to achieve transferability, Adv-Makeup implements a fine-grained meta-learning adversarial attack strategy to learn more general attack features from various models. Compared to existing techniques, sufficient visualization results demonstrate that Adv-Makeup is capable to generate much more imperceptible attacks under both digital and physical scenarios. Meanwhile, extensive quantitative experiments show that Adv-Makeup can significantly improve the attack success rate under black-box setting, even attacking commercial systems.

Adversarial Sticker: A Stealthy Attack Method in the Physical World

Abstract: To assess the vulnerability of deep learning in the physical world, recent works introduce adversarial patches and apply them on different tasks. In this paper, we propose another kind of adversarial patch: the Meaningful Adversarial Sticker, a physically feasible and stealthy attack method by using real stickers existing in our life. Unlike the previous adversarial patches by designing perturbations, our method manipulates the sticker's pasting position and rotation angle on the objects to perform physical attacks. Because the position and rotation angle are less affected by the printing loss and color distortion, adversarial stickers can keep good attacking performance in the physical world. Besides, to make adversarial stickers more practical in real scenes, we conduct attacks in the black-box setting with the limited information rather than the white-box setting with all the details of threat models. To effectively solve for the sticker's parameters, we design the Region based Heuristic Differential Evolution Algorithm, which utilizes the new-found regional aggregation of effective solutions and the adaptive adjustment strategy of the evaluation criteria. Our method is comprehensively verified in the face recognition and then extended to the image retrieval and traffic sign recognition. Extensive experiments show the proposed method is effective and efficient in complex physical conditions and has a good generalization for different tasks.

Representative physical attacks against face recognition systems

Physical attacks are carried out in the real world and become more threatening than digital attacks. Since Kurakin et al. {{cite:58d60d7}} first demonstrate DNNs are vulnerable to physically adversarial examples, a lot of attention is poured into the physical domain. In this section, we introduce different attack tasks in physical scenarios, discuss physical forms of adversarial examples, and analyze attack methods in the white-box setting and the black-box setting (Representative methods in the different attack tasks are briefly exhibited in Table {{table:82e1fb0f-3067-46d3-b45c-7c99a35b0431}} , {{table:64017f6b-68fe-43ce-a9c3-847591b67adf}} , {{table:85d324e9-ec68-4db4-87e4-c105d180ee51}} , respectively).


Main applications & technical challenges

System/channel modeling & signal processing

Hardware implementations & channel modeling

Hardware architectures & functionalies/features

Discuss the hardware implementations, performance gains, and applications of LISA with an emphasis on its differences and relationship with the backscatter communication and reflective relay.

Present an early overview of the potential of IRS for wireless communications, and discuss its basic concept, main applications, and key technical challenges.

Provide a survey on the optimization methods and analytical frameworks for LISA, and briefly discuss future research directions.

Introduce the system/channel modeling and provide a tutorial overview of the RIS fundamentals from a signal processing standpoint, including communication, localization, and sensing.

Discuss the potential improvements by IRS in NOMA in terms of channel gain, power allocation fairness, covergae range, and energy efficiency.

Envision the concept and physical architecture of software-controlled/defined metasurface and discuss the research challenges.

Introduce different hardware implementations of RIS using metasurface and reflectarray, and discuss channel modeling as well as challenges and opportunities in RIS-aided wireless networks.

Introduce the holographic MIMO surface and summarize its hardware architectures, functionalities/features, and communication applications.

Present the relevant channel models and discuss the benefits of RIS for localization and mapping in terms of improved accuracy and extended coverage.

Review the key features of RIS and raise three myths about the functionality, performance gain, and path loss.

Overview two typical deployment strategies for IRS-aided communications and propose a new IRS deployment strategy.

Large Intelligent Surface/Antennas (LISA): Making Reflective Radios Smart

Abstract: Large intelligent surface/antennas (LISA), a two-dimensional artificial structure with a large number of reflective-surface/antenna elements, is a promising reflective radio technology to construct programmable wireless environments in a smart way. Specifically, each element of the LISA adjusts the reflection of the incident electromagnetic waves with unnatural properties, such as negative refraction, perfect absorption, and anomalous reflection, thus the wireless environments can be software-defined according to various design objectives. In this paper, we introduce the reflective radio basics, including backscattering principles, backscatter communication, and reflective relay, and the fundamentals and implementations of LISA technology. Then, we present an overview of the state-of-the-art research on emerging applications of LISA-aided wireless networks. Finally, the limitations, challenges, and open issues associated with LISA for future wireless applications are discussed.

Towards Smart and Reconfigurable Environment: Intelligent Reflecting Surface Aided Wireless Network

Abstract: IRS is a new and revolutionizing technology that is able to significantly improve the performance of wireless communication networks, by smartly reconfiguring the wireless propagation environment with the use of massive low-cost passive reflecting elements integrated on a planar surface. Specifically, different elements of an IRS can independently reflect the incident signal by controlling its amplitude and/or phase and thereby collaboratively achieve fine-grained 3D passive beamforming for directional signal enhancement or nulling. In this article, we first provide an overview of the IRS technology, including its main applications in wireless communication, competitive advantages over existing technologies, hardware architecture as well as the corresponding new signal model. We then address the key challenges in designing and implementing the new IRS-aided hybrid (with both active and passive components) wireless network, as compared to the traditional network comprising active components only. Finally, numerical results are provided to show the great performance enhancement with the use of IRS in typical wireless networks.

Intelligent Surfaces for 6G Wireless Networks: A Survey of Optimization and Performance Analysis Techniques

Abstract: This paper surveys the optimization frameworks and performance analysis methods for large intelligent surfaces (LIS), which have been emerging as strong candidates to support the sixth-generation wireless physical platforms (6G). Due to their ability to adjust the behavior of interacting electromagnetic (EM) waves through intelligent manipulations of the reflections phase shifts, LIS have shown promising merits at improving the spectral efficiency of wireless networks. In this context, researchers have been recently exploring LIS technology in depth as a means to achieve programmable, virtualized, and distributed wireless network infrastructures. From a system level perspective, LIS have also been proven to be a low-cost, green, sustainable, and energy-efficient solution for 6G systems. This paper provides a unique blend that surveys the principles of operation of LIS, together with their optimization and performance analysis frameworks. The paper first introduces the LIS technology and its physical working principle. Then, it presents various optimization frameworks that aim to optimize specific objectives, namely, maximizing energy efficiency, sum-rate, secrecy-rate, and coverage. The paper afterwards discusses various relevant performance analysis works including capacity analysis, the impact of hardware impairments on capacity, uplink/downlink data rate analysis, and outage probability. The paper further presents the impact of adopting the LIS technology for positioning applications. Finally, we identify numerous exciting open challenges for LIS-aided 6G wireless networks, including resource allocation problems, hybrid radio frequency/visible light communication (RF-VLC) systems, health considerations, and localization.

Reconfigurable Intelligent Surfaces: A Signal Processing Perspective With Wireless Applications

Abstract: Antenna array technology enables directional transmission and reception of wireless signals, for communications, localization, and sensing purposes. The signal processing algorithms that underpin this technology began to be developed several decades ago [1], but it is first with the ongoing deployment of the fifth-generation (5G) wireless mobile networks that it becomes a mainstream technology [2]. The number of antenna elements in the arrays of the 5G base stations and user devices can be measured at the order of 100 and 10, respectively. As the networks shift towards using higher frequency bands, more antennas fit into a given aperture. The 5G developments enhance the transmitter and receiver functionalities, but the wireless channel propagation remains an uncontrollable system. This is illustrated in Fig. 1(a) and its mathematical notation will be introduced later. Transmitted signals with three different frequencies are shown to illustrate the fact that attenuation can vary greatly across frequencies. Looking beyond 5G, the advent of electromagnetic components that can shape how they interact with wireless signals enables partial control of the propagation. A reconfigurable intelligent surface (RIS) is a two-dimensional surface of engineered material whose properties are reconfigurable rather than static [4]. This article provides a tutorial on the fundamental properties of the RIS technology from a signal processing perspective. It is meant as a complement to recent surveys of electromagnetic and hardware aspects [4], [7], [11], acoustics [12], communication theory [13], and localization [8]. We will provide the formulas and derivations that are required to understand and analyze RIS-aided systems using signal processing, and exemplify how they can be utilized for improved communication, localization, and sensing.

What Role Do Intelligent Reflecting Surfaces Play in Multi-Antenna Non-Orthogonal Multiple Access?

Abstract: Massive multiple-input multiple-output (MIMO) and non-orthogonal multiple access (NOMA) are two key techniques for enabling massive connectivity in future wireless networks. A massive MIMO-NOMA system can deliver remarkable spectral improvements and low communication latency. Nevertheless, the uncontrollable stochastic behavior of the wireless channels can still degrade its performance. In this context, intelligent reflecting surface (IRS) has arisen as a promising technology for smartly overcoming the harmful effects of the wireless environment. The disruptive IRS concept of controlling the propagation channels via software can provide attractive performance gains to the communication networks, including higher data rates, improved user fairness, and, possibly, higher energy efficiency. In this article, in contrast to the existing literature, we demonstrate the main roles of IRSs in MIMO-NOMA systems. Specifically, we identify and perform a comprehensive discussion of the main performance gains that can be achieved in IRS-assisted massive MIMO-NOMA (IRS-NOMA) networks. We outline exciting futuristic use case scenarios for IRS-NOMA and expose the main related challenges and future research directions. Furthermore, throughout the article, we support our in-depth discussions with representative numerical results.

A New Wireless Communication Paradigm through Software-Controlled Metasurfaces

Abstract: Electromagnetic waves undergo multiple uncontrollable alterations as they propagate within a wireless environment. Free space path loss, signal absorption, as well as reflections, refractions, and diffractions caused by physical objects within the environment highly affect the performance of wireless communications. Currently, such effects are intractable to account for and are treated as probabilistic factors. This article proposes a radically different approach, enabling deterministic, programmable control over the behavior of wireless environments. The key enabler is the so-called HyperSurface tile, a novel class of planar meta-materials that can interact with impinging electromagnetic waves in a controlled manner. The HyperSurface tiles can effectively re-engineer electromagnetic waves, including steering toward any desired direction, full absorption, polarization manipulation, and more. Multiple tiles are employed to coat objects such as walls, furniture, and overall, any objects in indoor and outdoor environments. An external software service calculates and deploys the optimal interaction types per tile to best fit the needs of communicating devices. Evaluation via simulations highlights the potential of the new concept.

Reconfigurable Intelligent Surfaces for Wireless Communications: Principles, Challenges, and Opportunities

Abstract: Recently there has been a flurry of research on the use of reconfigurable intelligent surfaces (RIS) in wireless networks to create smart radio environments. In a smart radio environment, surfaces are capable of manipulating the propagation of incident electromagnetic waves in a programmable manner to actively alter the channel realization, which turns the wireless channel into a controllable system block that can be optimized to improve overall system performance. In this article, we provide a tutorial overview of reconfigurable intelligent surfaces (RIS) for wireless communications. We describe the working principles of reconfigurable intelligent surfaces (RIS) and elaborate on different candidate implementations using metasurfaces and reflectarrays. We discuss the channel models suitable for both implementations and examine the feasibility of obtaining accurate channel estimates. Furthermore, we discuss the aspects that differentiate RIS optimization from precoding for traditional MIMO arrays highlighting both the arising challenges and the potential opportunities associated with this emerging technology. Finally, we present numerical results to illustrate the power of an RIS in shaping the key properties of a MIMO channel.

Holographic MIMO Surfaces for 6G Wireless Networks: Opportunities, Challenges, and Trends

Abstract: Future wireless networks are expected to evolve toward an intelligent and software reconfigurable paradigm enabling ubiquitous communications between humans and mobile devices. They will also be capable of sensing, controlling, and optimizing the wireless environment to fulfill the visions of low-power, high-throughput, massively- connected, and low-latency communications. A key conceptual enabler that is recently gaining increasing popularity is the HMIMOS that refers to a low-cost transformative wireless planar structure comprised of sub-wavelength metallic or dielectric scattering particles, which is capable of shaping electromagnetic waves according to desired objectives. In this article, we provide an overview of HMIMOS communications including the available hardware architectures for reconfiguring such surfaces, and highlight the opportunities and key challenges in designing HMIMOS-enabled wireless communications.

Radio Localization and Mapping with Reconfigurable Intelligent Surfaces

Abstract: 5G radio at millimeter wave (mmWave) and beyond 5G concepts at 0.1-1 THz can exploit angle and delay measurements for localization, by the virtue of increased bandwidth and large antenna arrays but are limited in terms of blockage caused by obstacles. Reconfigurable intelligent surfaces (RISs) are seen as a transformative technology that can control the physical propagation environment in which they are embedded by passively reflecting EM waves in preferred directions. Whereas such RISs have been mainly intended for communication purposes, RISs can have great benefits in terms of performance, energy consumption, and cost for localization and mapping. These benefits as well as associated challenges are the main topics of this paper.

Reconfigurable Intelligent Surfaces: Three Myths and Two Critical Questions

Abstract: The search for physical-layer technologies that can play a key role in beyond-5G systems has started. One option is reconfigurable intelligent surfaces (RIS), which can collect wireless signals from a transmitter and passively beamform them towards the receiver. The technology has exciting prospects and is quickly gaining traction in the communication community, but in the current hype we have witnessed how several myths and overstatements are spreading in the literature. In this article, we take a neutral look at the RIS technology. We first review the fundamentals and then explain specific features that can be easily misinterpreted. In particular, we debunk three myths: 1) Current network technology can only control the transmitter and receiver, not the environment in between; 2) A better asymptotic array gain is achieved than with conventional beamforming; 3) The pathloss is the same as with anomalous mirrors. To inspire further research, we conclude by identifying two critical questions that must be answered for RIS to become a successful technology: 1) What is a convincing use case for RIS?; 2) How can we estimate channels and control an RIS in real time?

How to Deploy Intelligent Reflecting Surfaces in Wireless Network: BS-side, User-side, or Both Sides?

Abstract: The performance of wireless communication systems is fundamentally constrained by the random and uncontrollable wireless channel. By leveraging the recent advance in digitally-controlled metasurface, intelligent reflecting surface (IRS) has emerged as a promising solution to enhance the wireless network performance by smartly reconfiguring the radio propagation environment. Despite the substantial research on IRS-aided communications, this article addresses the important issue of how to deploy IRSs in a wireless network to achieve its optimum performance. We first compare the two conventional strategies of deploying IRS at the side of base station or distributed users in terms of various communication performance metrics, and then propose a new hybrid IRS deployment strategy by combining their complementary advantages. Moreover, the main challenges in optimizing IRS deployment as well as their promising solutions are discussed. A case study is also presented to compare the performance of different IRS deployment strategies and draw useful insights for practical design.

List of Representative Overview/Survey/Tutorial Papers on IRS/RIS

The above benefits of IRS have spurred extensive studies on investigating its design and performance in a variety of wireless systems, e.g., orthogonal frequency division multiplexing (OFDM) {{cite:8c2f470}}, {{cite:82e2c49}}, {{cite:0980e1e}}, {{cite:0982350}}, multi-antenna communication {{cite:ab7a5f3}}, {{cite:a491c1d}}, {{cite:e49c5d6}}, non-orthogonal multiple access (NOMA) {{cite:d0ecb2b}}, {{cite:02fbb2a}}, {{cite:3c9ff04}}.
Moreover, there are more than a dozen of overview/survey/tutorial papers that have disseminated the state-of-the-art results on IRS and its various equivalents for wireless communications with different focuses, including technical challenges {{cite:32cc94e}}, {{cite:e22f246}}, system models/designs {{cite:32cc94e}}, {{cite:e22f246}}, {{cite:e9b5391}}, {{cite:c29890e}}, {{cite:b81cb59}}, {{cite:fd2e0ce}}, applications {{cite:e22f246}}, {{cite:6bdfba6}}, {{cite:3a77e9c}}, {{cite:c1cd97c}}, hardware implementations {{cite:90fd672}}, {{cite:fa5f280}}, {{cite:9b7e053}}, {{cite:4c9c24d}}, functionalities {{cite:c29890e}}, {{cite:4c9c24d}}, {{cite:24f0663}}, and performance analysis {{cite:3a77e9c}}, {{cite:ee8e25b}}, which are summarized in Table {{table:d8a99bff-27d0-466e-ad45-266f4a8e165d}}  for the ease of reference.
It is noted that the existing works have mainly considered the ideal assumption of perfect channel state information (CSI), which, however, is difficult to achieve in practice, especially for IRSs without signal processing/transmission capabilities. While for some other works (e.g., {{cite:90fd672}}, {{cite:fa5f280}}, {{cite:9b7e053}}, {{cite:4c9c24d}}) that address IRS hardware implementations, they mainly focus on the physical modeling, hardware architecture, and characteristics of the metasurface, but without an in-depth discussion of the effects of practical IRS hardware on the system design and communication performance.
In view of the above, a dedicated work that comprehensively surveys the practical design and implementation issues of IRS from a communication viewpoint is still missing, which thus motivates this paper.

In contrast to the existing overview/survey/tutorial papers on IRS listed in Table {{table:d8a99bff-27d0-466e-ad45-266f4a8e165d}} , this paper aims to provide a comprehensive and up-to-date survey on the research in IRS-aided wireless communications, with an emphasis to address the practical challenges in IRS channel estimation and passive beamforming/reflection optimization for different channel models and system setups.
In particular, this paper overviews a wide class of signal processing and communication techniques for resolving the practical issues in IRS channel estimation and passive beamforming/reflection optimization, and studies the effects of various IRS hardware constraints/imperfections on the system design and achievable performance. Moreover, practical design challenges for emerging IRS architectures and other IRS applications in wireless networks are discussed as well to inspire future research.


Spatial Attentive Single-Image Deraining With a High Quality Real Rain Dataset

Abstract: Removing rain streaks from a single image has been drawing considerable attention as rain streaks can severely degrade the image quality and affect the performance of existing outdoor vision tasks. While recent CNN-based derainers have reported promising performances, deraining remains an open problem for two reasons. First, existing synthesized rain datasets have only limited realism, in terms of modeling real rain characteristics such as rain shape, direction and intensity. Second, there are no public benchmarks for quantitative comparisons on real rain images, which makes the current evaluation less objective. The core challenge is that real world rain/clean image pairs cannot be captured at the same time. In this paper, we address the single image rain removal problem in two ways. First, we propose a semi-automatic method that incorporates temporal priors and human supervision to generate a high-quality clean image from each input sequence of real rain images. Using this method, we construct a large-scale dataset of ∼29.5K rain/rain-free image pairs that covers a wide range of natural rain scenes. Second, to better cover the stochastic distribution of real rain streaks, we propose a novel SPatial Attentive Network (SPANet) to remove rain streaks in a local-to-global manner. Extensive experiments demonstrate that our network performs favorably against the state-of-the-art deraining methods.

Single Image Deraining: A Comprehensive Benchmark Analysis

Abstract: We present a comprehensive study and evaluation of existing single image deraining algorithms, using a new large-scale benchmark consisting of both synthetic and real-world rainy images.This dataset highlights diverse data sources and image contents, and is divided into three subsets (rain streak, rain drop, rain and mist), each serving different training or evaluation purposes. We further provide a rich variety of criteria for dehazing algorithm evaluation, ranging from full-reference metrics, to no-reference metrics, to subjective evaluation and the novel task-driven evaluation. Experiments on the dataset shed light on the comparisons and limitations of state-of-the-art deraining algorithms, and suggest promising future directions.

Benchamark datasets with different rain types.

Benchmark Datasets. Eight datasets are adopted as listed in Table {{table:b034b717-407d-43d7-95ca-d9da6719181e}} , including five synthesized ones and three real ones. Similar to other supervised methods, during the training process, what we explicitly need are the paired clean image {{formula:19e3a736-4e54-449b-86fa-5524933542c9}}  and the rain-affected image {{formula:929ba066-b36c-4f8f-b5d7-0be99213f976}} .Detailed explanations are included in supplementary material.


Towards Causal VQA: Revealing and Reducing Spurious Correlations by Invariant and Covariant Semantic Editing

Abstract: Despite significant success in Visual Question Answering (VQA), VQA models have been shown to be notoriously brittle to linguistic variations in the questions. Due to deficiencies in models and datasets, today’s models often rely on correlations rather than predictions that are causal w.r.t. data. In this paper, we propose a novel way to analyze and measure the robustness of the state of the art models w.r.t semantic visual variations as well as propose ways to make models more robust against spurious correlations. Our method performs automated semantic image manipulations and tests for consistency in model predictions to quantify the model robustness as well as generate synthetic data to counter these problems. We perform our analysis on three diverse, state of the art VQA models and diverse question types with a particular focus on challenging counting questions. In addition, we show that models can be made significantly more robust against inconsistent predictions using our edited data. Finally, we show that results also translate to real-world error cases of state of the art models, which results in improved overall performance

Adversarial VQA: A New Benchmark for Evaluating the Robustness of VQA Models

Abstract: Benefiting from large-scale pre-training, we have witnessed significant performance boost on the popular Visual Question Answering (VQA) task. Despite rapid progress, it remains unclear whether these state-of-the-art (SOTA) models are robust when encountering examples in the wild. To study this, we introduce Adversarial VQA, a new large-scale VQA benchmark, collected iteratively via an adversarial human-and-model-in-the-loop procedure. Through this new benchmark, we discover several interesting findings. (i) Surprisingly, we find that during dataset collection, non-expert annotators can easily attack SOTA VQA models successfully. (ii) Both large-scale pre-trained models and adversarial training methods achieve far worse performance on the new benchmark than over standard VQA v2 dataset, revealing the fragility of these models while demonstrating the effectiveness of our adversarial dataset. (iii) When used for data augmentation, our dataset can effectively boost model performance on other robust VQA benchmarks. We hope our Adversarial VQA dataset can shed new light on robustness study in the community and serve as a valuable benchmark for future work.

Current causal image question answering datasets.

Image question answering benchmarks evaluate models' capability of answering natural language questions based on a corresponded image. Recent image question answering benchmarks try to collect or generate balanced QA pairs to make the dataset distribution more balanced in question distribution. VQA v2.0 {{cite:be36575}} collects complementary QA pairs by replacing the image and the answer in a QA pairs. VQA-CP {{cite:0281c39}} resplited the VQA v1 dataset and VQA v2 dataset to construct two new dataset VQA-CP v1 and VQA-CP v2. As Fig. {{figure:9c55414e-5c48-4317-929d-55afd758aab9}}  shows, {{cite:0ea9591}} constructed IV-VQA and CV-VQA datasets using semantic editing to generate image and reexamine the image by human. {{cite:d56f17b}} proposed a human-machine adversarial to collect robust QA pairs. Fig. {{figure:4923dacb-1171-4233-bd38-d567ec15cc39}}  illustrates the adversarial data collecting procedure. In Table {{table:7dbd8547-508e-46a6-a359-05627528ef6b}} , we summarize these datasets in terms of image source, split numbers, collected or not, and rebalanced or not.


AI Challenger : A Large-scale Dataset for Going Deeper in Image Understanding

Abstract: Significant progress has been achieved in Computer Vision by leveraging large-scale image datasets. However, large-scale datasets for complex Computer Vision tasks beyond classification are still limited. This paper proposed a large-scale dataset named AIC (AI Challenger) with three sub-datasets, human keypoint detection (HKD), large-scale attribute dataset (LAD) and image Chinese captioning (ICC). In this dataset, we annotate class labels (LAD), keypoint coordinate (HKD), bounding box (HKD and LAD), attribute (LAD) and caption (ICC). These rich annotations bridge the semantic gap between low-level images and high-level concepts. The proposed dataset is an effective benchmark to evaluate and improve different computational methods. In addition, for related tasks, others can also use our dataset as a new resource to pre-train their models.

CrowdPose: Efficient Crowded Scenes Pose Estimation and a New Benchmark

Abstract: Multi-person pose estimation is fundamental to many computer vision tasks and has made significant progress in recent years. However, few previous methods explored the problem of pose estimation in crowded scenes while it remains challenging and inevitable in many scenarios. Moreover, current benchmarks cannot provide an appropriate evaluation for such cases. In this paper, we propose a novel and efficient method to tackle the problem of pose estimation in the crowd and a new dataset to better evaluate algorithms. Our model consists of two key components: joint-candidate single person pose estimation (SPPE) and global maximum joints association. With multi-peak prediction for each joint and global association using the graph model, our method is robust to inevitable interference in crowded scenes and very efficient in inference. The proposed method surpasses the state-of-the-art methods on CrowdPose dataset by 5.2 mAP and results on MSCOCO dataset demonstrate the generalization ability of our method.

PoseTrack: A Benchmark for Human Pose Estimation and Tracking

Abstract: Existing systems for video-based pose estimation and tracking struggle to perform well on realistic videos with multiple people and often fail to output body-pose trajectories consistent over time. To address this shortcoming this paper introduces PoseTrack which is a new large-scale benchmark for video-based human pose estimation and articulated tracking. Our new benchmark encompasses three tasks focusing on i) single-frame multi-person pose estimation, ii) multi-person pose estimation in videos, and iii) multi-person articulated tracking. To establish the benchmark, we collect, annotate and release a new dataset that features videos with multiple people labeled with person tracks and articulated pose. A public centralized evaluation server is provided to allow the research community to evaluate on a held-out test set. Furthermore, we conduct an extensive experimental study on recent approaches to articulated pose tracking and provide analysis of the strengths and weaknesses of the state of the art. We envision that the proposed benchmark will stimulate productive research both by providing a large and representative training dataset as well as providing a platform to objectively evaluate and compare the proposed methods. The benchmark is freely accessible at https://posetrack.net/.

2D pose estimation datasets. Kpts is short for keypoints. For the train/val/test set, the number of images in image-based datasets and the number of videos in video-based datasets are listed.

we introduce 2D datasets according to the categories of 1) image-level or video-level, and 2) single person or multiple persons. The dataset summary is given in Table {{table:451d65ee-4621-487c-b74a-31be2883b053}} . Part of the example images is shown in Fig. {{figure:70b25eff-d1f8-4d5f-9df9-5bc18ae30af4}} .