Families in the Wild (FIW): Large-Scale Kinship Image Database and Benchmarks

Abstract: We present the largest kinship recognition dataset to date, Families in the Wild (FIW). Motivated by the lack of a single, unified dataset for kinship recognition, we aim to provide a dataset that captivates the interest of the research community. With only a small team, we were able to collect, organize, and label over 10,000 family photos of 1,000 families with our annotation tool designed to mark complex hierarchical relationships and local label information in a quick and efficient manner. We include several benchmarks for two image-based tasks, kinship verification and family recognition. For this, we incorporate several visual features and metric learning methods as baselines. Also, we demonstrate that a pre-trained Convolutional Neural Network (CNN) as an off-the-shelf feature extractor outperforms the other feature types. Then, results were further boosted by fine-tuning two deep CNNs on FIW data: (1) for kinship verification, a triplet loss function was learned on top of the network of pre-train weights; (2) for family recognition, a family-specific softmax classifier was added to the network.

Tri-Subject Kinship Verification: Understanding the Core of A Family

Abstract: One major challenge in computer vision is to go beyond the modeling of individual objects and to investigate the bi- (one-versus-one) or tri- (one-versus-two) relationship among multiple visual entities, answering such questions as whether a child in a photo belongs to the given parents. The child-parents relationship plays a core role in a family, and understanding such kin relationship would have a fundamental impact on the behavior of an artificial intelligent agent working in the human world. In this work, we tackle the problem of one-versus-two (tri-subject) kinship verification and our contributions are threefold: 1) a novel relative symmetric bilinear model (RSBM) is introduced to model the similarity between the child and the parents, by incorporating the prior knowledge that a child may resemble one particular parent more than the other; 2) a spatially voted method for feature selection, which jointly selects the most discriminative features for the child-parents pair, while taking local spatial information into account; and 3) a large-scale tri-subject kinship database characterized by over 1,000 child-parents families. Extensive experiments on KinFaceW, Family101, and our newly released kinship database show that the proposed method outperforms several previous state of the art methods, while could also be used to significantly boost the performance of one-versus-one kinship verification when the information about both parents are available.

Datasets with kinship labels

The availability of labelled datasets is the cornerstone of the development of facial models for aging and kinship. In this section, we review recently released datasets which contain annotations for age progression, age estimation, age-invariant facial characterization, and kinship verification. Particular emphasis is put on data capturing naturalistic, real-world conditions, often referred to as in the wild {{cite:04f6363}}. A complete catalogue of the available datasets for facial aging modeling is listed in Table {{table:2e0a7ac2-0aa3-4392-8668-e28f4c356880}}  while Table {{table:f1f42705-3c59-4bb8-835f-26e77ffe06a4}}  contains the available facial data for kinship verification.

Kinship verification from faces is a relatively new task,
aiming to verify whether the individuals in a pair of facial visual inputs are blood related. There are eleven different kinds of blood relations: (i) Father- Daughter, (ii) Mother- Daughter, (iii) Father- Son, (iv) Mother- Son, (v) Sister- Sister, (vi) Sister- Brother, (vii) Brother- Brother. (viii) Grandfather- Grandson, (ix) Grandmother- Grandson, (x) Grandfather- Granddaughter and (xi) Grandmother- Granddaughter.
A catalogue of dataset with kinship annotations is tabulated in Table {{table:f1f42705-3c59-4bb8-835f-26e77ffe06a4}} . The vast majority of the available data is annotated in terms of only 4 kin relations.

Similar to age related tasks, the problem of kinship modeling has been attacked using mainly in the wild datasets. The nature of the labels makes the task of gathering and annotating such datasets very laborious. As a result most of the datasets in Table {{table:f1f42705-3c59-4bb8-835f-26e77ffe06a4}}  contain a small number of images, which is not ideal for modeling large variations. What is more, these images, more often than not, originate from the same photo. Cropping the data from the same image can significantly bias the task of classification, by adding factors like the environment, lighting, chrominance and the image quality {{cite:f6caec5}}. Therefore, datasets that contain such images are considered biased and can not be used for the task of kinship verification. This is evident when comparing the state-of-the-art performance between classic benchmarks like KinFaceW-II (i.e., classification accuracy 96.2%) and more recent ones like FIW (i.e., classification accuracy 71%).


Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection

Abstract: In this paper, we present an open-set object detector, called Grounding DINO, by marrying Transformer-based detector DINO with grounded pre-training, which can detect arbitrary objects with human inputs such as category names or referring expressions. The key solution of open-set object detection is introducing language to a closed-set detector for open-set concept generalization. To effectively fuse language and vision modalities, we conceptually divide a closed-set detector into three phases and propose a tight fusion solution, which includes a feature enhancer, a language-guided query selection, and a cross-modality decoder for cross-modality fusion. While previous works mainly evaluate open-set object detection on novel categories, we propose to also perform evaluations on referring expression comprehension for objects specified with attributes. Grounding DINO performs remarkably well on all three settings, including benchmarks on COCO, LVIS, ODinW, and RefCOCO/+/g. Grounding DINO achieves a $52.5$ AP on the COCO detection zero-shot transfer benchmark, i.e., without any training data from COCO. It sets a new record on the ODinW zero-shot benchmark with a mean $26.1$ AP. Code will be available at \url{https://github.com/IDEA-Research/GroundingDINO}.

Grounded Language-Image Pre-training

Abstract: This paper presents a grounded language-image pretraining (GLIP) model for learning object-level, language-aware, and semantic-rich visual representations. GLIP unifies object detection and phrase grounding for pre-training. The unification brings two benefits: 1) it allows GLIP to learn from both detection and grounding data to improve both tasks and bootstrap a good grounding model; 2) GLIP can leverage massive image-text pairs by generating grounding boxes in a self-training fashion, making the learned representations semantic-rich. In our experiments, we pre-train GLIP on 27M grounding data, including 3M human-annotated and 24M web-crawled image-text pairs. The learned representations demonstrate strong zero-shot and few-shot transferability to various object-level recognition tasks. 1) When directly evaluated on COCO and LVIS (without seeing any images in COCO during pre-training), GLIP achieves 49.8 AP and 26.9 AP, respectively, surpassing many supervised baselines.11Supervised baselines on COCO object detection: Faster-RCNN w/ ResNet50 (40.2) or ResNet101 (42.0), and DyHead w/ Swin-Tiny (49.7). 2) After fine-tuned on COCO, GLIP achieves 60.8 AP on val and 61.5 AP on test-dev, surpassing prior SoTA. 3) When transferred to 13 downstream object detection tasks, a 1-shot GLIP rivals with a fully-supervised Dynamic Head. Code will be released at https://github.com/microsoft/GLIP.

Grounded Language-Image Pre-training

Grounded Language-Image Pre-training

Grounded Language-Image Pre-training

Grounded Language-Image Pre-training

Results with different success distances for ObjectNav.

Since zero-shot tasks are difficult, we choose 3m as the success distance. We also report success rates from 1m to 3m as in Table {{table:65836526-67b9-418e-bb80-ce19a371ff59}}  for object navigation given a category (ObjectNav). From the results, we can see that the agent performs worse with smaller success distances.


Bayesian active learning by disagreement

Greed Is Good: Exploration and Exploitation Trade-offs in Bayesian Optimisation

Abstract: The performance of acquisition functions for Bayesian optimisation to locate the global optimum of continuous functions is investigated in terms of the Pareto front between exploration and exploitation. We show that Expected Improvement (EI) and the Upper Confidence Bound (UCB) always select solutions to be expensively evaluated on the Pareto front, but Probability of Improvement is not guaranteed to do so and Weighted Expected Improvement does so only for a restricted range of weights. We introduce two novel -greedy acquisition functions. Extensive empirical evaluation of these together with random search, purely exploratory, and purely exploitative search on 10 benchmark problems in 1 to 10 dimensions shows that -greedy algorithms are generally at least as effective as conventional acquisition functions (e.g., EI and UCB), particularly with a limited budget. In higher dimensions, -greedy approaches are shown to have improved performance over conventional approaches. These results are borne out on a real-world computational fluid dynamics optimisation problem and a robotics active learning problem. Our analysis and experiments suggest that the most effective strategy, particularly in higher dimensions, is to be mostly greedy, occasionally selecting a random exploratory solution.

Bayesian Active Learning for Classification and Preference Learning

Abstract: Information theoretic active learning has been widely studied for probabilistic models. For simple regression an optimal myopic policy is easily tractable. However, for other tasks and with more complex models, such as classication with nonparametric models, the optimal solution is harder to compute. Current approaches make approximations to achieve tractability. We propose an approach that expresses information gain in terms of predictive entropies, and apply this method to the Gaussian Process Classier (GPC). Our approach makes minimal approximations to the full information theoretic objective. Our experimental performance compares favourably to many popular active learning algorithms, and has equal or lower computational complexity. We compare well to decision theoretic approaches also, which are privy to more information and require much more computational time. Secondly, by developing further a reformulation of binary preference learning to a classication problem, we extend our algorithm to Gaussian Process preference learning.

A Tutorial on Thompson Sampling

Abstract: Thompson sampling is an algorithm for online decision problems where actions are taken sequentially in a manner that must balance between exploiting what is known to maximize immediate performance and investing to accumulate new information that may improve future performance. The algorithm addresses a broad range of problems in a computationally efficient manner and is therefore enjoying wide use. This tutorial covers the algorithm and its application, illustrating concepts through a range of examples, including Bernoulli bandit problems, shortest path problems, product recommendation, assortment, active learning with neural networks, and reinforcement learning in Markov decision processes. Most of these problems involve complex information structures, where information revealed by taking an action informs beliefs about other actions. We will also discuss when and why Thompson sampling is or is not effective and relations to alternative algorithms.

List of informativeness measures implemented in PyRelationAL currently. The task column indicates whether the corresponding informativeness measure is defined for classification, regression, or is agnostic (Any) to the ML task being performed.

A critical question underpinning AL is how to measure the informativeness of each state in {{formula:f39a1160-b7e9-4c71-a422-6efa71fe8471}} . A number of heuristics have been proposed in the past which can broadly be split into two categories: 1.) uncertainty-based category and 2.) diversity-based category. These are not mutually exclusive as the two notions can be combined to positive effect {{cite:5cc23dd}}, {{cite:bb08c61}}, {{cite:40442f3}}. The former relies on some measure of uncertainties of the predictions obtained from the set of functions {{formula:9e1f5a45-1991-4c19-8e29-3e2b49984538}}  at each iteration. In practice, {{formula:73b7b0df-abeb-4381-8629-d54f200db87a}}  can be defined intrinsically by the class of ML models (e.g. Gaussian processes or Bayesian neural networks) or a subset of it can be obtained using heuristics such as ensembling {{cite:7470a2b}}, {{cite:97d24f5}}, {{cite:f6150f9}} or Bayesian inference approximation methods like MCDropout {{cite:6049de8}}. Various informativeness scores can be computed from the uncertainty estimates (see Table {{table:4baa8179-b355-4011-9751-1dc58719eb2e}} ) and used to select the query set {{formula:760485a8-6db6-494a-ab42-1e197e070c17}} . In contrast, diversity-based heuristics rely on a distance measure {{formula:162a8076-dc99-4c0c-8a45-1e044562a8a9}} , which can depend on {{formula:555fa2ea-0e35-4a91-8365-268dec9e855c}} , to score states in {{formula:a7be97e5-beb5-47fd-8f5e-da9d3ce7090a}}  with respect to elements in {{formula:641588df-fb6f-497c-9ca2-4840b61d93e1}} . The choice of informativeness measure is an important challenge of AL as well as the choice of query selection strategy from the informativeness scores.

The informativeness measure is often task-dependent; for instance, uncertainty measures are computed differently for continuous and categorical targets. Table {{table:4baa8179-b355-4011-9751-1dc58719eb2e}}  gives the list of informativeness measures from the literature that are implemented in PyRelationAL and Appendix  provides more details and discussion around the various measures. Note that all individual informativeness measure can be combined together. For instance, the Upper Confidence Bound (UCB) score for regression as defined in {{cite:ba00d07}} is a linear combination of the greedy score and least confidence score for regression tasks, with trade-off controlled by a hyperparameter. The final step is to select {{formula:b03ea898-d0f8-4a9a-98d4-7d9a1ed52f34}}  from {{formula:efe23da4-b3ee-4122-9ac5-00f1c3ee3fb9}}  based on a chosen criteria combined with the informativeness scores.

In this section, we provide more details about each type of informativeness measure discussed in the main and described some of the specific scores mentioned in Table {{table:4baa8179-b355-4011-9751-1dc58719eb2e}} .

Comparative analysis on selected strategies and datasets in PyRelationAL

In this section we perform a simple empirical comparative analysis of selected serial active learning strategies constructed with the informativeness measures in Table {{table:4baa8179-b355-4011-9751-1dc58719eb2e}}  across the current datasets in PyRelationAL. Serial active learning strategies perform one instance queries at each query loop iteration such that
{{formula:2fd41302-9ef6-4b09-b865-9a131847ca9c}}


Fairwashing: the risk of rationalization

Abstract: Black-box explanation is the problem of explaining how a machine learning model -- whose internal logic is hidden to the auditor and generally complex -- produces its outcomes. Current approaches for solving this problem include model explanation, outcome explanation as well as model inspection. While these techniques can be beneficial by providing interpretability, they can be used in a negative manner to perform fairwashing, which we define as promoting the false perception that a machine learning model respects some ethical values. In particular, we demonstrate that it is possible to systematically rationalize decisions taken by an unfair black-box model using the model explanation as well as the outcome explanation approaches with a given fairness metric. Our solution, LaundryML, is based on a regularized rule list enumeration algorithm whose objective is to search for fair rule lists approximating an unfair black-box model. We empirically evaluate our rationalization technique on black-box models trained on real-world datasets and show that one can obtain rule lists with high fidelity to the black-box model while being considerably less unfair at the same time.

Exacerbating Algorithmic Bias through Fairness Attacks

Abstract: Algorithmic fairness has attracted significant attention in recent years, with many quantitative measures suggested for characterizing the fairness of different machine learning algorithms. Despite this interest, the robustness of those fairness measures with respect to an intentional adversarial attack has not been properly addressed. Indeed, most adversarial machine learning has focused on the impact of malicious attacks on the accuracy of the system, without any regard to the system's fairness. We propose new types of data poisoning attacks where an adversary intentionally targets the fairness of a system. Specifically, we propose two families of attacks that target fairness measures. In the anchoring attack, we skew the decision boundary by placing poisoned points near specific target points to bias the outcome. In the influence attack on fairness, we aim to maximize the covariance between the sensitive attributes and the decision outcome and affect the fairness of the model. We conduct extensive experiments that indicate the effectiveness of our proposed attacks.

Fairness Through Robustness: Investigating Robustness Disparity in Deep Learning

Abstract: Deep neural networks (DNNs) are increasingly used in real-world applications (e.g. facial recognition). This has resulted in concerns about the fairness of decisions made by these models. Various notions and measures of fairness have been proposed to ensure that a decision-making system does not disproportionately harm (or benefit) particular subgroups of the population. In this paper, we argue that traditional notions of fairness that are only based on models' outputs are not sufficient when the model is vulnerable to adversarial attacks. We argue that in some cases, it may be easier for an attacker to target a particular subgroup, resulting in a form of robustness bias. We show that measuring robustness bias is a challenging task for DNNs and propose two methods to measure this form of bias. We then conduct an empirical study on state-of-the-art neural networks on commonly used real-world datasets such as CIFAR-10, CIFAR-100, Adience, and UTKFace and show that in almost all cases there are subgroups (in some cases based on sensitive attributes like race, gender, etc) which are less robust and are thus at a disadvantage. We argue that this kind of bias arises due to both the data distribution and the highly complex nature of the learned decision boundary in the case of DNNs, thus making mitigation of such biases a non-trivial task. Our results show that robustness bias is an important criterion to consider while auditing real-world systems that rely on DNNs for decision making. Code to reproduce all our results can be found here: https://github.com/nvedant07/Fairness-Through-Robustness

Characterizing the risk of fairwashing

Abstract: Fairwashing refers to the risk that an unfair black-box model can be explained by a fairer model through post-hoc explanation manipulation. In this paper, we investigate the capability of fairwashing attacks by analyzing their fidelity-unfairness trade-offs. In particular, we show that fairwashed explanation models can generalize beyond the suing group (i.e., data points that are being explained), meaning that a fairwashed explainer can be used to rationalize subsequent unfair decisions of a black-box model. We also demonstrate that fairwashing attacks can transfer across black-box models, meaning that other black-box models can perform fairwashing without explicitly using their predictions. This generalization and transferability of fairwashing attacks imply that their detection will be difficult in practice. Finally, we propose an approach to quantify the risk of fairwashing, which is based on the computation of the range of the unfairness of high-fidelity explainers.

Adversarial Inter-Group Link Injection Degrades the Fairness of Graph Neural Networks

Abstract: We present evidence for the existence and effectiveness of adversarial attacks on graph neural networks (GNNs) that aim to degrade fairness. These attacks can disadvantage a particular subgroup of nodes in GNN-based node classification, where nodes of the underlying network have sensitive attributes, such as race or gender. We conduct qualitative and experimental analyses explaining how adversarial link injection impairs the fairness of GNN predictions. For example, an attacker can compromise the fairness of GNN-based node classification by injecting adversarial links between nodes belonging to opposite subgroups and opposite class labels. Our experiments on empirical datasets demonstrate that adversarial fairness attacks can significantly degrade the fairness of GNN predictions (attacks are effective) with a low perturbation rate (attacks are efficient) and without a significant drop in accuracy (attacks are deceptive). This work demonstrates the vulnerability of GNN models to adversarial fairness attacks. We hope our findings raise awareness about this issue in our community and lay a foundation for the future development of GNN models that are more robust to such attacks.

Exploiting Fairness to Enhance Sensitive Attributes Reconstruction

Abstract: In recent years, a growing body of work has emerged on how to learn machine learning models under fairness constraints, often expressed with respect to some sensitive attributes. In this work, we consider the setting in which an adversary has black-box access to a target model and show that information about this model's fairness can be exploited by the adversary to enhance his reconstruction of the sensitive attributes of the training data. More precisely, we propose a generic reconstruction correction method, which takes as input an initial guess made by the adversary and corrects it to comply with some user-defined constraints (such as the fairness information) while minimizing the changes in the adversary's guess. The proposed method is agnostic to the type of target model, the fairness-aware learning method as well as the auxiliary knowledge of the adversary. To assess the applicability of our approach, we have conducted a thorough experimental evaluation on two state-of-the-art fair learning methods, using four different fairness metrics with a wide range of tolerances and with three datasets of diverse sizes and sensitive attributes. The experimental results demonstrate the effectiveness of the proposed approach to improve the reconstruction of the sensitive attributes of the training set.

Related work concerning adversarial attacks on fairness metrics of machine learning models. We abbreviate the following: data (D), model (M), image (I), tabular (T), graph (Gr), neural network (N), black-box (B), and group (G).  lists other abbreviations.

Closely related to adversarial attacks on explanations are attacks on machine learning fairness metrics, e.g. predictive equality {{cite:a578d8c}} and (statistical) demographic parity {{cite:6559c99}}. Intuitively, algorithms targeting model predictions and accuracy can be applied to manipulate other functions of the model output as well. Table {{table:de08150b-c250-4d7a-af54-53cf3c94cb68}}  lists a representative set of adversarial attacks on group fairness metrics, with the corresponding strategy of changing data {{cite:43b89ed}}, the model {{cite:1abcfcd}}, {{cite:0890b9a}}, or jointly changing data and the model {{cite:baffd22}}, {{cite:2bfa538}}, {{cite:a61a30a}}.


NExT-QA: Next Phase of Question-Answering to Explaining Temporal Actions

Abstract: We introduce NExT-QA, a rigorously designed video question answering (VideoQA) benchmark to advance video understanding from describing to explaining the temporal actions. Based on the dataset, we set up multi-choice and open-ended QA tasks targeting causal action reasoning, temporal action reasoning, and common scene comprehension. Through extensive analysis of baselines and established VideoQA techniques, we find that top-performing methods excel at shallow scene descriptions but are weak in causal and temporal action reasoning. Furthermore, the models that are effective on multi-choice QA, when adapted to open-ended QA, still struggle in generalizing the answers. This raises doubt on the ability of these models to reason and highlights possibilities for improvement. With detailed results for different question types and heuristic observations for future works, we hope NExT-QA will guide the next generation of VQA research to go beyond superficial description towards a deeper understanding of videos. (The dataset and related resources are available at https://github.com/doc-doc/NExT-QA.git).

TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answering

Abstract: Vision and language understanding has emerged as a subject undergoing intense study in Artificial Intelligence. Among many tasks in this line of research, visual question answering (VQA) has been one of the most successful ones, where the goal is to learn a model that understands visual content at region-level details and finds their associations with pairs of questions and answers in the natural language form. Despite the rapid progress in the past few years, most existing work in VQA have focused primarily on images. In this paper, we focus on extending VQA to the video domain and contribute to the literature in three important ways. First, we propose three new tasks designed specifically for video VQA, which require spatio-temporal reasoning from videos to answer questions correctly. Next, we introduce a new large-scale dataset for video VQA named TGIF-QA that extends existing VQA work with our new tasks. Finally, we propose a dual-LSTM based approach with both spatial and temporal attention, and show its effectiveness over conventional VQA techniques through empirical evaluations.

This research is supported by the Sea-NExT joint Lab. Major work was done when Junbin was a research intern at Sea AI Lab. We greatly thank Angela Yao as well as the anonymous reviewers for their thoughtful comments towards a better work.

Amrani, E., Ben-Ari, R., Rotman, D., Bronstein, A.: Noise estimation using
density estimation for self-supervised multimodal learning. In: AAAI
Conference on Artificial Intelligence (AAAI). vol. 35, pp. 6644–6652 (2021)
Anderson, P., He, X., Buehler, C., Teney, D., Johnson, M., Gould, S., Zhang,
L.: Bottom-up and top-down attention for image captioning and visual question
answering. In: Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition (CVPR). pp. 6077–6086 (2018)
Bain, M., Nagrani, A., Varol, G., Zisserman, A.: Frozen in time: A joint video
and image encoder for end-to-end retrieval. In: Proceedings of the IEEE/CVF
International Conference on Computer Vision (ICCV). pp. 1728–1738 (2021)
Battaglia, P.W., Hamrick, J.B., Bapst, V., Sanchez-Gonzalez, A., Zambaldi, V.,
Malinowski, M., Tacchetti, A., Raposo, D., Santoro, A., Faulkner, R., et al.:
Relational inductive biases, deep learning, and graph networks. arXiv
preprint arXiv:1806.01261 (2018)
Bertasius, G., Wang, H., Torresani, L.: Is space-time attention all you need
for video understanding? In: ICML. pp. 813–824. PMLR (2021)
Buch, S., Eyzaguirre, C., Gaidon, A., Wu, J., Fei-Fei, L., Niebles, J.C.:
Revisiting the" video" in video-language understanding. In: Proceedings of
the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
pp. 2917–2927 (2022)
Chen, X., Fang, H., Lin, T.Y., Vedantam, R., Gupta, S., Dollár, P.,
Zitnick, C.L.: Microsoft coco captions: Data collection and evaluation
server. arXiv preprint arXiv:1504.00325 (2015)
Chen, Y.C., Li, L., Yu, L., El Kholy, A., Ahmed, F., Gan, Z., Cheng, Y., Liu,
J.: Uniter: Universal image-text representation learning. In: European
Conference on Computer Vision (ECCV). pp. 104–120. Springer (2020)
Cherian, A., Hori, C., Marks, T.K., Le Roux, J.: (2.5+ 1) d spatio-temporal
scene graphs for video question answering. In: Proceedings of the AAAI
Conference on Artificial Intelligence. vol. 36, pp. 444–453 (2022)
Dang, L.H., Le, T.M., Le, V., Tran, T.: Hierarchical object-oriented
spatio-temporal reasoning for video question answering. In: IJCAI (August
2021)
Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: Pre-training of deep
bidirectional transformers for language understanding. In: NAACL (2019)
Ding, D., Hill, F., Santoro, A., Reynolds, M., Botvinick, M.: Attention over
learned object embeddings enables complex visual reasoning. Advances in
neural information processing systems (NeurIPS) 34 (2021)
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X.,
Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al.:
An image is worth 16x16 words: Transformers for image recognition at scale.
In: International Conference on Representation Learning (ICLR) (2020)
Fan, C., Zhang, X., Zhang, S., Wang, W., Zhang, C., Huang, H.: Heterogeneous
memory enhanced multimodal attention model for video question answering. In:
Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern
Recognition (CVPR). pp. 1999–2007 (2019)
Fu, T.J., Li, L., Gan, Z., Lin, K., Wang, W.Y., Wang, L., Liu, Z.: Violet:
End-to-end video-language transformers with masked visual-token modeling. In:
arXiv preprint arXiv:2111.12681 (November 2021)
Gao, J., Ge, R., Chen, K., Nevatia, R.: Motion-appearance co-memory networks
for video question answering. In: Proceedings of the IEEE/CVF Conference on
Computer Vision and Pattern Recognition (CVPR). pp. 6576–6585 (2018)
Geng, S., Gao, P., Chatterjee, M., Hori, C., Le Roux, J., Zhang, Y., Li, H.,
Cherian, A.: Dynamic graph representation learning for video dialog via
multi-modal shuffled transformers. In: AAAI Conference on Artificial
Intelligence (AAAI) (2021)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image
recognition. In: Proceedings of the IEEE/CVF Conference on Computer Vision
and Pattern Recognition (CVPR). pp. 770–778 (2016)
Huang, D., Chen, P., Zeng, R., Du, Q., Tan, M., Gan, C.: Location-aware graph
convolutional networks for video question answering. In: AAAI Conference on
Artificial Intelligence (AAAI). vol. 34, pp. 11021–11028 (2020)
Jang, Y., Song, Y., Yu, Y., Kim, Y., Kim, G.: Tgif-qa: Toward spatio-temporal
reasoning in visual question answering. In: Proceedings of the IEEE/CVF
Conference on Computer Vision and Pattern Recognition (CVPR). pp. 2758–2766
(2017)
Jia, C., Yang, Y., Xia, Y., Chen, Y.T., Parekh, Z., Pham, H., Le, Q., Sung,
Y.H., Li, Z., Duerig, T.: Scaling up visual and vision-language
representation learning with noisy text supervision. In: ICML. pp.
4904–4916. PMLR (2021)
Jiang, J., Chen, Z., Lin, H., Zhao, X., Gao, Y.: Divide and conquer:
Question-guided spatio-temporal contextual attention for video question
answering. In: AAAI Conference on Artificial Intelligence (AAAI). vol. 34,
pp. 11101–11108 (2020)
Jiang, P., Han, Y.: Reasoning with heterogeneous graph alignment for video
question answering. In: AAAI Conference on Artificial Intelligence (AAAI)
(2020)
Kamath, A., Singh, M., LeCun, Y., Synnaeve, G., Misra, I., Carion, N.:
Mdetr-modulated detection for end-to-end multi-modal understanding. In:
Proceedings of the IEEE/CVF International Conference on Computer Vision
(ICCV). pp. 1780–1790 (2021)
Kipf, T.N., Welling, M.: Semi-supervised classification with graph
convolutional networks. In: International Conference on Representation
Learning (ICLR) (2017)
Krishna, R., Chami, I., Bernstein, M., Fei-Fei, L.: Referring relationships.
In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern
Recognition (CVPR). pp. 6867–6876 (2018)
Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S.,
Kalantidis, Y., Li, L.J., Shamma, D.A., et al.: Visual genome: Connecting
language and vision using crowdsourced dense image annotations. IJCV
123(1), 32–73 (2017)
Le, T.M., Le, V., Venkatesh, S., Tran, T.: Hierarchical conditional relation
networks for video question answering. In: Proceedings of the IEEE/CVF
Conference on Computer Vision and Pattern Recognition (CVPR). pp. 9972–9981
(2020)
Lei, J., Li, L., Zhou, L., Gan, Z., Berg, T.L., Bansal, M., Liu, J.: Less is
more: Clipbert for video-and-language learning via sparse sampling. In:
Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern
Recognition (CVPR). pp. 7331–7341 (2021)
Lei, J., Yu, L., Bansal, M., Berg, T.L.: Tvqa: Localized, compositional video
question answering. In: Empirical Methods in Natural Language Processing
(EMNLP) (2018)
Li, J., Selvaraju, R., Gotmare, A., Joty, S., Xiong, C., Hoi, S.C.H.: Align
before fuse: Vision and language representation learning with momentum
distillation. In: Advances in neural information processing systems
(NeurIPS). vol. 34 (2021)
Li, L., Chen, Y.C., Cheng, Y., Gan, Z., Yu, L., Liu, J.: Hero: Hierarchical
encoder for video+ language omni-representation pre-training. In: Proceedings
of the 2020 Conference on Empirical Methods in Natural Language Processing
(EMNLP). pp. 2046–2065 (2020)
Li, X., Song, J., Gao, L., Liu, X., Huang, W., He, X., Gan, C.: Beyond rnns:
Positional self-attention with co-attention for video question answering. In:
AAAI Conference on Artificial Intelligence (AAAI). pp. 8658–8665 (2019)
Li, X., Yin, X., Li, C., Zhang, P., Hu, X., Zhang, L., Wang, L., Hu, H., Dong,
L., Wei, F., et al.: Oscar: Object-semantics aligned pre-training for
vision-language tasks. In: European Conference on Computer Vision (ECCV). pp.
121–137. Springer (2020)
Li, Y., Wang, X., Xiao, J., Ji, W., Chua, T.S.: Invariant grounding for video
question answering. In: Proceedings of the IEEE/CVF Conference on Computer
Vision and Pattern Recognition (CVPR). pp. 2928–2937 (2022)
Liu, F., Liu, J., Wang, W., Lu, H.: Hair: Hierarchical visual-semantic
relational reasoning for video question answering. In: Proceedings of the
IEEE/CVF International Conference on Computer Vision (ICCV). pp. 1698–1707
(October 2021)
Liu, Z., Ning, J., Cao, Y., Wei, Y., Zhang, Z., Lin, S., Hu, H.: Video swin
transformer. In: Proceedings of the IEEE/CVF Conference on Computer Vision
and Pattern Recognition (CVPR). pp. 3202–3211 (2022)
Lu, J., Batra, D., Parikh, D., Lee, S.: Vilbert: Pretraining task-agnostic
visiolinguistic representations for vision-and-language tasks. In: Advances
in neural information processing systems (NeurIPS). pp. 13–23 (2019)
Miech, A., Alayrac, J.B., Smaira, L., Laptev, I., Sivic, J., Zisserman, A.:
End-to-end learning of visual representations from uncurated instructional
videos. In: Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition (CVPR). pp. 9879–9889 (2020)
Miech, A., Zhukov, D., Alayrac, J.B., Tapaswi, M., Laptev, I., Sivic, J.:
Howto100m: Learning a text-video embedding by watching hundred million
narrated video clips. In: Proceedings of the IEEE/CVF International
Conference on Computer Vision (ICCV). pp. 2630–2640 (2019)
Park, J., Lee, J., Sohn, K.: Bridge to answer: Structure-aware graph
interaction network for video question answering. In: Proceedings of the
IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp.
15526–15535 (2021)
Peng, L., Yang, S., Bin, Y., Wang, G.: Progressive graph attention network for
video question answering. In: ACM MM. pp. 2871–2879 (2021)
Peng, M., Wang, C., Gao, Y., Shi, Y., Zhou, X.D.: Multilevel hierarchical
network with multiscale sampling for video question answering. IJCAI (2022)
Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry,
G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual
models from natural language supervision. In: ICML. pp. 8748–8763. PMLR
(2021)
Ren, S., He, K., Girshick, R., Sun, J.: Faster r-cnn: Towards real-time object
detection with region proposal networks. Advances in neural information
processing systems (NeurIPS) 28 (2015)
Sanh, V., Debut, L., Chaumond, J., Wolf, T.: Distilbert, a distilled version of
bert: smaller, faster, cheaper and lighter. Advances in neural information
processing systems (NeurIPS) (2019)
Seo, A., Kang, G.C., Park, J., Zhang, B.T.: Attend what you need:
Motion-appearance synergistic networks for video question answering. In: ACL.
pp. 6167–6177 (2021)
Seo, P.H., Nagrani, A., Schmid, C.: Look before you speak: Visually
contextualized utterances. In: Proceedings of the IEEE/CVF Conference on
Computer Vision and Pattern Recognition (CVPR). pp. 16877–16887 (2021)
Shang, X., Di, D., Xiao, J., Cao, Y., Yang, X., Chua, T.S.: Annotating objects
and relations in user-generated videos. In: Proceedings of the 2019 on
International Conference on Multimedia Retrieval (ICMR). pp. 279–287 (2019)
Shang, X., Xiao, J., Di, D., Chua, T.S.: Relation understanding in videos: A
grand challenge overview. In: Proceedings of the 27th ACM International
Conference on Multimedia (MM). pp. 2652–2656 (2019)
Sharma, P., Ding, N., Goodman, S., Soricut, R.: Conceptual captions: A cleaned,
hypernymed, image alt-text dataset for automatic image captioning. In: ACL.
pp. 2556–2565 (2018)
Su, W., Zhu, X., Cao, Y., Li, B., Lu, L., Wei, F., Dai, J.: Vl-bert:
Pre-training of generic visual-linguistic representations. In: International
Conference on Representation Learning (ICLR) (2020)
Sun, C., Myers, A., Vondrick, C., Murphy, K., Schmid, C.: Videobert: A joint
model for video and language representation learning. In: Proceedings of the
IEEE/CVF International Conference on Computer Vision (ICCV). pp. 7464–7473
(2019)
Tan, H., Bansal, M.: Lxmert: Learning cross-modality encoder representations
from transformers. In: EMNLP. pp. 5100–5111 (2019)
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N.,
Kaiser, Ł., Polosukhin, I.: Attention is all you need. In: Advances in
neural information processing systems (NeurIPS). vol. 30 (2017)
Wang, L., Chang, X., Li, S., Chu, Y., Li, H., Zhang, W., He, X., Song, L.,
Zhou, J., Yang, H.: Tcl: Transformer-based dynamic graph modelling via
contrastive learning. arXiv preprint arXiv:2105.07944 (2021)
Wang, X., Gupta, A.: Videos as space-time region graphs. In: European
conference on computer vision (ECCV). pp. 399–417 (2018)
Xiao, J., Shang, X., Yang, X., Tang, S., Chua, T.S.: Visual relation grounding
in videos. In: European Conference on Computer Vision (ECCV). pp. 447–464.
Springer (2020)
Xiao, J., Shang, X., Yao, A., Chua, T.S.: Next-qa: Next phase of
question-answering to explaining temporal actions. In: Proceedings of the
IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp.
9777–9786 (2021)
Xiao, J., Yao, A., Liu, Z., Li, Y., Ji, W., Chua, T.S.: Video as conditional
graph hierarchy for multi-granular question answering. In: Proceedings of the
AAAI Conference on Artificial Intelligence (AAAI). pp. 2804–2812 (2022)
Xie, S., Sun, C., Huang, J., Tu, Z., Murphy, K.: Rethinking spatiotemporal
feature learning: Speed-accuracy trade-offs in video classification. In:
European Conference on Computer Vision (ECCV). pp. 305–321 (2018)
Xu, D., Zhao, Z., Xiao, J., Wu, F., Zhang, H., He, X., Zhuang, Y.: Video
question answering via gradually refined attention over appearance and
motion. In: ACM MM. pp. 1645–1653 (2017)
Xu, H., Ghosh, G., Huang, P.Y., Okhonko, D., Aghajanyan, A., Metze, F.,
Zettlemoyer, L., Feichtenhofer, C.: Videoclip: Contrastive pre-training for
zero-shot video-text understanding. In: EMNLP. pp. 6787–6800 (2021)
Yang, A., Miech, A., Sivic, J., Laptev, I., Schmid, C.: Just ask: Learning to
answer questions from millions of narrated videos. In: Proceedings of the
IEEE/CVF International Conference on Computer Vision (ICCV). pp. 1686–1697
(2021)
Yi, K., Gan, C., Li, Y., Kohli, P., Wu, J., Torralba, A., Tenenbaum, J.B.:
Clevrer: Collision events for video representation and reasoning. In:
International Conference on Learning Representations (ICLR) (2019)
Ying, C., Cai, T., Luo, S., Zheng, S., Ke, G., He, D., Shen, Y., Liu, T.Y.: Do
transformers really perform badly for graph representation? Advances in
neural information processing systems (NeurIPS) 34 (2021)
Yu, W., Zheng, H., Li, M., Ji, L., Wu, L., Xiao, N., Duan, N.: Learning from
inside: Self-driven siamese sampling and reasoning for video question
answering. Advances in neural information processing systems (NeurIPS)
34 (2021)
Yu, Y., Kim, J., Kim, G.: A joint sequence fusion model for video question
answering and retrieval. In: European Conference on Computer Vision (ECCV).
pp. 471–487 (2018)
Yun, S., Jeong, M., Kim, R., Kang, J., Kim, H.J.: Graph transformer networks.
Advances in neural information processing systems (NeurIPS) 32
(2019)
Zellers, R., Lu, X., Hessel, J., Yu, Y., Park, J.S., Cao, J., Farhadi, A.,
Choi, Y.: Merlot: Multimodal neural script knowledge models. In: Advances in
neural information processing systems (NeurIPS). vol. 34 (2021)
Zhong, Y., Ji, W., Xiao, J., Li, Y., Deng, W., Chua, T.S.: Video question
answering: Datasets, algorithms and challenges. arXiv preprint
arXiv:2203.01225 (2022)
Zhu, L., Yang, Y.: Actbert: Learning global-local video-text representations.
In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern
Recognition (CVPR). pp. 8746–8755 (2020)
Zoph, B., Ghiasi, G., Lin, T.Y., Cui, Y., Liu, H., Cubuk, E.D., Le, Q.:
Rethinking pre-training and self-training. In: Advances in neural information
processing systems (NeurIPS). vol. 33, pp. 3833–3845 (2020)

Data Statistics
The statistical details of the experimented datasets are presented in Table {{table:60d5c189-ac81-4d84-afb6-808fe8781d09}} . For better comparison with previous works, we focus on the multi-choice QA task in NExT-QA {{cite:f0ba9b2}} though it has also defined open-ended QA. For TGIF-QA {{cite:3637f8a}}, we also conduct experiments on a latest version {{cite:ca4c154}} which generates more challenging negative answers for each question in the multi-choice tasks. In particular, we further fix the `redundant answer' issue as we find that there are about 10% of questions have redundant candidate answers and some of the candidate answers are even identical to the correct one. The rectified annotations will be released along with the code.
Data statistics. OE: Open-Ended QA. MC: Multi-Choice QA, VLen (s): Average video length in seconds.

Datasets
Main Challenges
#Videos/#QAs
Train
Val
Test
VLen (s)
QA
NExT-QA {{cite:f0ba9b2}}
Causal & Temporal Interaction
5.4K/48K
3.8K/34K
0.6K/5K
1K/9K
44
MC
3*TGIF-QA {{cite:3637f8a}}
Repetition Action
22.8K/22.7K
20.5K/20.5K
-
2.3K/2.3K
3
MC

State Transition
29.5K/58.9K
26.4K/52.7K
-
3.1K/6.2K
3
MC

Frame QA
39.5K/53.1K
32.3K/39.4K
-
7.1K/13.7K
3
OE
MSRVTT-QA {{cite:29f5b1b}}
Descriptive QA
10K/ 244K
6.5K/159K
0.5K/12K
3K/73K
15
OE
{{table:befcb059-f6aa-4f20-a8e6-227af4b795df}}{{table:60d5c189-ac81-4d84-afb6-808fe8781d09}}
Implementation Details
For training with QA annotations, we firstly train the whole model (except for the object detection model) end-to-end, and then freeze BERT to fine-tune the other parts of the best model obtained at the 1st stage. The best results in the two stages are determined as final results. Note that our hyper-parameters are mostly searched on the NExT-QA validation set and kept unchanged for other datasets. The maximum epoch varies from 10 to 30 among different datasets. For pretraining with data crawled from the Web, we randomly select 0.18M video-text data (less than 10%) from WebVid2.5M
https://m-bain.github.io/webvid-dataset/
{{cite:d931952}}. The videos are then extracted at 5 frames per second and are processed in the same way as for QA. We then optimize the model with an initial learning rate of {{formula:0827b664-e502-458c-ae5e-f5851bdca60d}}  and batch size 64. The number of negative descriptions of a video for cross-modal matching is set to 63, and they are randomly selected from the descriptions of other videos in the whole training set. Besides, a text token is corrupted at a probability of 15% in masked language modelling. Following {{cite:64f0318}}, a corrupted token will be replaced with 1) the `[MASK]' token by a chance of 80%, 2) a random token by a chance of 10%, and 3) the same token by a chance of 10%. We train the model by maximal 2 epochs which gives to the best generalization results, and it takes about 2 hours.

Additional Model Analysis
Similarity Comparison vs. Classification
To study the reason for the poor performance of the classification model variant described in Sec. REF  of the main text, we visualize the training and validation accuracy with regard to different training epochs in Fig. {{figure:590dbad3-7f26-4876-a992-fb4b5b937430}} . The results indicate that the classification model variant suffers from serious over-fitting issues, especially on NExT-QA {{cite:f0ba9b2}} whose QA contents are relative complex but with less training data. To study whether the problem comes from the classification formulation or the cross-modal transformer, we further substitute the cross-modal transformer (CM-Trans) with our cross-modal interaction (CM) module introduced in Sec. REF  of the main text. We find that such a substitution can slightly alleviate the problem. For example, on NExT-QA val set, the accuracy increases from 45.82% to 46.98%. Nevertheless, the performance is still much worse than a comparison-based model implementation (i.e. 55.02%). This experiment reveals two facts: 1) Formulating QA problem as classification is the major cause for the weak performance. 2) The cross-modal transformer exacerbates the over-fitting problem, possibly because it involves additional parameters.
Accuracy with regard to different training epochs.

{{figure:590dbad3-7f26-4876-a992-fb4b5b937430}}
Study of Video Sampling
In Fig. {{figure:8de5ae64-95cc-431b-aa37-d8aa94e90d01}} , we study the effect of sampled video clips and region proposals on NExT-QA {{cite:f0ba9b2}} test set. Regarding the number of sampled video clips, we find that the setting of 8 clips steadily wins on 4 clips. This is understandable as the videos in NExT-QA are relatively long. As for the sampled regions, when learning the model from scratch, the setting of 5 regions gives relatively better result, e.g., 53.68%. Nonetheless, when pretraining are considered, the setting of 20 regions gives better result, e.g., 55.70%. Such difference could be due to that learning with more regions can yield over-fitting issues when the dataset is not large enough, since the constructed graph become much larger and more complex. Our speculation is also supported by the fact that the accuracy increases with the number of sampled regions when we only sample 4 video clips and thus less number of total graph nodes.
Investigation of sampled video clips and region proposals per frame. Results are reported on NExT-QA test set.

{{figure:8de5ae64-95cc-431b-aa37-d8aa94e90d01}}
Model Efficiency
Comparison of memory and time based on NExT-QA {{cite:f0ba9b2}}. (2m{{formula:25f441fd-a72f-4143-8661-dfe83eec3244}} 8: 2 minutes per epoch and 8 epochs in total.)

2*Models
2*Acc@All
2*#Params (M)
GPU Memory
Time

Train
Infer
Train
Infer(FLOPs)
VQA-T [52]
45.30
156.5
5.6G
2.6G
2m{{formula:27bb7ab6-c952-4e9a-8d7a-56a5ed2c0f81}} 8
2448M
VGT (BERT)
55.02
133.7
16.2G
3.9G
7m{{formula:b3e6e718-2c8c-4341-828c-deb4f27f5f75}} 5
7121M
VGT (DistilBERT)
53.46
90.5
10.0G
3.5G
5m{{formula:f2654aaa-e3ad-4d1a-a72e-ce50209dbd6c}} 7
3922M
{{table:356aacf3-2493-4eab-a3cf-04cc0ca6029f}}{{table:d18aad4e-de6e-4198-9e3a-90c17e649119}}We compare VGT with VQA-T {{cite:64f0318}} in Tab. {{table:d18aad4e-de6e-4198-9e3a-90c17e649119}}  for better understanding of the memory and time cost. Experiments are done on 1 Tesla V100 GPU with batch size 64. We use 1 example to report inference FLOPs.
Memory: VGT has less training parameters (133.7M vs. 156.5M) and thus smaller model size than VQA-T (511M vs. 600M). The BERT encoder in VGT takes 82% of the parameters, the vision part is lightweight with only 24M parameters. VGT needs more GPU memory for training. Yet, the memory for inference are fairly small and close to that of VQA-T. We also implement a smaller version of VGT by replacing BERT with DistilBERT {{cite:f9bd18f}} as in VQA-T. With nearly 0.6{{formula:1dcec8ac-7646-45a0-9a28-31398cfa9f11}}  number of VQA-T's parameters (90.5/156.5M), we can still achieve strong performances (i.e. 53.46%).
Time: Our FLOPs on 1 example is {{formula:d72128d1-4bb8-48b7-8924-d86441ce1a4f}} 2.9{{formula:7c410e0d-cad0-43ad-a2f4-cc221cec4097}}  that of VQA-T and {{formula:1aba2931-e0bf-4b19-958d-233fdaeeaba2}} 1.6{{formula:7bb302a9-f7bf-478d-aec1-160510f48db3}}  if we use DistilBERT. However, VGT converges much faster and needs much fewer epochs (total FLOPs) to get results superior to VQA-T when training with the same data. For example, on NExT-QA, VGT's result at epoch 2 (50.16%) already significantly surpasses VQA-T's best result (45.30%) achieved at epoch 8. Also, VGT's result without pretraining can surpasses that of VQA-T pretrained with million-scale data. In this sense, VGT needs much fewer total FLOPs than VQA-T and other similar pretrained models for visual reasoning.


Gated Graph Sequence Neural Networks

Abstract: Abstract: Graph-structured data appears frequently in domains including chemistry, natural language semantics, social networks, and knowledge bases. In this work, we study feature learning techniques for graph-structured inputs. Our starting point is previous work on Graph Neural Networks (Scarselli et al., 2009), which we modify to use gated recurrent units and modern optimization techniques and then extend to output sequences. The result is a flexible and broadly useful class of neural network models that has favorable inductive biases relative to purely sequence-based models (e.g., LSTMs) when the problem is graph-structured. We demonstrate the capabilities on some simple AI (bAbI) and graph algorithm learning tasks. We then show it achieves state-of-the-art performance on a problem from program verification, in which subgraphs need to be matched to abstract data structures.

The More You Know: Using Knowledge Graphs for Image Classification

Abstract: One characteristic that sets humans apart from modern learning-based computer vision algorithms is the ability to acquire knowledge about the world and use that knowledge to reason about the visual world. Humans can learn about the characteristics of objects and the relationships that occur between them to learn a large variety of visual concepts, often with few examples. This paper investigates the use of structured prior knowledge in the form of knowledge graphs and shows that using this knowledge improves performance on image classification. We build on recent work on end-to-end learning on graphs, introducing the Graph Search Neural Network as a way of efficiently incorporating large knowledge graphs into a vision classification pipeline. We show in a number of experiments that our method outperforms standard neural network baselines for multi-label classification.

Deep Reasoning with Knowledge Graph for Social Relationship Understanding

Abstract: Social relationships (e.g., friends, couple etc.) form the basis of the social network in our daily life. Automatically interpreting such relationships bears a great potential for the intelligent systems to understand human behavior in depth and to better interact with people at a social level. Human beings interpret the social relationships within a group not only based on the people alone, and the interplay between such social relationships and the contextual information around the people also plays a significant role. However, these additional cues are largely overlooked by the previous studies. We found that the interplay between these two factors can be effectively modeled by a novel structured knowledge graph with proper message propagation and attention. And this structured knowledge can be efficiently integrated into the deep neural network architecture to promote social relationship understanding by an end-to-end trainable Graph Reasoning Model (GRM), in which a propagation mechanism is learned to propagate node message through the graph to explore the interaction between persons of interest and the contextual objects. Meanwhile, a graph attentional mechanism is introduced to explicitly reason about the discriminative objects to promote recognition. Extensive experiments on the public benchmarks demonstrate the superiority of our method over the existing leading competitors.

DRAG: Dynamic Region-Aware GCN for Privacy-Leaking Image Detection

Abstract: The daily practice of sharing images on social media raises a severe issue about privacy leakage. To address the issue, privacy-leaking image detection is studied recently, with the goal to automatically identify images that may leak privacy. Recent advance on this task benefits from focusing on crucial objects via pretrained object detectors and modeling their correlation. However, these methods have two limitations: 1) they neglect other important elements like scenes, textures, and objects beyond the capacity of pretrained object detectors; 2) the correlation among objects is fixed, but a fixed correlation is not appropriate for all the images. To overcome the limitations, we propose the Dynamic Region-Aware Graph Convolutional Network (DRAG) that dynamically finds out crucial regions including objects and other important elements, and models their correlation adaptively for each input image. To find out crucial regions, we cluster spatially-correlated feature channels into several region-aware feature maps. Further, we dynamically model the correlation with the self-attention mechanism and explore the interaction among the regions with a graph convolutional network. The DRAG achieved an accuracy of 87% on the largest dataset for privacy-leaking image detection, which is 10 percentage points higher than the state of the art. The further case study demonstrates that it found out crucial regions containing not only objects but other important elements like textures.

Main components of the architecture of graph-based methods.

Graph-based models for image classification

Prior knowledge structured as a knowledge graph can improve image classification performance {{cite:1d3f95f}}.
The Graph Search Neural Network (GSNN) {{cite:1d3f95f}} incorporates prior knowledge into Graph Neural Networks (GNN) {{cite:8784452}} to solve a multi-task vision classification problem (see Tab. {{table:bcdcf498-b715-4dfb-8a42-a7224c583ee9}} ). GSNN is based on the Gated Graph Neural Network (GGNN) {{cite:ef58cba}}, reducing computational requirements and observing the flow of information through the propagation model. GGNN uses the Gated Recurrent Unit (GRU) {{cite:00a9a1d}} to update the hidden state of each node with information from the neighbouring nodes.
GGNN is a differential recurrent neural network that operates on graph data representations, iteratively propagating the relationships to learn node-level and graph-level representations.
The Graph Reasoning Model (GRM) {{cite:34d49ed}} uses objects and interactions between people to predict social relationships.
The graph model weighs the predicted relationships with a graph attention mechanism based on Graph Attention Networks (GAT) {{cite:31fc0ac}}.
GRM uses prior knowledge on social relationships, co-occurrences and objects in the scene as a structured graph.
Interactions between people of interest and contextual objects are modelled by GGNN {{cite:ef58cba}}, where nodes are initialised with the corresponding semantic regions.
The model learns about relevant objects that carry relevant task-information.


CrowdHuman: A Benchmark for Detecting Human in a Crowd

Abstract: Human detection has witnessed impressive progress in recent years. However, the occlusion issue of detecting human in highly crowded environments is far from solved. To make matters worse, crowd scenarios are still under-represented in current human detection benchmarks. In this paper, we introduce a new dataset, called CrowdHuman, to better evaluate detectors in crowd scenarios. The CrowdHuman dataset is large, rich-annotated and contains high diversity. There are a total of $470K$ human instances from the train and validation subsets, and $~22.6$ persons per image, with various kinds of occlusions in the dataset. Each human instance is annotated with a head bounding-box, human visible-region bounding-box and human full-body bounding-box. Baseline performance of state-of-the-art detection frameworks on CrowdHuman is presented. The cross-dataset generalization results of CrowdHuman dataset demonstrate state-of-the-art performance on previous dataset including Caltech-USA, CityPersons, and Brainwash without bells and whistles. We hope our dataset will serve as a solid baseline and help promote future research in human detection tasks.

Age Progression/Regression by Conditional Adversarial Autoencoder

Abstract: If I provide you a face image of mine (without telling you the actual age when I took the picture) and a large amount of face images that I crawled (containing labeled faces of different ages but not necessarily paired), can you show me what I would look like when I am 80 or what I was like when I was 5? The answer is probably a No. Most existing face aging works attempt to learn the transformation between age groups and thus would require the paired samples as well as the labeled query image. In this paper, we look at the problem from a generative modeling perspective such that no paired samples is required. In addition, given an unlabeled image, the generative model can directly produce the image with desired age attribute. We propose a conditional adversarial autoencoder (CAAE) that learns a face manifold, traversing on which smooth age progression and regression can be realized simultaneously. In CAAE, the face is first mapped to a latent vector through a convolutional encoder, and then the vector is projected to the face manifold conditional on age through a deconvolutional generator. The latent vector preserves personalized face features (i.e., personality) and the age condition controls progression vs. regression. Two adversarial networks are imposed on the encoder and generator, respectively, forcing to generate more photo-realistic faces. Experimental results demonstrate the appealing performance and flexibility of the proposed framework by comparing with the state-of-the-art and ground truth.

Microsoft COCO: Common Objects in Context

Abstract: We present a new dataset with the goal of advancing the state-of-the-art in object recognition by placing the question of object recognition in the context of the broader question of scene understanding. This is achieved by gathering images of complex everyday scenes containing common objects in their natural context. Objects are labeled using per-instance segmentations to aid in precise object localization. Our dataset contains photos of 91 objects types that would be easily recognizable by a 4 year old. With a total of 2.5 million labeled instances in 328k images, the creation of our dataset drew upon extensive crowd worker involvement via novel user interfaces for category detection, instance spotting and instance segmentation. We present a detailed statistical analysis of the dataset in comparison to PASCAL, ImageNet, and SUN. Finally, we provide baseline performance analysis for bounding box and segmentation detection results using a Deformable Parts Model.

Overview of the recognition tasks used in the user study. The tasks can be divided into mainly two categories, Detection and Non-classification, and each task was presented to the participants together with different source unlabeled image datasets.

As discussed earlier, there are two potential challenges for classification-based design: difficulties for non-experts in setting up abstract categories and the existence of tasks that cannot be formulated as classification.
We prepared ten dummy tasks listed in Table {{table:a35795a7-3905-4e87-a003-67b049f7528d}}  corresponding to these two challenging cases.
The first five tasks correspond to the cases where users need to introduce abstract categories, i.e., categories encompassing multiple specific categories found in the images.
In task 1, for example, the expected text annotation should include category information as “this is a terrestrial animal” and “this is not a terrestrial animal”.
Annotations that describe specific animal names rather than target categories are considered failure cases.
The last five non-classification tasks are the cases that require continuous labels such as regression or segmentation.
The possibility of representing continuous quantities numerically in a language output has been discussed in recent studies {{cite:4860f62}}, {{cite:3e62c4b}}.
While it is assumed for non-classification tasks that numerical expressions are included in the ideal output text, the participants were allowed to linguistically express the rough quantity.
For example, in task 8, the participants could also describe the answer as “large parts are occupied” or “almost no plants or trees” instead of numerically expressing the percentage of the area.
In both cases, the dummy task descriptions given to the participants were only a use-case scenario and brief task requirements and did not provide specific instructions or concrete examples about the annotation.

Each task was presented to the participants together with different unlabeled source image datasets as shown in Table {{table:a35795a7-3905-4e87-a003-67b049f7528d}} .
We used existing classification datasets for the former five tasks and randomly selected 40 categories to match the smallest number of categories among all datasets.
Here we assume the scenario where non-expert users use image search engines or equivalent to collect images for annotation.
Images were stored in subfolders with their original category names to simulate the image search and collection process for the development of ML systems.
For the latter five tasks, we randomly selected 200 images from each dataset.
In this case, all images were stored and presented in a single folder.
In both cases, all image files were renamed as sequentially numbered files in random order.


AVA: A Video Dataset of Spatio-temporally Localized Atomic Visual Actions

Abstract: This paper introduces a video dataset of spatio-temporally localized Atomic Visual Actions (AVA). The AVA dataset densely annotates 80 atomic visual actions in 430 15-minute video clips, where actions are localized in space and time, resulting in 1.58M action labels with multiple labels per person occurring frequently. The key characteristics of our dataset are: (1) the definition of atomic visual actions, rather than composite actions; (2) precise spatio-temporal annotations with possibly multiple annotations for each person; (3) exhaustive annotation of these atomic actions over 15-minute video clips; (4) people temporally linked across consecutive segments; and (5) using movies to gather a varied set of action representations. This departs from existing datasets for spatio-temporal action recognition, which typically provide sparse annotations for composite actions in short video clips. We will release the dataset publicly. AVA, with its realistic scene and action complexity, exposes the intrinsic difficulty of action recognition. To benchmark this, we present a novel approach for action localization that builds upon the current state-of-the-art methods, and demonstrates better performance on JHMDB and UCF101-24 categories. While setting a new state of the art on existing datasets, the overall results on AVA are low at 15.6% mAP, underscoring the need for developing new approaches for video understanding.

COIN: A Large-Scale Dataset for Comprehensive Instructional Video Analysis

Abstract: There are substantial instruction videos on the Internet, which enables us to acquire knowledge for completing various tasks. However, most existing datasets for instruction video analysis have the limitations in diversity and scale, which makes them far from many real-world applications where more diverse activities occur. Moreover, it still remains a great challenge to organize and harness such data. To address these problems, we introduce a large-scale dataset called “COIN" for COmprehensive INstruction video analysis. Organized with a hierarchical structure, the COIN dataset contains 11,827 videos of 180 tasks in 12 domains (e.g., vehicles, gadgets, etc.) related to our daily life. With a new developed toolbox, all the videos are annotated effectively with a series of step descriptions and the corresponding temporal boundaries. Furthermore, we propose a simple yet effective method to capture the dependencies among different steps, which can be easily plugged into conventional proposal-based action detection methods for localizing important steps in instruction videos. In order to provide a benchmark for instruction video analysis, we evaluate plenty of approaches on the COIN dataset under different evaluation criteria. We expect the introduction of the COIN dataset will promote the future in-depth research on instruction video analysis for the community.

Cross-task weakly supervised learning from instructional videos

Abstract: In this paper we investigate learning visual models for the steps of ordinary tasks using weak supervision via instructional narrations and an ordered list of steps instead of strong supervision via temporal annotations. At the heart of our approach is the observation that weakly supervised learning may be easier if a model shares components while learning different steps: `pour egg' should be trained jointly with other tasks involving `pour' and `egg'. We formalize this in a component model for recognizing steps and a weakly supervised learning framework that can learn this model under temporal constraints from narration and the list of steps. Past data does not permit systematic studying of sharing and so we also gather a new dataset, CrossTask, aimed at assessing cross-task sharing. Our experiments demonstrate that sharing across tasks improves performance, especially when done at the component level and that our component model can parse previously unseen tasks by virtue of its compositionality.

HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips

Abstract: Learning text-video embeddings usually requires a dataset of video clips with manually provided captions. However, such datasets are expensive and time consuming to create and therefore difficult to obtain on a large scale. In this work, we propose instead to learn such embeddings from video data with readily available natural language annotations in the form of automatically transcribed narrations. The contributions of this work are three-fold. First, we introduce HowTo100M: a large-scale dataset of 136 million video clips sourced from 1.22M narrated instructional web videos depicting humans performing and describing over 23k different visual tasks. Our data collection procedure is fast, scalable and does not require any additional manual annotation. Second, we demonstrate that a text-video embedding trained on this data leads to state-of-the-art results for text-to-video retrieval and action localization on instructional video datasets such as YouCook2 or CrossTask. Finally, we show that this embedding transfers well to other domains: fine-tuning on generic Youtube videos (MSR-VTT dataset) and movies (LSMDC dataset) outperforms models trained on these datasets alone. Our dataset, code and models will be publicly available at: www.di.ens.fr/willow/research/howto100m/.

The Kinetics Human Action Video Dataset

Abstract: We describe the DeepMind Kinetics human action video dataset. The dataset contains 400 human action classes, with at least 400 video clips for each action. Each clip lasts around 10s and is taken from a different YouTube video. The actions are human focussed and cover a broad range of classes including human-object interactions such as playing instruments, as well as human-human interactions such as shaking hands. We describe the statistics of the dataset, how it was collected, and give some baseline performance figures for neural network architectures trained and tested for human action classification on this dataset. We also carry out a preliminary analysis of whether imbalance in the dataset leads to bias in the classifiers.

Every Moment Counts: Dense Detailed Labeling of Actions in Complex Videos

Abstract: Every moment counts in action recognition. A comprehensive understanding of human activity in video requires labeling every frame according to the actions occurring, placing multiple labels densely over a video sequence. To study this problem we extend the existing THUMOS dataset and introduce MultiTHUMOS, a new dataset of dense labels over unconstrained internet videos. Modeling multiple, dense labels benefits from temporal relations within and across classes. We define a novel variant of long short-term memory (LSTM) deep networks for modeling these temporal relations via multiple input and output connections. We show that this model improves action labeling accuracy and further enables deeper understanding tasks ranging from structured retrieval to action prediction.

UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild

Abstract: We introduce UCF101 which is currently the largest dataset of human actions. It consists of 101 action classes, over 13k clips and 27 hours of video data. The database consists of realistic user uploaded videos containing camera motion and cluttered background. Additionally, we provide baseline action recognition results on this new dataset using standard bag of words approach with overall performance of 44.5%. To the best of our knowledge, UCF101 is currently the most challenging dataset of actions due to its large number of classes, large number of clips and also unconstrained nature of such clips.

Towards Automatic Learning of Procedures from Web Instructional Videos

Abstract: The potential for agents, whether embodied or software, to learn by observing other agents performing procedures involving objects and actions is rich. Current research on automatic procedure learning heavily relies on action labels or video subtitles, even during the evaluation phase, which makes them infeasible in real-world scenarios. This leads to our question: can the human-consensus structure of a procedure be learned from a large set of long, unconstrained videos (e.g., instructional videos from YouTube) with only visual evidence? To answer this question, we introduce the problem of procedure segmentation--to segment a video procedure into category-independent procedure segments. Given that no large-scale dataset is available for this problem, we collect a large-scale procedure segmentation dataset with procedure segments temporally localized and described; we use cooking videos and name the dataset YouCook2. We propose a segment-level recurrent network for generating procedure segments by modeling the dependencies across segments. The generated segments can be used as pre-processing for other tasks, such as dense video captioning and event parsing. We show in our experiments that the proposed model outperforms competitive baselines in procedure segmentation.

YouTube-8M: A Large-Scale Video Classification Benchmark

Abstract: Many recent advancements in Computer Vision are attributed to large datasets. Open-source software packages for Machine Learning and inexpensive commodity hardware have reduced the barrier of entry for exploring novel approaches at scale. It is possible to train models over millions of examples within a few days. Although large-scale datasets exist for image understanding, such as ImageNet, there are no comparable size video classification datasets. In this paper, we introduce YouTube-8M, the largest multi-label video classification dataset, composed of ~8 million videos (500K hours of video), annotated with a vocabulary of 4800 visual entities. To get the videos and their labels, we used a YouTube video annotation system, which labels videos with their main topics. While the labels are machine-generated, they have high-precision and are derived from a variety of human-based signals including metadata and query click signals. We filtered the video labels (Knowledge Graph entities) using both automated and manual curation strategies, including asking human raters if the labels are visually recognizable. Then, we decoded each video at one-frame-per-second, and used a Deep CNN pre-trained on ImageNet to extract the hidden representation immediately prior to the classification layer. Finally, we compressed the frame features and make both the features and video-level labels available for download. We trained various (modest) classification models on the dataset, evaluated them using popular evaluation metrics, and report them as baselines. Despite the size of the dataset, some of our models train to convergence in less than a day on a single machine using TensorFlow. We plan to release code for training a TensorFlow model and for computing metrics.

Video datasets for self-supervised training and/or evaluation and the respective tasks associated with the annotations provided.

In self-supervised learning for videos, we have a large-scale dataset which is used in pre-training to learn representations. We have several smaller “down-stream" datasets that have annotations for specific tasks. Table {{table:1cdda826-0fa3-4080-a8e5-2e8845c4dbe8}}  summarizes the video-based datasets used in self-supervised learning. The most commonly used datasets for pre-training are the Kinetics {{cite:fb91edd}} and HowTo100M {{cite:7e158fb}} datasets. The most common datasets used for downstream evaluation are UCF101 {{cite:bedb2fc}} and HMDB51 {{cite:1b78eaf}}. Because there have been recent studies demonstrating that these two datasets do not necessarily require temporal learning {{cite:fecb10a}}, the Something-Something (SS) {{cite:8350fe9}} dataset has been proposed, but is not widely adopted for evaluation.


Supervised Learning for Phase Shift Design

The fingerprinting database to unveil mapping between measured user location and RIS unit cells to maximize received signal strength

An optimization where the DDPG algorithm is responsible to search for the optimum action at each decision during the network learning phase

A novel usupervised learning architecture for beamforming design optimization

The architecture created an active learning scheme that quickly adapts to changing environmental conditions with high performance

The feed-forward NN fed with reflected symbols performing phase matrix optimization

Supervised Learning approach for the secrecy rate optimization at receiver and phase configuration optimization on the RIS side with reduced computational complexity

Deep reinforcement learning approach achieving higher SNR for noisy channels and mobile scenarios comparing to conventional techniques

A method that addresses the challenge in path loss optimization for RIS-aided terahertz communication having high molecular absorption and attenuation

The MLP-based NN addressing dynamic positioning of multiple RISs to overcome the storage and computational performance limitations

Deep Reinforcement Learning-based novel architecture capable of learning channel behaviour

Indoor Signal Focusing with Deep Learning Designed Reconfigurable Intelligent Surfaces

Abstract: Reconfigurable Intelligent Surfaces (RISs) comprised of tunable unit elements have been recently considered in indoor communication environments for focusing signal reflections to intended user locations. However, the current proofs of concept require complex operations for the RIS configuration, which are mainly realized via wired control connections. In this paper, we present a deep learning method for efficient online wireless configuration of RISs when deployed in indoor communication environments. According to the proposed method, a database of coordinate fingerprints is implemented during an offline training phase. This fingerprinting database is used to train the weights and bias of a properly designed Deep Neural Network (DNN), whose role is to unveil the mapping between the measured coordinate information at a user location and the configuration of the RIS's unit cells that maximizes this user's received signal strength. During the online phase of the presented method, the trained DNN is fed with the measured position information at the target user to output the optimal phase configurations of the RIS for signal power focusing on this intended location. Our realistic simulation results using ray tracing on a three dimensional indoor environment demonstrate that the proposed DNN-based configuration method exhibits its merits for all considered cases, and effectively increases the achievable throughput at the target user location.

Optimization-driven Machine Learning for Intelligent Reflecting Surfaces Assisted Wireless Networks

Abstract: Intelligent reflecting surface (IRS) has been recently employed to reshape the wireless channels by controlling individual scattering elements' phase shifts, namely, passive beamforming. Due to the large size of scattering elements, the passive beamforming is typically challenged by the high computational complexity and inexact channel information. In this article, we focus on machine learning (ML) approaches for performance maximization in IRS-assisted wireless networks. In general, ML approaches provide enhanced flexibility and robustness against uncertain information and imprecise modeling. Practical challenges still remain mainly due to the demand for a large dataset in offline training and slow convergence in online learning. These observations motivate us to design a novel optimization-driven ML framework for IRS-assisted wireless networks, which takes both advantages of the efficiency in model-based optimization and the robustness in model-free ML approaches. By splitting the decision variables into two parts, one part is obtained by the outer-loop ML approach, while the other part is optimized efficiently by solving an approximate problem. Numerical results verify that the optimization-driven ML approach can improve both the convergence and the reward performance compared to conventional model-free learning approaches.

Resource Allocation In IRSs Aided MISO-NOMA Networks: A Machine Learning Approach

Abstract: A novel framework of intelligent reflecting surface (IRS)-aided multiple-input single-output (MISO) non-orthogonal multiple access (NOMA) network is proposed, where a base station (BS) serves multiple clusters with unfixed number of users in each cluster. The goal is to maximize the sum rate of all users by jointly optimizing the passive beamforming vector at the IRS, decoding order and power allocation coefficient vector, subject to the rate requirements of users. In order to tackle the formulated problem, a three-step approach is proposed. More particularly, a long short-term memory (LSTM) based algorithm is first adopted for predicting the mobility of users. Secondly, a K-means based Gaussian mixture model (K-GMM) algorithm is proposed for user clustering. Thirdly, a deep Q-network (DQN) based algorithm is invoked for jointly determining the phase shift matrix and power allocation policy. Simulation results are provided for demonstrating that the proposed algorithm outperforms the benchmarks, while the performance of IRS-NOMA system is better than IRS-OMA system.

Deep Reinforcement Learning for Intelligent Reflecting Surfaces: Towards Standalone Operation

Abstract: The promising coverage and spectral efficiency gains of intelligent reflecting surfaces (IRSs) are attracting increasing interest. To adopt these surfaces in practice, however, several challenges need to be addressed. One of these main challenges is how to configure the reflecting coefficients on these passive surfaces without requiring massive channel estimation or beam training overhead. Earlier work suggested leveraging supervised learning tools to predict the IRS reflection matrices. While this approach has the potential of reducing the beam training overhead, it requires collecting large datasets for training the neural network models. In this paper, we propose a novel deep reinforcement learning framework for predicting the IRS reflection matrices with minimal beam training overhead. Simulation results show that the proposed online learning framework can converge to the optimal rate that assumes perfect channel knowledge. This represents an important step towards realizing a standalone IRS operation, where the surface configures itself without any control from the infrastructure.

Deep Learning-based Phase Reconfiguration for Intelligent Reflecting Surfaces

Abstract: Intelligent reflecting surfaces (IRSs), consisting of reconfigurable metamaterials, have recently attracted attention as a promising cost-effective technology that can bring new features to wireless communications. These surfaces can be used to partially control the propagation environment and can potentially provide a power gain that is proportional to the square of the number of IRS elements when configured in a proper way. However, the configuration of the local phase matrix at the IRSs can be quite a challenging task since they are purposely designed to not have any active components, therefore, they are not able to process any pilot signal. In addition, a large number of elements at the IRS may create a huge training overhead. In this paper, we present a deep learning (DL) approach for phase reconfiguration at an IRS in order to learn and make use of the local propagation environment. The proposed method uses the received pilot signals reflected through the IRS to train the deep feedforward network. The performance of the proposed approach is evaluated and the numerical results are presented.

Truly Intelligent Reflecting Surface-Aided Secure Communication Using Deep Learning

Abstract: This paper considers machine learning for physical layer security design for communication in a challenging wireless environment. The radio environment is assumed to be programmable with the aid of a meta material-based intelligent reflecting surface (IRS) allowing customisable path loss, multi-path fading and interference effects. In particular, the fine-grained reflections from the IRS elements are exploited to create channel advantage for maximizing the secrecy rate at a legitimate receiver. A deep learning (DL) technique has been developed to tune the reflections of the IRS elements in real-time. Simulation results demonstrate that the DL approach yields comparable performance to the conventional approaches while significantly reducing the computational complexity.

On the Robustness of Deep Reinforcement Learning in IRS-Aided Wireless Communications Systems

Abstract: We consider an Intelligent Reflecting Surface (IRS)-aided multiple-input single-output (MISO) system for downlink transmission. We compare the performance of Deep Reinforcement Learning (DRL) and conventional optimization methods in finding optimal phase shifts of the IRS elements to maximize the user signal-to-noise (SNR) ratio. Furthermore, we evaluate the robustness of these methods to channel impairments and changes in the system. We demonstrate numerically that DRL solutions show more robustness to noisy channels and user mobility.

Hybrid Beamforming for RIS-Empowered Multi-hop Terahertz Communications: A DRL-based Method

Abstract: Wireless communication in the TeraHertz band (0.1–10 THz) is envisioned as one of the key enabling technologies for the future six generation (6G) wireless communication systems. However, very high propagation attenuations and molecular absorptions of THz frequencies often limit the signal transmission distance and coverage range. Benefited from the recent breakthrough on the reconfigurable intelligent surfaces (RIS) for realizing smart radio propagation environment, we propose a novel hybrid beamforming scheme for the multi-hop RIS-assisted communication networks to improve the coverage range at THz-band frequencies. We investigate the joint design of digital beamforming matrix at the BS and analog beamforming matrices at the RISs, by leveraging the recent advances in deep reinforcement learning (DRL) to combat the propagation loss. Simulation results show that our proposed scheme is able to improve 50% more coverage range of THz communications compared with the benchmarks. Furthermore, it is also shown that our proposed DRL-based method is a state-of-the-art method to solve the NP-bard beamforming problem, especially when the signals at RIS-empowered THz communication networks experience multiple hops.

Phase Configuration Learning in Wireless Networks with Multiple Reconfigurable Intelligent Surfaces

Abstract: Reconfigurable Intelligent Surfaces (RISs) are recently gaining remarkable attention as a low-cost, hardware-efficient, and highly scalable technology capable of offering dynamic control of electro-magnetic wave propagation. Their envisioned dense deployment over various obstacles of the, otherwise passive, wireless communication environment has been considered as a revolutionary means to transform them into network entities with reconfigurable properties, providing increased environmental intelligence for diverse communication objectives. One of the major challenges with RIS-empowered wireless communications is the low-overhead dynamic configuration of multiple RISs, which according to the current hardware designs have very limited computing and storage capabilities. In this paper, we consider a typical communication pair between two nodes that is assisted by a plurality of RISs, and devise low-complexity supervised learning approaches for the RISs’ phase configurations. By assuming common tunable phases in groups of each RIS’s unit elements, we present multi-layer perceptron Neural Network (NN) architectures that can be trained either with positioning values or the instantaneous channel coefficients. We investigate centralized and individual training of the RISs, as well as their federation, and assess their computational requirements. Our simulation results, including comparisons with the optimal phase configuration scheme, showcase the benefits of adopting individual NNs at RISs for the link budget performance boosting.

Deep Learning-Empowered Predictive Beamforming for IRS-Assisted Multi-User Communications

Abstract: The realization of practical intelligent reflecting surface (IRS)-assisted multi-user communication (IRS-MUC) systems critically depends on the proper beamforming design exploiting accurate channel state information (CSI). However, channel estimation (CE) in IRS-MUC systems requires a significantly large training overhead due to the numerous reflection elements involved in IRS. In this paper, we adopt a deep learning approach to implicitly learn the historical channel features and directly predict the IRS phase shifts for the next time slot to maximize the average achievable sum-rate of an IRS-MUC system taking into account the user mobility. By doing this, only a low-dimension multiple-input single-output (MISO) CE is needed for transmit beamforming design, thus significantly reducing the CE overhead. To this end, a location-aware convolutional long short-term memory network (LA-CLNet) is first developed to facilitate predictive beamforming at IRS, where the convolutional and recurrent units are jointly adopted to exploit both the spatial and temporal features of channels simultaneously. Given the predictive IRS phase shift beamforming, an instantaneous CSI (ICSI)-aware fully-connected neural network (IA-FNN) is then proposed to optimize the transmit beamforming matrix at the access point. Simulation results demonstrate that the sum-rate performance achieved by the proposed method approaches that of the genie-aided scheme with the full perfect ICSI.

An overview of DL-based passive beamforming design studies.

Passive Beamforming Design with DL for System Optimization

Passive beamforming design is among the most critical problems for the RIS technology to be included in the next generation of communication systems. The success of phase reconfiguration not only increases the quality of signal transformation but also increases the adaptability of RISs to different communication environments such as indoor/outdoor applications. Especially in highly dynamic channels, such as application scenarios with high mobility, the inadequacy of existing control systems has revealed the necessity of a new perspective. In preserving the fully passive nature of RISs with reduced costs and system complexity, truly intelligent approaches have a bright future for next-generation communications. Therefore, DL-based passive beamforming techniques have extensively been addressed by the literature as presented in Table {{table:818c6bd3-25be-4608-ba86-04fbbcd48b3f}} .
Current DL-based studies on RIS-aided communication systems are composed of various NN structures depending on their advantages for target applications. mlp, a member of the feed-forward artificial-NN and one of the most basic types of DNN, can be given as an instance. {{cite:ad556f6}} offers an mlp architecture for beamforming design on the RIS side. The considered RIS scheme, which has a hybrid structure in which active and passive elements are present simultaneously, performs pilot training with randomly distributed active components. The trained data set, in which the beamforming on the RIS side is established as an input-output pair, is built with the supervised learning scheme. The use of active elements can be considered as a drawback of this study. The possible advantages of the supervised learning-based MLP algorithm with fully passive RIS have been inspected in {{cite:fbdd88a}} to overcome the storage and computational performance limitations in the dynamic positioning of multiple RISs. The MLP-based NN has fed by dynamic channel information as well as RIS positioning information. The proposed position-trained NN results are superior to benchmark results retrieved by an exhaustive study.


Facial Attribute Capsules for Noise Face Super Resolution

Abstract: Existing face super-resolution (SR) methods mainly assume the input image to be noise-free. Their performance degrades drastically when applied to real-world scenarios where the input image is always contaminated by noise. In this paper, we propose a Facial Attribute Capsules Network (FACN) to deal with the problem of high-scale super-resolution of noisy face image. Capsule is a group of neurons whose activity vector models different properties of the same entity. Inspired by the concept of capsule, we propose an integrated representation model of facial information, which named Facial Attribute Capsule (FAC). In the SR processing, we first generated a group of FACs from the input LR face, and then reconstructed the HR face from this group of FACs. Aiming to effectively improve the robustness of FAC to noise, we generate FAC in semantic, probabilistic and facial attributes manners by means of integrated learning strategy. Each FAC can be divided into two sub-capsules: Semantic Capsule (SC) and Probabilistic Capsule (PC). Them describe an explicit facial attribute in detail from two aspects of semantic representation and probability distribution. The group of FACs model an image as a combination of facial attribute information in the semantic space and probabilistic space by an attribute-disentangling way. The diverse FACs could better combine the face prior information to generate the face images with fine-grained semantic attributes. Extensive benchmark experiments show that our method achieves superior hallucination results and outperforms state-of-the-art for very low resolution (LR) noise face image super resolution.

Learning Spatial Attention for Face Super-Resolution

Abstract: General image super-resolution techniques have difficulties in recovering detailed face structures when applying to low resolution face images. Recent deep learning based methods tailored for face images have achieved improved performance by jointly trained with additional task such as face parsing and landmark prediction. However, multi-task learning requires extra manually labeled data. Besides, most of the existing works can only generate relatively low resolution face images (e.g., $128\times128$), and their applications are therefore limited. In this paper, we introduce a novel SPatial Attention Residual Network (SPARNet) built on our newly proposed Face Attention Units (FAUs) for face super-resolution. Specifically, we introduce a spatial attention mechanism to the vanilla residual blocks. This enables the convolutional layers to adaptively bootstrap features related to the key face structures and pay less attention to those less feature-rich regions. This makes the training more effective and efficient as the key face structures only account for a very small portion of the face image. Visualization of the attention maps shows that our spatial attention network can capture the key face structures well even for very low resolution faces (e.g., $16\times16$). Quantitative comparisons on various kinds of metrics (including PSNR, SSIM, identity similarity, and landmark detection) demonstrate the superiority of our method over current state-of-the-arts. We further extend SPARNet with multi-scale discriminators, named as SPARNetHD, to produce high resolution results (i.e., $512\times512$). We show that SPARNetHD trained with synthetic data cannot only produce high quality and high resolution outputs for synthetically degraded face images, but also show good generalization ability to real world low quality face images.

Speed and overhead comparison of typical non-blind methods that measured on {{formula:172dbaac-8075-455b-9f86-bbcc7fcc36e1}}  images. We test all models using an NVIDIA RTX 3090 GPU.

Regarding the non-blind task, we chose to focus on evaluating non-blind super-resolution methods due to their predominant emphasis in the field. TABLE {{table:bb73cdc2-44bc-4c6e-b31b-f6b96235d97b}}  presents a compilation of ten state-of-the-art non-blind methods, including fine-tuned image restoration methods {{cite:0906e90}}, {{cite:98b7b36}}, methods based on attention mechanisms {{cite:d862dbe}}, {{cite:11ec04a}}, {{cite:6b39cff}}, {{cite:b3c5cf4}}, and methods relying on various priors {{cite:e92d270}}, {{cite:d342737}}, {{cite:ca6764f}}. Among these, methods employing hybrid attention mechanisms, namely CTCNet {{cite:11ec04a}}, SCTANet {{cite:6b39cff}}, and SFMNet {{cite:b3c5cf4}}, achieve either the best or second-best performance across all metrics on both test sets. TABLE {{table:68215d8c-339a-4dd6-a4b5-9a36144eb9c1}}  provides detailed information about the model characteristics of these methods, including parameters, computation, and inference duration. Furthermore, Fig. {{table:bb73cdc2-44bc-4c6e-b31b-f6b96235d97b}}  visually illustrates the efficiency of these techniques through three perspectives: performance, inference speed, and model size. Notably, attention-based methods, particularly SFMNet {{cite:b3c5cf4}}, stand out as they achieve superior performance while maintaining smaller computational loads. Finally, Fig. {{figure:86aa0bbe-658b-4e1f-841d-e6f41b2f14fc}}  provides a visual comparison of these methods.

A range of state-of-the-art methods were selected for the blind task, including approaches that do not rely on prior such as network architecture design {{cite:81887e2}}, {{cite:ddc3e7b}} and diffusion modeling techniques {{cite:c9cbf43}}). Additionally, techniques utilizing internally-specific priors such as parsing maps {{cite:16eda58}} and 3D face shapes {{cite:8f7b469}} were considered. Furthermore, methods employing external compensatory prior like pre-trained StyleGAN prior {{cite:85ae21f}}, {{cite:33d5cb1}}, {{cite:8f7b469}}, pre-trained VQGAN prior {{cite:850db8f}}, {{cite:e8b72a5}}, face dictionary {{cite:78c17b0}}, {{cite:9f5f2cd}}, and reference prior {{cite:60adc34}}) were also included. To evaluate the application scope of non-blind and blind methods, we randomly selected several real face photos and restored them using the fine-tuned non-blind method SFMNet {{cite:b3c5cf4}} and the blind method GFPGAN {{cite:33d5cb1}} respectively. As shown in Fig. {{table:68215d8c-339a-4dd6-a4b5-9a36144eb9c1}} , it is evident that SFMNet struggles to effectively handle real-world photos, while GFPGAN, despite showing some racial bias in certain images, generally offers superior visual quality. Consequently, the blind method holds greater promise for real-world applications.