Aroma: code recommendation via structural code search

Abstract: Programmers often write code that has similarity to existing code written somewhere. A tool that could help programmers to search such similar code would be immensely useful. Such a tool could help programmers to extend partially written code snippets to completely implement necessary functionality, help to discover extensions to the partial code which are commonly included by other programmers, help to cross-check against similar code written by other programmers, or help to add extra code which would fix common mistakes and errors. We propose Aroma, a tool and technique for code recommendation via structural code search. Aroma indexes a huge code corpus including thousands of open-source projects, takes a partial code snippet as input, searches the corpus for method bodies containing the partial code snippet, and clusters and intersects the results of the search to recommend a small set of succinct code snippets which both contain the query snippet and appear as part of several methods in the corpus. We evaluated Aroma on 2000 randomly selected queries created from the corpus, as well as 64 queries derived from code snippets obtained from Stack Overflow, a popular website for discussing code. We implemented Aroma for 4 different languages, and developed an IDE plugin for Aroma. Furthermore, we conducted a study where we asked 12 programmers to complete programming tasks using Aroma, and collected their feedback. Our results indicate that Aroma is capable of retrieving and recommending relevant code snippets efficiently.

SourcererCC: Scaling Code Clone Detection to Big-Code

Abstract: Despite a decade of active research, there has been a marked lack in clone detection techniques that scale to large repositories for detecting near-miss clones. In this paper, we present a token-based clone detector, SourcererCC, that can detect both exact and near-miss clones from large inter-project repositories using a standard workstation. It exploits an optimized inverted-index to quickly query the potential clones of a given code block. Filtering heuristics based on token ordering are used to significantly reduce the size of the index, the number of code-block comparisons needed to detect the clones, as well as the number of required token-comparisons needed to judge a potential clone. We evaluate the scalability, execution time, recall and precision of SourcererCC, and compare it to four publicly available and state-of-the-art tools. To measure recall, we use two recent benchmarks: (1) a big benchmark of real clones, BigCloneBench, and (2) a Mutation/Injection-based framework of thousands of fine-grained artificial clones. We find SourcererCC has both high recall and precision, and is able to scale to a large inter-project repository (25K projects, 250MLOC) using a standard workstation.

Code clone search tools.

A code clone search tool takes a piece of code as a query and returns a list of similar codes {{cite:c44d7d7}}, {{cite:759efa5}}, {{cite:7c78930}}. It differs from code clone detection {{cite:52c5d1a}}, {{cite:b8ed8d8}} because it is query-centric, retrieves only clones that are associate with the query, instead of looking for a complete set of clone pairs in the codebase as the clone detection, and cares about the tool scalability {{cite:c44d7d7}}. Table {{table:b42e72e4-96e4-4434-aba4-0f21cc72d71a}}  shows the reviewed studies related to the code clone search task.

When developing software user interfaces (UIs) developers commonly draft UI sketches and implement corresponding UI code with related APIs. This often takes enormous efforts. UI code search tools take UI sketches as a query and search for UI code snippets that match the requirement of the UI sketch from a codebase. Table {{table:b42e72e4-96e4-4434-aba4-0f21cc72d71a}}  shows three UI code search tools we reviewed in this study. Reiss et al. {{cite:6729224}} converted the image of a developer's UI sketch into a scalable vector graphic (SVG) diagram, and reduced the search space by using an existing text-based search engine {{formula:108ab595-6805-43c5-a0e3-13d37c75d33a}}  {{cite:05edf5b}} with related keywords. After a series of transformations for the candidate code, this tool returns the ones that can be compiled and run. Behrang et al. {{cite:082b509}} proposed a similar tool GUIFetch but measures the query-code relevancy with a comprehensive metric based on the screen similarity in terms of the screen type, size, and position, and the screen transition similarity. To improve query representation, Xie et al. {{cite:785e7db}} adapted the pix2code {{cite:06b1018}} tool to automatically extract the code structure from a UI sketch. Pix2code uses a DL model that can capture the UI components types and their hierarchical relationships, which was trained with manually labeled UI sketches. To sort candidate UI code, Xie et al. {{cite:785e7db}} measured the layout distance between query and candidate code based on the Levenshtein distance {{cite:1d67ced}}.


Basketball video with single moving camera

Light Cascaded Convolutional Neural Networks for Accurate Player Detection

Abstract: Vision based player detection is important in sports applications. Accuracy, efficiency, and low memory consumption are desirable for real-time tasks such as intelligent broadcasting and automatic event classification. In this paper, we present a cascaded convolutional neural network (CNN) that satisfies all three of these requirements. Our method first trains a binary (player/non-player) classification network from labeled image patches. Then, our method efficiently applies the network to a whole image in testing. We conducted experiments on basketball and soccer games. Experimental results demonstrate that our method can accurately detect players under challenging conditions such as varying illumination, highly dynamic camera movements and motion blur. Comparing with conventional CNNs, our approach achieves state-of-the-art accuracy on both games with 1000x fewer parameters (i.e., it is light}.

Single-Camera Basketball Tracker through Pose and Semantic Feature Fusion

Abstract: Tracking sports players is a widely challenging scenario, specially in single-feed videos recorded in tight courts, where cluttering and occlusions cannot be avoided. This paper presents an analysis of several geometric and semantic visual features to detect and track basketball players. An ablation study is carried out and then used to remark that a robust tracker can be built with Deep Learning features, without the need of extracting contextual ones, such as proximity or color similarity, nor applying camera stabilization techniques. The presented tracker consists of: (1) a detection step, which uses a pretrained deep learning model to estimate the players pose, followed by (2) a tracking step, which leverages pose and semantic information from the output of a convolutional layer in a VGG network. Its performance is analyzed in terms of MOTA over a basketball dataset with more than 10k instances.

Summary of deep learning methods and application in team sports (AUC: Area Under Curve; mAP: mean Average Precision; MAPE: Mean Absolute Percentage Error; MOTA: Multi Object Tracking Accuracy)

Several recent studies have attempted to assess the performance of deep learning methods in sports analytics. The core idea of all methods is to use CNN. However, each study proposes a different structure of the network and training method for increasing the performance. In this section, we summarize the state-of-the-art networks and their application in sports analytics. Table {{table:84177e33-904f-4d9c-b061-f2ddd1bc511c}}  is a brief review of these methods.


Fine-Grained Visual Classification of Aircraft

Abstract: This paper introduces FGVC-Aircraft, a new dataset containing 10,000 images of aircraft spanning 100 aircraft models, organised in a three-level hierarchy. At the finer level, differences between models are often subtle but always visually measurable, making visual recognition challenging but possible. A benchmark is obtained by defining corresponding classification tasks and evaluation protocols, and baseline results are presented. The construction of this dataset was made possible by the work of aircraft enthusiasts, a strategy that can extend to the study of number of other object classes. Compared to the domains usually considered in fine-grained visual classification (FGVC), for example animals, aircraft are rigid and hence less deformable. They, however, present other interesting modes of variation, including purpose, size, designation, structure, historical style, and branding.

Fine-Grained Car Detection for Visual Census Estimation

Abstract: Targeted socio-economic policies require an accurate understanding of a country’s demographic makeup. To that end, the United States spends more than 1 billion dollars a year gathering census data such as race, gender, education, occupation and unemployment rates. Compared to the traditional method of collecting surveys across many years which is costly and labor intensive, data-driven, machine learning-driven approaches are cheaper and faster—with the potential ability to detect trends in close to real time. In this work, we leverage the ubiquity of Google Street View images and develop a computer vision pipeline to predict income, per capita carbon emission, crime rates and other city attributes from a single source of publicly available visual data. We first detect cars in 50 million images across 200 of the largest US cities and train a model to predict demographic attributes using the detected cars. To facilitate our work, we have collected the largest and most challenging fine-grained dataset reported to date consisting of over 2600 classes of cars comprised of images from Google Street View and other web sources, classified by car experts to account for even the most subtle of visual differences. We use this data to construct the largest scale fine-grained detection system reported to date. Our prediction results correlate well with ground truth income data (r=0.82), Massachusetts department of vehicle registration, and sources investigating crime rates, income segregation, per capita carbon emission, and other market research. Finally, we learn interesting relationships between cars and neighborhoods allowing us to perform the first large scale sociological analysis of cities using computer vision techniques.

RPC: A Large-Scale Retail Product Checkout Dataset

Abstract: Over recent years, emerging interest has occurred in integrating computer vision technology into the retail industry. Automatic checkout (ACO) is one of the critical problems in this area which aims to automatically generate the shopping list from the images of the products to purchase. The main challenge of this problem comes from the large scale and the fine-grained nature of the product categories as well as the difficulty for collecting training images that reflect the realistic checkout scenarios due to continuous update of the products. Despite its significant practical and research value, this problem is not extensively studied in the computer vision community, largely due to the lack of a high-quality dataset. To fill this gap, in this work we propose a new dataset to facilitate relevant research. Our dataset enjoys the following characteristics: (1) It is by far the largest dataset in terms of both product image quantity and product categories. (2) It includes single-product images taken in a controlled environment and multi-product images taken by the checkout system. (3) It provides different levels of annotations for the check-out images. Comparing with the existing datasets, ours is closer to the realistic setting and can derive a variety of research problems. Besides the dataset, we also benchmark the performance on this dataset with various approaches. The dataset and related resources can be found at \url{this https URL}.

Summary of the popular FGVC datasets. “Source" refers to the acquisition manner of the data, i.e., collected from website or by photographing in real scenes. “Imbalance" represents the number of images in the largest class divided by the number of images in the smallest class. “Insts" refers to the number of all instances in the dataset. “BBox" and “Parts" refer to the object bounding box annotations and the part annotations, respectively. “Texts" refers to whether fine-grained text descriptions are provided for each image. “n/a" denotes “not available".

To see the above issue, we summarize the statistics of the commonly used FGVC datasets in Table {{table:6fd2b425-b5ed-4d4a-b2fa-66a995678511}} , which reveals two trends along the development of FGVC.
First, along with time, the collected images tend to cover either increasing instances or increasing categories. Especially, the most recent dataset {{cite:de47cde}} has a nearly ten-fold increase in the number of instances compared to the earliest one {{cite:21c4988}}. In this development, many efforts have been devoted to pursuing more and more images to enrich the descriptions of instances, encouraging the data-driven models to extract more discriminative features for the instances. This is particularly important in FGVC since its performance heavily relies on the discriminability and the stability of the instance appearance in different environments. However, all the mentioned datasets unfortunately ignore the fact that environment may change the appearances of instances periodically or change their morphological states permanently, which is considered as the issue of concept drift. This exists widely in real world, particularly in all cases of biological growth and biological adaption such as human face, animals, and plants which have varied appearances in different ages and different environments. Thus, the existing FGVC methods usuallly have degenerated performance since the shifted visual features do not follow the distribution characterized by the dataset during model training. Another factor of this failure is that the existing models do not possess the ability of adapting to the changes of the instances in the deployed environment.
Second, the collected images tend to follow a comprehensive distribution of the real instances. An important viewpoint conveys that the instances in real world often follow a long-tailed distribution as the instance frequency of each class is generally different. As a result, a collected dataset could have a large number of under-represented classes and a few classes with more than sufficient instances {{cite:69d1ac7}}. However, this issue has not been considered in the widely used datasets {{cite:d6240b4}}, {{cite:916ec37}}, {{cite:ef19882}}, but been noted in the recently published datasets {{cite:245d684}}, {{cite:de47cde}} that collect data from real world. Considering that the Internet images are often prettified in which case the included instances exhibit healthy, tidy, and gorgeous appearances, the images photographed in real world present more natural contexts. Specifically, the instances in the same species cannot be in a consistent state all the time in real world, even in the same place and at the same time. The state is affected by various factors such as light and water. However, the previous datasets only consider one or several collection areas,
which may restrict the diversity of the instance images. This brings the issue that the imbalance in the distribution of collected data could not adequately describe the challenge of the real world long-tailed distribution.


Deep Multigrid: learning prolongation and restriction matrices

Abstract: This paper proposes the method to optimize restriction and prolongation operators in the two-grid method. The proposed method is straightforwardly extended to the geometric multigrid method (GMM). GMM is used in solving discretized partial differential equation (PDE) and based on the restriction and prolongation operators. The operators are crucial for fast convergence of GMM, but they are unknown. To find them we propose a reformulation of the two-grid method in terms of a deep neural network with a specific architecture. This architecture is based on the idea that every operation in the two-grid method can be considered as a layer of a deep neural network. The parameters of layers correspond to the restriction and prolongation operators. Therefore, we state an optimization problem with respect to these operators and get optimal ones through backpropagation approach. To illustrate the performance of the proposed approach, we carry out experiments on the discretized Laplace equation, Helmholtz equation and singularly perturbed convection-diffusion equation and demonstrate that proposed approach gives operators, which lead to faster convergence.

Machine-Learned Preconditioners for Linear Solvers in Geophysical Fluid Flows

Abstract: It is tested whether machine learning methods can be used for preconditioning to increase the performance of the linear solver -- the backbone of the semi-implicit, grid-point model approach for weather and climate models. Embedding the machine-learning method within the framework of a linear solver circumvents potential robustness issues that machine learning approaches are often criticized for, as the linear solver ensures that a sufficient, pre-set level of accuracy is reached. The approach does not require prior availability of a conventional preconditioner and is highly flexible regarding complexity and machine learning design choices. Several machine learning methods are used to learn the optimal preconditioner for a shallow-water model with semi-implicit timestepping that is conceptually similar to more complex atmosphere models. The machine-learning preconditioner is competitive with a conventional preconditioner and provides good results even if it is used outside of the dynamical range of the training dataset.

Representative researches in three aspects

The Development of Intelligent Iterative Methods

Over recent decades, substantial research has been dedicated to intelligent iterative methods, exploring method, component, and parameter perspectives. Pertinent literature is outlined in Table {{table:c4ad5de1-f963-4142-8a42-0c9727a30f59}} . Considering the varying components of different iterative methods and the diverse optimization strategies applied to distinct components, the second column “Iterative" signifies the corresponding iterative methods encompassing these components. Meanwhile, the “Precond" designation indicates literature employing neural networks to optimize the preconditioner {{formula:f8d76ce0-5914-481e-8b31-ac8ae95ed46c}} . This section offers an overview of advancements and the present state of intelligent iterative methods.


MultiEURLEX - A multi-lingual and multi-label legal document classification dataset for zero-shot cross-lingual transfer

Abstract: We introduce MULTI-EURLEX, a new multilingual dataset for topic classification of legal documents. The dataset comprises 65k European Union (EU) laws, officially translated in 23 languages, annotated with multiple labels from the EUROVOC taxonomy. We highlight the effect of temporal concept drift and the importance of chronological, instead of random splits. We use the dataset as a testbed for zero-shot cross-lingual transfer, where we exploit annotated training documents in one language (source) to classify documents in another language (target). We find that fine-tuning a multilingually pretrained model (XLM-ROBERTA, MT5) in a single source language leads to catastrophic forgetting of multilingual knowledge and, consequently, poor zero-shot transfer to other languages. Adaptation strategies, namely partial fine-tuning, adapters, BITFIT, LNFIT, originally proposed to accelerate fine-tuning for new end-tasks, help retain multilingual knowledge from pretraining, substantially improving zero-shot cross-lingual transfer, but their impact also depends on the pretrained model used and the size of the label set.

Paragraph-level Rationale Extraction through Regularization: A case study on European Court of Human Rights Cases

Abstract: Interpretability or explainability is an emerging research field in NLP. From a user-centric point of view, the goal is to build models that provide proper justification for their decisions, similar to those of humans, by requiring the models to satisfy additional constraints. To this end, we introduce a new application on legal text where, contrary to mainstream literature targeting word-level rationales, we conceive rationales as selected paragraphs in multi-paragraph structured court cases. We also release a new dataset comprising European Court of Human Rights cases, including annotations for paragraph-level rationales. We use this dataset to study the effect of already proposed rationale constraints, i.e., sparsity, continuity, and comprehensiveness, formulated as regularizers. Our findings indicate that some of these constraints are not beneficial in paragraph-level rationale extraction, while others need re-formulation to better handle the multi-label nature of the task we consider. We also introduce a new constraint, singularity, which further improves the quality of rationales, even compared with noisy rationale supervision. Experimental results indicate that the newly introduced task is very challenging and there is a large scope for further research.

Main characteristics of the examined datasets. We report the application domain, the number of documents across training/validation/test splits, the size of the label set, and the average number of labels per document.

The datasets and models we use are presented in Table {{table:56ffb323-d9ee-43e0-b81e-d07ba534a595}} . For MIMIC and ECtHR, which contain long documents (1-4K tokens), we convert the vanilla Transformer-based language models into Longformer models {{cite:5efba9c}}, and encode up to 2,048 tokens. For both phase one and phase two model training, we use a learning rate of 3e-5, a batch size of 32, and train models for a maximum of 100 epochs with early stopping after 5 epochs of no improvement in the macro-F1 score on the development set. Based on hyperparameter tuning conducted on a 10K sample of the BIOASQ and EURLEX datasets (see Appendix  for more details), we use cross-attention of two layers, with two heads each, and set {{formula:cd0a41e6-13ff-450d-a144-bb7264e5b5e5}} .

The datasets included in this study, for the most part, contain large training sets (see Table {{table:56ffb323-d9ee-43e0-b81e-d07ba534a595}} ), which need not always be the case in real-world scenarios. Low data availability exacerbates the issue of label skewness in classification tasks, so we expect to see higher gains from retrieval augmentation in lower-resource settings. To test this, we train models on samples of the EURLEX dataset of size 5K, 10K, 20K and compare those to the models trained on the full 55K training set. Results, shown in Figure {{figure:2f46f977-294a-4d71-918a-e1d8ab0658e5}} , support the hypothesis, with gains from retrieval augmentation starting off high at 5K samples and steadily diminishing as the amount of training data grows.


Corpus Wide Argument Mining - a Working Solution

Abstract: One of the main tasks in argument mining is the retrieval of argumentative content pertaining to a given topic. Most previous work addressed this task by retrieving a relatively small number of relevant documents as the initial source for such content. This line of research yielded moderate success, which is of limited use in a real-world system. Furthermore, for such a system to yield a comprehensive set of relevant arguments, over a wide range of topics, it requires leveraging a large and diverse corpus in an appropriate manner. Here we present a first end-to-end high-precision, corpus-wide argument mining system. This is made possible by combining sentence-level queries over an appropriate indexing of a very large corpus of newspaper articles, with an iterative annotation scheme. This scheme addresses the inherent label bias in the data and pinpoints the regions of the sample space whose manual labeling is required to obtain high-precision among top-ranked candidates.

A Large-scale Dataset for Argument Quality Ranking: Construction and Analysis

Abstract: Identifying the quality of free-text arguments has become an important task in the rapidly expanding field of computational argumentation. In this work, we explore the challenging task of argument quality ranking. To this end, we created a corpus of 30,497 arguments carefully annotated for point-wise quality, released as part of this work. To the best of our knowledge, this is the largest dataset annotated for point-wise argument quality, larger by a factor of five than previously released datasets. Moreover, we address the core issue of inducing a labeled score from crowd annotations by performing a comprehensive evaluation of different approaches to this problem. In addition, we analyze the quality dimensions that characterize this dataset. Finally, we present a neural method for argument quality ranking, which outperforms several baselines on our own dataset, as well as previous methods published for another dataset.

Fast End-to-End Wikification

Abstract: Wikification of large corpora is beneficial for various NLP applications. Existing methods focus on quality performance rather than run-time, and are therefore non-feasible for large data. Here, we introduce RedW, a run-time oriented Wikification solution, based on Wikipedia redirects, that can Wikify massive corpora with competitive performance. We further propose an efficient method for estimating RedW confidence, opening the door for applying more demanding methods only on top of RedW lower-confidence results. Our experimental results support the validity of the proposed approach.

From Arguments to Key Points: Towards Automatic Argument Summarization

Abstract: Generating a concise summary from a large collection of arguments on a given topic is an intriguing yet understudied problem. We propose to represent such summaries as a small set of talking points, termed key points, each scored according to its salience. We show, by analyzing a large dataset of crowd-contributed arguments, that a small number of key points per topic is typically sufficient for covering the vast majority of the arguments. Furthermore, we found that a domain expert can often predict these key points in advance. We study the task of argument-to-key point mapping, and introduce a novel large-scale dataset for this task. We report empirical results for an extensive set of experiments with this dataset, showing promising performance.

Project Debater APIs assessment

Table {{table:b6340ac2-2ab7-47a5-b0a2-33dfb4d17539}}  includes assessment results for various services. For each service, we specify the benchmark that was used for testing, the evaluation measure(s) and the results. If the results in the table are quoted from one of our papers, this is indicated by ✓. Unless otherwise mentioned, the results are from the same paper that is cited for the dataset. In cases where the results for the service were not available (this happens, for example, if the current service implementation is different from the one described in the paper), we ran the service on the benchmark and reported the results.


Tree Diversity and Dynamics of the Forest of Seu Nico, Viçosa, Minas Gerais, Brazil

Abstract: Abstract Background To understand future changes in community composition due to global changes, the knowledge about forest community dynamics is of crucial importance. To improve our understanding about processes and patterns involved in maintaining species rich Neotropical ecosystems, we provide here a dataset from the one hectare Forest of Seu Nico (FSN) Dynamics Plot from Southeastern Brazil. New information We report diameter at breast height, basal area and height measurements of 2868 trees and treelets identified from two census spanning over a nine-year period. Furthermore, soil properties and understory light availability of all 100 10 x 10m subplots from the one hectare FSN Dynamics Plot during the second census are given.

Dataset of MIGRAME Project (Global Change, Altitudinal Range Shift and Colonization of Degraded Habitats in Mediterranean Mountains)

Abstract: Abstract In this data paper, we describe the dataset of the Global Change, Altitudinal Range Shift and Colonization of Degraded Habitats in Mediterranean Mountains (MIGRAME) project, which aims to assess the capacity of altitudinal migration and colonization of marginal habitats by Quercus pyrenaica Willd. forests in Sierra Nevada (southern Spain) considering two global-change drivers: temperature increase and land-use changes. The dataset includes information of the forest structure (diameter size, tree height, and abundance) of the Quercus pyrenaica ecosystem in Sierra Nevada obtained from 199 transects sampled at the treeline ecotone, mature forest, and marginal habitats (abandoned cropland and pine plantations). A total of 3839 occurrence records were collected and 5751 measurements recorded. The dataset is included in the Sierra Nevada Global-Change Observatory (OBSNEV), a long-term research project designed to compile socio-ecological information on the major ecosystem types in order to identify the impacts of global change in this mountain range.

Structure and tree species composition in different habitats of savanna used by indigenous people in the Northern Brazilian Amazon

Abstract: Abstract Background Woody plant diversity from the Amazonian savannas has been poorly quantified. In order to improve the knowledge on wood plants of these regional ecosystems, a tree inventory was carried out in four different habitats used by indigenous people living in the savanna areas of the Northern Brazilian Amazon. The habitats were divided into two types (or groups) of vegetation formations: forest (riparian forest, forest island, and buritizal = Mauritia palm formation) and non-forest (typical savanna). The inventory was carried out in two hectares established in the Darora Indigenous Community region, north of the state of Roraima. New information The typical savanna is the most densely populated area (709 stems ha-1); however, it has the lowest tree species richness (nine species, seven families) in relation to typical forest habitats: riparian forest (22 species, 13 families and 202 stems ha-1), forest islands (13 species, 10 families and 264 stems ha-1), and buritizal (19 species, 15 families and 600 stems ha-1). The tree structure (density and dominance) of the forest habitats located in the savanna areas studied in this work is smaller in relation to forest habitats derived from continuous areas of other parts of the Amazon. These environments are derived from Paleoclimatic fragmentation, and are currently affected by the impact of intensive use of natural resources as timberselective logging and some land conversion for agriculture.

Review of open access forest inventories datasets

Inventory datasets are summarized in Table {{table:57e20eb3-2be8-4f30-ac67-a1c39a461445}} , the size of the datasets is quantified by the number of trees.
Inventory datasets are composed of various measurements. They commonly contain tree height, canopy diameter, diameter at breast height (DBH) or diameter at soil height (DSH) {{cite:c9a679e}}, {{cite:eadaeaf}}, {{cite:c55f43a}}, {{cite:7593dc4}}, {{cite:fc5adce}}, {{cite:66cf436}}. In specific cases, wood density, bark density and bark thickness are also measured {{cite:92b1038}}, {{cite:3254f87}}, {{cite:cf8bae4}}. These information are particularly useful to estimate the tree density, the above ground biomass (AGB) or the tree carbon stock at large scale {{cite:f17052e}} even if the inventories have not been released with the estimated maps {{cite:8f5a2b4}}, {{cite:a3c31e6}}.

{{cite:eadaeaf}}


Can GANs model any complex distributions?

What can we learn about the GAN's latent space?

What can we learn about the GAN's latent space?

What can we do by moving around in the latent space?

What can we do by moving around in the latent space?

What can we do by moving around in the latent space?

What can we do by moving around in the latent space?

What can we do by moving around in the latent space?

No, cannot model/generate heavy-tailed distributions.

Impose regularizing constraints during unpaired image translation

Photo-realistic multiple attribute transformation with identity preservation

Open-ended primitive visual concept vocabulary

On some theoretical limitations of Generative Adversarial Networks

Abstract: Generative Adversarial Networks have become a core technique in Machine Learning to generate unknown distributions from data samples. They have been used in a wide range of context without paying much attention to the possible theoretical limitations of those models. Indeed, because of the universal approximation properties of Neural Networks, it is a general assumption that GANs can generate any probability distribution. Recently, people began to question this assumption and this article is in line with this thinking. We provide a new result based on Extreme Value Theory showing that GANs can't generate heavy tailed distributions. The full proof of this result is given.

Abstract: Despite the successes in capturing continuous distributions, the application of generative adversarial networks (GANs) to discrete settings, like natural language tasks, is rather restricted. The fundamental reason is the difficulty of back-propagation through discrete random variables combined with the inherent instability of the GAN training objective. To address these problems, we propose Maximum-Likelihood Augmented Discrete Generative Adversarial Networks. Instead of directly optimizing the GAN objective, we derive a novel and low-variance objective using the discriminator's output that follows corresponds to the log-likelihood. Compared with the original, the new objective is proved to be consistent in theory and beneficial in practice. The experimental results on various discrete datasets demonstrate the effectiveness of the proposed approach.

Latent Space Oddity: on the Curvature of Deep Generative Models

Abstract: Deep generative models provide a systematic way to learn nonlinear data distributions, through a set of latent variables and a nonlinear "generator" function that maps latent points into the input space. The nonlinearity of the generator imply that the latent space gives a distorted view of the input space. Under mild conditions, we show that this distortion can be characterized by a stochastic Riemannian metric, and demonstrate that distances and interpolants are significantly improved under this metric. This in turn improves probability distributions, sampling algorithms and clustering in the latent space. Our geometric analysis further reveals that current generators provide poor variance estimates and we propose a new generator architecture with vastly improved variance estimates. Results are demonstrated on convolutional and fully connected variational autoencoders, but the formalism easily generalize to other deep generative models.

Interpreting the Latent Space of GANs for Semantic Face Editing

Abstract: Despite the recent advance of Generative Adversarial Networks (GANs) in high-fidelity image synthesis, there lacks enough understanding of how GANs are able to map a latent code sampled from a random distribution to a photo-realistic image. Previous work assumes the latent space learned by GANs follows a distributed representation but observes the vector arithmetic phenomenon. In this work, we propose a novel framework, called InterFaceGAN, for semantic face editing by interpreting the latent semantics learned by GANs. In this framework, we conduct a detailed study on how different semantics are encoded in the latent space of GANs for face synthesis. We find that the latent code of well-trained generative models actually learns a disentangled representation after linear transformations. We explore the disentanglement between various semantics and manage to decouple some entangled semantics with subspace projection, leading to more precise control of facial attributes. Besides manipulating gender, age, expression, and the presence of eyeglasses, we can even vary the face pose as well as fix the artifacts accidentally generated by GAN models. The proposed method is further applied to achieve real image manipulation when combined with GAN inversion methods or some encoder-involved models. Extensive results suggest that learning to synthesize faces spontaneously brings a disentangled and controllable facial attribute representation.

Unsupervised Discovery of Interpretable Directions in the GAN Latent Space

Abstract: The latent spaces of GAN models often have semantically meaningful directions. Moving in these directions corresponds to human-interpretable image transformations, such as zooming or recoloring, enabling a more controllable generation process. However, the discovery of such directions is currently performed in a supervised manner, requiring human labels, pretrained models, or some form of self-supervision. These requirements severely restrict a range of directions existing approaches can discover. In this paper, we introduce an unsupervised method to identify interpretable directions in the latent space of a pretrained GAN model. By a simple model-agnostic procedure, we find directions corresponding to sensible semantic manipulations without any form of (self-)supervision. Furthermore, we reveal several non-trivial findings, which would be difficult to obtain by existing methods, e.g., a direction corresponding to background removal. As an immediate practical benefit of our work, we show how to exploit this finding to achieve competitive performance for weakly-supervised saliency detection.

Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling

Abstract: We study the problem of 3D object generation. We propose a novel framework, namely 3D Generative Adversarial Network (3D-GAN), which generates 3D objects from a probabilistic space by leveraging recent advances in volumetric convolutional networks and generative adversarial nets. The benefits of our model are three-fold: first, the use of an adversarial criterion, instead of traditional heuristic criteria, enables the generator to capture object structure implicitly and to synthesize high-quality 3D objects; second, the generator establishes a mapping from a low-dimensional probabilistic space to the space of 3D objects, so that we can sample objects without a reference image or CAD models, and explore the 3D object manifold; third, the adversarial discriminator provides a powerful 3D shape descriptor which, learned without supervision, has wide applications in 3D object recognition. Experiments demonstrate that our method generates high-quality 3D objects, and our unsupervisedly learned features achieve impressive performance on 3D object recognition, comparable with those of supervised learning methods.

Score-based Generative Modeling in Latent Space

Abstract: Score-based generative models (SGMs) have recently demonstrated impressive results in terms of both sample quality and distribution coverage. However, they are usually applied directly in data space and often require thousands of network evaluations for sampling. Here, we propose the Latent Score-based Generative Model (LSGM), a novel approach that trains SGMs in a latent space, relying on the variational autoencoder framework. Moving from data to latent space allows us to train more expressive generative models, apply SGMs to non-continuous data, and learn smoother SGMs in a smaller space, resulting in fewer network evaluations and faster sampling. To enable training LSGMs end-to-end in a scalable and stable manner, we (i) introduce a new score-matching objective suitable to the LSGM setting, (ii) propose a novel parameterization of the score function that allows SGM to focus on the mismatch of the target distribution with respect to a simple Normal one, and (iii) analytically derive multiple techniques for variance reduction of the training objective. LSGM obtains a state-of-the-art FID score of 2.10 on CIFAR-10, outperforming all existing generative results on this dataset. On CelebA-HQ-256, LSGM is on a par with previous SGMs in sample quality while outperforming them in sampling time by two orders of magnitude. In modeling binary images, LSGM achieves state-of-the-art likelihood on the binarized OMNIGLOT dataset. Our project page and code can be found at https://nvlabs.github.io/LSGM .

Stabilizing Generative Adversarial Networks: A Survey

Abstract: Generative Adversarial Networks (GANs) are a type of generative model which have received much attention due to their ability to model complex real-world data. Despite their recent successes, the process of training GANs remains challenging, suffering from instability problems such as non-convergence, vanishing or exploding gradients, and mode collapse. In recent years, a diverse set of approaches have been proposed which focus on stabilizing the GAN training procedure. The purpose of this survey is to provide a comprehensive overview of the GAN training stabilization methods which can be found in the literature. We discuss the advantages and disadvantages of each approach, offer a comparative summary, and conclude with a discussion of open problems.

Image Synthesis with Adversarial Networks: a Comprehensive Survey and Case Studies

Abstract: Generative Adversarial Networks (GANs) have been extremely successful in various application domains such as computer vision, medicine, and natural language processing. Moreover, transforming an object or person to a desired shape become a well-studied research in the GANs. GANs are powerful models for learning complex distributions to synthesize semantically meaningful samples. However, there is a lack of comprehensive review in this field, especially lack of a collection of GANs loss-variant, evaluation metrics, remedies for diverse image generation, and stable training. Given the current fast GANs development, in this survey, we provide a comprehensive review of adversarial models for image synthesis. We summarize the synthetic image generation methods, and discuss the categories including image-to-image translation, fusion image generation, label-to-image mapping, and text-to-image translation. We organize the literature based on their base models, developed ideas related to architectures, constraints, loss functions, evaluation metrics, and training datasets. We present milestones of adversarial models, review an extensive selection of previous works in various categories, and present insights on the development route from the model-based to data-driven methods. Further, we highlight a range of potential future research directions. One of the unique features of this review is that all software implementations of these GAN methods and datasets have been collected and made available in one place at https://github.com/pshams55/GAN-Case-Study.

Rayleigh EigenDirections (REDs): GAN latent space traversals for multidimensional features

Abstract: We present a method for finding paths in a deep generative model's latent space that can maximally vary one set of image features while holding others constant. Crucially, unlike past traversal approaches, ours can manipulate multidimensional features of an image such as facial identity and pixels within a specified region. Our method is principled and conceptually simple: optimal traversal directions are chosen by maximizing differential changes to one feature set such that changes to another set are negligible. We show that this problem is nearly equivalent to one of Rayleigh quotient maximization, and provide a closed-form solution to it based on solving a generalized eigenvalue equation. We use repeated computations of the corresponding optimal directions, which we call Rayleigh EigenDirections (REDs), to generate appropriately curved paths in latent space. We empirically evaluate our method using StyleGAN2 on two image domains: faces and living rooms. We show that our method is capable of controlling various multidimensional features out of the scope of previous latent space traversal methods: face identity, spatial frequency bands, pixels within a region, and the appearance and position of an object. Our work suggests that a wealth of opportunities lies in the local analysis of the geometry and semantics of latent spaces.

Optimizing Latent Space Directions For GAN-based Local Image Editing

Abstract: Generative Adversarial Network (GAN) based localized image editing can suffer from ambiguity between semantic attributes. We thus present a novel objective function to evaluate the locality of an image edit. By introducing the supervision from a pre-trained segmentation network and optimizing the objective function, our framework, called Locally Effective Latent Space Direction (LELSD), is applicable to any dataset and GAN architecture. Our method is also computationally fast and exhibits a high extent of disentanglement, which allows users to interactively perform a sequence of edits on an image. Our experiments on both GAN-generated and real images qualitatively demonstrate the high quality and advantages of our method.

Enjoy Your Editing: Controllable GANs for Image Editing via Latent Space Navigation

Abstract: Controllable semantic image editing enables a user to change entire image attributes with a few clicks, e.g., gradually making a summer scene look like it was taken in winter. Classic approaches for this task use a Generative Adversarial Net (GAN) to learn a latent space and suitable latent-space transformations. However, current approaches often suffer from attribute edits that are entangled, global image identity changes, and diminished photo-realism. To address these concerns, we learn multiple attribute transformations simultaneously, integrate attribute regression into the training of transformation functions, and apply a content loss and an adversarial loss that encourages the maintenance of image identity and photo-realism. We propose quantitative evaluation strategies for measuring controllable editing performance, unlike prior work, which primarily focuses on qualitative evaluation. Our model permits better control for both single- and multiple-attribute editing while preserving image identity and realism during transformation. We provide empirical results for both natural and synthetic images, highlighting that our model achieves state-of-the-art performance for targeted image manipulation.

Toward a Visual Concept Vocabulary for GAN Latent Space

Abstract: A large body of recent work has identified transformations in the latent spaces of generative adversarial networks (GANs) that consistently and interpretably transform generated images. But existing techniques for identifying these transformations rely on either a fixed vocabulary of prespecified visual concepts, or on unsupervised disentanglement techniques whose alignment with human judgments about perceptual salience is unknown. This paper introduces a new method for building open-ended vocabularies of primitive visual concepts represented in a GAN’s latent space. Our approach is built from three components: (1) automatic identification of perceptually salient directions based on their layer selectivity; (2) human annotation of these directions with free-form, compositional natural language descriptions; and (3) decomposition of these annotations into a visual concept vocabulary, consisting of distilled directions labeled with single words. Experiments show that concepts learned with our approach are reliable and composable—generalizing across classes, contexts, and observers, and enabling fine-grained manipulation of image style and content.

Investigating Cosmological GAN Emulators Using Latent Space Interpolation

Abstract: Generative adversarial networks (GANs) have been recently applied as a novel emulation technique for large scale structure simulations. Recent results show that GANs can be used as a fast, efficient and computationally cheap emulator for producing novel weak lensing convergence maps as well as cosmic web data in 2-D and 3-D. However, like any algorithm, the GAN approach comes with a set of limitations, such as an unstable training procedure and the inherent randomness of the produced outputs. In this work we employ a number of techniques commonly used in the machine learning literature to address the mentioned limitations. In particular, we train a GAN to produce both weak lensing convergence maps and dark matter overdensity field data for multiple redshifts, cosmological parameters and modified gravity models. In addition, we train a GAN using the newest Illustris data to emulate dark matter, gas and internal energy distribution data simultaneously. Finally, we apply the technique of latent space interpolation to control which outputs the algorithm produces. Our results indicate a 1-20% difference between the power spectra of the GAN-produced and the training data samples depending on the dataset used and whether Gaussian smoothing was applied. Finally, recent research on generative models suggests that such algorithms can be treated as mappings from a lower-dimensional input (latent) space to a higher dimensional (data) manifold. We explore such a theoretical description as a tool for better understanding the latent space interpolation procedure.

3DMaterialGAN: Learning 3D Shape Representation from Latent Space for Materials Science Applications

Abstract: In the field of computer vision, unsupervised learning for 2D object generation has advanced rapidly in the past few years. However, 3D object generation has not garnered the same attention or success as its predecessor. To facilitate novel progress at the intersection of computer vision and materials science, we propose a 3DMaterialGAN network that is capable of recognizing and synthesizing individual grains whose morphology conforms to a given 3D polycrystalline material microstructure. This Generative Adversarial Network (GAN) architecture yields complex 3D objects from probabilistic latent space vectors with no additional information from 2D rendered images. We show that this method performs comparably or better than state-of-the-art on benchmark annotated 3D datasets, while also being able to distinguish and generate objects that are not easily annotated, such as grain morphologies. The value of our algorithm is demonstrated with analysis on experimental real-world data, namely generating 3D grain structures found in a commercially relevant wrought titanium alloy, which were validated through statistical shape comparison. This framework lays the foundation for the recognition and synthesis of polycrystalline material microstructures, which are used in additive manufacturing, aerospace, and structural design applications.

Latent-Space Laplacian Pyramids for Adversarial Representation Learning with 3D Point Clouds

Abstract: Constructing high-quality generative models for 3D shapes is a fundamental task in computer vision with diverse applications in geometry processing, engineering, and design. Despite the recent progress in deep generative modelling, synthesis of finely detailed 3D surfaces, such as high-resolution point clouds, from scratch has not been achieved with existing approaches. In this work, we propose to employ the latent-space Laplacian pyramid representation within a hierarchical generative model for 3D point clouds. We combine the recently proposed latent-space GAN and Laplacian GAN architectures to form a multi-scale model capable of generating 3D point clouds at increasing levels of detail. Our evaluation demonstrates that our model outperforms the existing generative models for 3D point clouds.

Questions and Answers about GANs

With the ability to explore the latent space to edit images, researchers have begun to learn disentangled representations of the latent space, making it a significant research problem in current computer vision research. Now that we have an understanding of what a GAN is, we will shift our focus to the latent space and the nature of distributions modeled by a GAN by answering the following questions summarized in the table {{table:614bc608-b065-4dc2-b99d-bb0ec6ea9a28}} :


Learning Audio-Visual Correlations From Variational Cross-Modal Generation

Abstract: People can easily imagine the potential sound while seeing an event. This natural synchronization between audio and visual signals reveals their intrinsic correlations. To this end, we propose to learn the audio-visual correlations from the perspective of cross-modal generation in a self-supervised manner, the learned correlations can be then readily applied in multiple downstream tasks such as the audio-visual cross-modal localization and retrieval. We introduce a novel Variational AutoEncoder (VAE) framework that consists of Multiple encoders and a Shared decoder (MS-VAE) with an additional Wasserstein distance constraint to tackle the problem. Extensive experiments demonstrate that the optimized latent representation of the proposed MS-VAE can effectively learn the audio-visual correlations and can be readily applied in multiple audio-visual downstream tasks to achieve competitive performance even without any given label information during training.

End-to-End Video-to-Speech Synthesis Using Generative Adversarial Networks

Abstract: Video-to-speech is the process of reconstructing the audio speech from a video of a spoken utterance. Previous approaches to this task have relied on a two-step process where an intermediate representation is inferred from the video and is then decoded into waveform audio using a vocoder or a waveform reconstruction algorithm. In this work, we propose a new end-to-end video-to-speech model based on generative adversarial networks (GANs) which translates spoken video to waveform end-to-end without using any intermediate representation or separate waveform synthesis algorithm. Our model consists of an encoder–decoder architecture that receives raw video as input and generates speech, which is then fed to a waveform critic and a power critic. The use of an adversarial loss based on these two critics enables the direct synthesis of the raw audio waveform and ensures its realism. In addition, the use of our three comparative losses helps establish direct correspondence between the generated audio and the input video. We show that this model is able to reconstruct speech with remarkable realism for constrained datasets such as GRID, and that it is the first end-to-end model to produce intelligible speech for Lip Reading in the Wild (LRW), featuring hundreds of speakers recorded entirely “in the wild.” We evaluate the generated samples in two different scenarios—seen and unseen speakers—using four objective metrics which measure the quality and intelligibility of artificial speech. We demonstrate that the proposed approach outperforms all previous works in most metrics on GRID and LRW.

Robust Latent Representations Via Cross-Modal Translation and Alignment

Abstract: Multi-modal learning relates information across observation modalities of the same physical phenomenon to leverage complementary information. Most multi-modal machine learning methods require that all the modalities used for training are also available for testing. This is a limitation when signals from some modalities are unavailable or severely degraded. To address this limitation, we aim to improve the testing performance of uni-modal systems using multiple modalities during training only. The proposed multi-modal training framework uses cross-modal translation and correlation-based latent space alignment to improve the representations of a worse performing (or weaker) modality. The translation from the weaker to the better performing (or stronger) modality generates a multi-modal intermediate encoding that is representative of both modalities. This encoding is then correlated with the stronger modality representation in a shared latent space. We validate the proposed framework on the AVEC 2016 dataset (RECOLA) for continuous emotion recognition and show the effectiveness of the framework that achieves state-of- the-art (uni-modal) performance for weaker modalities.

Deep Fusion: An Attention Guided Factorized Bilinear Pooling for Audio-video Emotion Recognition

Abstract: Automatic emotion recognition (AER) is a challenging task due to the abstract concept and multiple expressions of emotion. Although there is no consensus on a definition, human emotional states usually can be apperceived by auditory and visual systems. Inspired by this cognitive process in human beings, it’s natural to simultaneously utilize audio and visual information in AER. However, most traditional fusion approaches only build a linear paradigm, such as feature concatenation and multi-system fusion, which hardly captures complex association between audio and video. In this paper, we introduce factorized bilinear pooling (FBP) to deeply integrate the features of audio and video. Specifically, the features are selected through the embedded attention mechanism from respective modalities to obtain the emotion-related regions. The whole pipeline can be completed in a neural network. Validated on the AFEW database of the audio-video sub-challenge in EmotiW2018, the proposed approach achieves an accuracy of 62.48%, outperforming the state-of-the-art result.

Broaden Your Views for Self-Supervised Video Learning

Abstract: Most successful self-supervised learning methods are trained to align the representations of two independent views from the data. State-of-the-art methods in video are inspired by image techniques, where these two views are similarly extracted by cropping and augmenting the resulting crop. However, these methods miss a crucial element in the video domain: time. We introduce BraVe, a self-supervised learning framework for video. In BraVe, one of the views has access to a narrow temporal window of the video while the other view has a broad access to the video content. Our models learn to generalise from the narrow view to the general content of the video. Furthermore, BraVe processes the views with different backbones, enabling the use of alternative augmentations or modalities into the broad view such as optical flow, randomly convolved RGB frames, audio or their combinations. We demonstrate that BraVe achieves state-of-the-art results in self-supervised representation learning on standard video and audio classification benchmarks including UCF101, HMDB51, Kinetics, ESC-50 and AudioSet.

Cross-Modal Attention Consistency for Video-Audio Unsupervised Learning

Abstract: Cross-modal correlation provides an inherent supervision for video unsupervised representation learning. Existing methods focus on distinguishing different video clips by visual and audio representations. We human visual perception could attend to regions where sounds are made, and our auditory perception could also ground their frequencies of sounding objects, which we call bidirectional local correspondence. Such supervision is intuitive but not well explored in the contrastive learning framework. This paper introduces a pretext task, Cross-Modal Attention Consistency (CMAC), for exploring the bidirectional local correspondence property. The CMAC approach aims to align the regional attention generated purely from the visual signal with the target attention generated under the guidance of acoustic signal, and do a similar alignment for frequency grounding on the acoustic attention. Accompanied by a remoulded cross-modal contrastive loss where we consider additional within-modal interactions, the CMAC approach works effectively for enforcing the bidirectional alignment. Extensive experiments on six downstream benchmarks demonstrate that CMAC can improve the state-of-the-art performance on both visual and audio modalities.

VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text

Abstract: We present a framework for learning multimodal representations from unlabeled data using convolution-free Transformer architectures. Specifically, our Video-Audio-Text Transformer (VATT) takes raw signals as inputs and extracts multimodal representations that are rich enough to benefit a variety of downstream tasks. We train VATT end-to-end from scratch using multimodal contrastive losses and evaluate its performance by the downstream tasks of video action recognition, audio event classification, image classification, and text-to-video retrieval. Furthermore, we study a modality-agnostic, single-backbone Transformer by sharing weights among the three modalities. We show that the convolution-free VATT outperforms state-of-the-art ConvNet-based architectures in the downstream tasks. Especially, VATT's vision Transformer achieves the top-1 accuracy of 82.1% on Kinetics-400, 83.6% on Kinetics-600, 72.7% on Kinetics-700, and 41.1% on Moments in Time, new records while avoiding supervised pre-training. Transferring to image classification leads to 78.7% top-1 accuracy on ImageNet compared to 64.7% by training the same Transformer from scratch, showing the generalizability of our model despite the domain gap between videos and images. VATT's audio Transformer also sets a new record on waveform-based audio event recognition by achieving the mAP of 39.4% on AudioSet without any supervised pre-training. VATT's source code is publicly available.

Look, Listen, and Attend: Co-Attention Network for Self-Supervised Audio-Visual Representation Learning

Abstract: When watching videos, the occurrence of a visual event is often accompanied by an audio event, e.g., the voice of lip motion, the music of playing instruments. There is an underlying correlation between audio and visual events, which can be utilized as free supervised information to train a neural network by solving the pretext task of audio-visual synchronization. In this paper, we propose a novel self-supervised framework with co-attention mechanism to learn generic cross-modal representations from unlabelled videos in the wild, and further benefit downstream tasks. Specifically, we explore three different co-attention modules to focus on discriminative visual regions correlated to the sounds and introduce the interactions between them. Experiments show that our model achieves state-of-the-art performance on the pretext task while having fewer parameters compared with existing methods. To further evaluate the generalizability and transferability of our approach, we apply the pre-trained model on two downstream tasks, i.e., sound source localization and action recognition. Extensive experiments demonstrate that our model provides competitive results with other self-supervised methods, and also indicate that our approach can tackle the challenging scenes which contain multiple sound sources.

AVLnet: Learning Audio-Visual Language Representations from Instructional Videos

Abstract: Current methods for learning visually grounded language from videos often rely on time-consuming and expensive data collection, such as human annotated textual summaries or machine generated automatic speech recognition transcripts. In this work, we introduce Audio-Video Language Network (AVLnet), a self-supervised network that learns a shared audio-visual embedding space directly from raw video inputs. We circumvent the need for annotation and instead learn audio-visual language representations directly from randomly segmented video clips and their raw audio waveforms. We train AVLnet on publicly available instructional videos and evaluate our model on video clip and language retrieval tasks on three video datasets. Our proposed model outperforms several state-of-the-art text-video baselines by up to 11.8% in a video clip retrieval task, despite operating on the raw audio instead of manually annotated text captions. Further, we show AVLnet is capable of integrating textual information, increasing its modularity and improving performance by up to 20.3% on the video clip retrieval task. Finally, we perform analysis of AVLnet's learned representations, showing our model has learned to relate visual objects with salient words and natural sounds.

Self-Supervised MultiModal Versatile Networks

Abstract: Videos are a rich source of multi-modal supervision. In this work, we learn representations using self-supervision by leveraging three modalities naturally present in videos: vision, audio and language. To this end, we introduce the notion of a multimodal versatile network -- a network that can ingest multiple modalities and whose representations enable downstream tasks in multiple modalities. In particular, we explore how best to combine the modalities, such that fine-grained representations of audio and vision can be maintained, whilst also integrating text into a common embedding. Driven by versatility, we also introduce a novel process of deflation, so that the networks can be effortlessly applied to the visual data in the form of video or a static image. We demonstrate how such networks trained on large collections of unlabelled video data can be applied on video, video-text, image and audio tasks. Equipped with these representations, we obtain state-of-the-art performance on multiple challenging benchmarks including UCF101, HMDB51 and ESC-50 when compared to previous self-supervised work.

Self-supervised object detection from audio-visual correspondence

Abstract: We tackle the problem of learning object detectors without supervision. Differently from weakly-supervised object detection, we do not assume image-level class labels. Instead, we extract a supervisory signal from audio-visual data, using the audio component to “teach” the object detector. While this problem is related to sound source localisation, it is considerably harder because the detector must classify the objects by type, enumerate each instance of the object, and do so even when the object is silent. We tackle this problem by first designing a self-supervised framework with a contrastive objective that jointly learns to classify and localise objects. Then, without using any supervision, we simply use these self-supervised labels and boxes to train an image-based object detector. With this, we outperform previous unsupervised and weakly-supervised detectors for the task of object detection and sound source localization. We also show that we can align this detector to ground-truth classes with as little as one label per pseudo-class, and show how our method can learn to detect generic objects that go beyond instruments, such as airplanes and cats.

Cross-Modal Music-Video Recommendation: A Study of Design Choices

Abstract: In this work, we study music/video cross-modal recommendation, i.e. recommending a music track for a video or vice versa. We rely on a self-supervised learning paradigm to learn from a large amount of unlabelled data. We rely on a self-supervised learning paradigm to learn from a large amount of unlabelled data. More precisely, we jointly learn audio and video embeddings by using their co-occurrence in music-video clips. In this work, we build upon a recent video-music retrieval system (the VM-NET), which originally relies on an audio representation obtained by a set of statistics computed over handcrafted features. We demonstrate here that using audio representation learning such as the audio embeddings provided by the pre-trained MuSimNet, OpenL3, MusicCNN or by AudioSet, largely improves recommendations. We also validate the use of the cross-modal triplet loss originally proposed in the VM-NET compared to the binary cross-entropy loss commonly used in self-supervised learning. We perform all our experiments using the Music Video Dataset (MVD).

Deep Triplet Neural Networks with Cluster-CCA for Audio-Visual Cross-Modal Retrieval

Abstract: Cross-modal retrieval aims to retrieve data in one modality by a query in another modality, which has been a very interesting research issue in the field of multimedia, information retrieval, and computer vision, and database. Most existing works focus on cross-modal retrieval between text-image, text-video, and lyrics-audio. Little research addresses cross-modal retrieval between audio and video due to limited audio-video paired datasets and semantic information. The main challenge of the audio-visual cross-modal retrieval task focuses on learning joint embeddings from a shared subspace for computing the similarity across different modalities, where generating new representations is to maximize the correlation between audio and visual modalities space. In this work, we propose TNN-C-CCA, a novel deep triplet neural network with cluster canonical correlation analysis, which is an end-to-end supervised learning architecture with an audio branch and a video branch. We not only consider the matching pairs in the common space but also compute the mismatching pairs when maximizing the correlation. In particular, two significant contributions are made. First, a better representation by constructing a deep triplet neural network with triplet loss for optimal projections can be generated to maximize correlation in the shared subspace. Second, positive examples and negative examples are used in the learning stage to improve the capability of embedding learning between audio and video. Our experiment is run over fivefold cross validation, where average performance is applied to demonstrate the performance of audio-video cross-modal retrieval. The experimental results achieved on two different audio-visual datasets show that the proposed learning architecture with two branches outperforms existing six canonical correlation analysis–based methods and four state-of-the-art-based cross-modal retrieval methods.

Language learning using Speech to Image retrieval

Abstract: Humans learn language by interaction with their environment and listening to other humans. It should also be possible for computational models to learn language directly from speech but so far most approaches require text. We improve on existing neural network approaches to create visually grounded embeddings for spoken utterances. Using a combination of a multi-layer GRU, importance sampling, cyclic learning rates, ensembling and vectorial self-attention our results show a remarkable increase in image-caption retrieval performance over previous work. Furthermore, we investigate which layers in the model learn to recognise words in the input. We find that deeper network layers are better at encoding word presence, although the final layer has slightly lower performance. This shows that our visually grounded sentence encoder learns to recognise words from the input even though it is not explicitly trained for word recognition.

Large-Scale Representation Learning from Visually Grounded Untranscribed Speech

Abstract: Systems that can associate images with their spoken audio captions are an important step towards visually grounded language learning. We describe a scalable method to automatically generate diverse audio for image captioning datasets. This supports pretraining deep networks for encoding both audio and images, which we do via a dual encoder that learns to align latent representations from both modalities. We show that a masked margin softmax loss for such models is superior to the standard triplet loss. We fine-tune these models on the Flickr8k Audio Captions Corpus and obtain state-of-the-art results—improving recall in the top 10 from 29.6% to 49.5%. We also obtain human ratings on retrieval outputs to better assess the impact of incidentally matching image-caption pairs that were not associated in the data, finding that automatic evaluation substantially underestimates the quality of the retrieved results.

Variational Autoencoder with CCA for Audio–Visual Cross-modal Retrieval

Abstract: Cross-modal retrieval is to utilize one modality as a query to retrieve data from another modality, which has become a popular topic in information retrieval, machine learning, and databases. Finding a method to effectively measure the similarity between different modality data is the major challenge of cross-modal retrieval. Although several research works have calculated the correlation between different modality data via learning a common subspace representation, the encoder’s ability to extract features from multi-modal information is not satisfactory. In this article, we present a novel variational autoencoder architecture for audio–visual cross-modal retrieval by learning paired audio–visual correlation embedding and category correlation embedding as constraints to reinforce the mutuality of audio–visual information. On the one hand, audio encoder and visual encoder separately encode audio data and visual data into two different latent spaces. Further, two mutual latent spaces are respectively constructed by canonical correlation analysis. On the other hand, probabilistic modeling methods are used to deal with possible noise and missing information in the data. Additionally, in this way, the cross-modal discrepancies from intra-modal and inter-modal information are simultaneously eliminated in the joint embedding subspace. We conduct extensive experiments over two benchmark datasets. The experimental results confirm that the proposed architecture is effective in learning audio–visual correlation and is appreciably better than the existing cross-modal retrieval methods.

Learning Explicit and Implicit Latent Common Spaces for Audio-Visual Cross-Modal Retrieval

Abstract: Learning common subspace is prevalent way in cross-modal retrieval to solve the problem of data from different modalities having inconsistent distributions and representations that cannot be directly compared. Previous cross-modal retrieval methods focus on projecting the cross-modal data into a common space by learning the correlation between them to bridge the modality gap. However, the rich semantic information in the video and the heterogeneous nature of audio-visual data leads to more serious heterogeneous gaps intuitively, which may lead to the loss of key semantic content of video with single clue by the previous methods when eliminating the modality gap, while the semantics of the categories may undermine the properties of the original features. In this work, we aim to learn effective audio-visual representations to support audio-visual cross-modal retrieval (AVCMR). We propose a novel model that maps audio-visual modalities into two distinct shared latent subspaces: explicit and implicit shared spaces. In particular, the explicit shared space is used to optimize pairwise correlations, where learned representations across modalities capture the commonalities of audio-visual pairs and reduce the modality gap. The implicit shared space is used to preserve the distinctive features between modalities by maintaining the discrimination of audio/video patterns from different semantic categories. Finally, the fusion of the features learned from the two latent subspaces is used for the similarity computation of the AVCMR task. The comprehensive experimental results on two audio-visual datasets demonstrate that our proposed model for using two different latent subspaces for audio-visual cross-modal learning is effective and significantly outperforms the state-of-the-art cross-modal models that learn features from a single subspace.

Localizing Visual Sounds the Hard Way

Abstract: The objective of this work is to localize sound sources that are visible in a video without using manual annotations. Our key technical contribution is to show that, by training the network to explicitly discriminate challenging image fragments, even for images that do contain the object emitting the sound, we can significantly boost the localization performance. We do so elegantly by introducing a mechanism to mine hard samples and add them to a contrastive learning formulation automatically. We show that our algorithm achieves state-of-the-art performance on the popular Flickr SoundNet dataset. Furthermore, we introduce the VGG-Sound Source (VGG-SS) benchmark, a new set of annotations for the recently-introduced VGG-Sound dataset, where the sound sources visible in each video clip are explicitly marked with bounding box annotations. This dataset is 20 times larger than analogous existing ones, contains 5K videos spanning over 200 categories, and, differently from Flickr SoundNet, is video-based. On VGG-SS, we also show that our algorithm achieves state-of-the-art performance against several baselines. Code and datasets can be found at http://www.robots.ox.ac.uk/˜vgg/research/lvs/.

Learning Representations from Audio-Visual Spatial Alignment

Abstract: We introduce a novel self-supervised pretext task for learning representations from audio-visual content. Prior work on audio-visual representation learning leverages correspondences at the video level. Approaches based on audio-visual correspondence (AVC) predict whether audio and video clips originate from the same or different video instances. Audio-visual temporal synchronization (AVTS) further discriminates negative pairs originated from the same video instance but at different moments in time. While these approaches learn high-quality representations for downstream tasks such as action recognition, their training objectives disregard spatial cues naturally occurring in audio and visual signals. To learn from these spatial cues, we tasked a network to perform contrastive audio-visual spatial alignment of 360{\deg} video and spatial audio. The ability to perform spatial alignment is enhanced by reasoning over the full spatial content of the 360{\deg} video using a transformer architecture to combine representations from multiple viewpoints. The advantages of the proposed pretext task are demonstrated on a variety of audio and visual downstream tasks, including audio-visual correspondence, spatial alignment, action recognition, and video semantic segmentation.

Audio-Visual Event Localization via Recursive Fusion by Joint Co-Attention

Abstract: The major challenge in audio-visual event localization task lies in how to fuse information from multiple modalities effectively. Recent works have shown that the attention mechanism is beneficial to the fusion process. In this paper, we propose a novel joint attention mechanism with multi-modal fusion methods for audio-visual event localization. Particularly, we present a concise yet valid architecture that effectively learns representations from multiple modalities in a joint manner. Initially, visual features are combined with auditory features and then turned into joint representations. Next, we make use of the joint representations to attend to visual features and auditory features, respectively. With the help of this joint co-attention, new visual and auditory features are produced, and thus both features can enjoy the mutually improved benefits from each other. It is worth noting that the joint co-attention unit is recursive meaning that it can be performed multiple times for obtaining better joint representations progressively. Extensive experiments on the public AVE dataset have shown that the proposed method achieves significantly better results than the state-of-the-art methods.

Exploiting Audio-Visual Consistency with Partial Supervision for Spatial Audio Generation

Abstract: Human perceives rich auditory experience with distinct sound heard by ears. Videos recorded with binaural audio particular simulate how human receives ambient sound. However, a large number of videos are with monaural audio only, which would degrade the user experience due to the lack of ambient information. To address this issue, we propose an audio spatialization framework to convert a monaural video into a binaural one exploiting the relationship across audio and visual components. By preserving the left-right consistency in both audio and visual modalities, our learning strategy can be viewed as a self-supervised learning technique, and alleviates the dependency on a large amount of video data with ground truth binaural audio data during training. Experiments on benchmark datasets confirm the effectiveness of our proposed framework in both semi-supervised and fully supervised scenarios, with ablation studies and visualization further support the use of our model for audio spatialization.

Co-Separating Sounds of Visual Objects

Abstract: Learning how objects sound from video is challenging, since they often heavily overlap in a single audio channel. Current methods for visually-guided audio source separation sidestep the issue by training with artificially mixed video clips, but this puts unwieldy restrictions on training data collection and may even prevent learning the properties of "true" mixed sounds. We introduce a co-separation training paradigm that permits learning object-level sounds from unlabeled multi-source videos. Our novel training objective requires that the deep neural network's separated audio for similar-looking objects be consistently identifiable, while simultaneously reproducing accurate video-level audio tracks for each source training pair. Our approach disentangles sounds in realistic test videos, even in cases where an object was not observed individually during training. We obtain state-of-the-art results on visually-guided audio source separation and audio denoising for the MUSIC, AudioSet, and AV-Bench datasets.

The Sound of Motions

Abstract: Sounds originate from object motions and vibrations of surrounding air. Inspired by the fact that humans is capable of interpreting sound sources from how objects move visually, we propose a novel system that explicitly captures such motion cues for the task of sound localization and separation. Our system is composed of an end-to-end learnable model called Deep Dense Trajectory (DDT), and a curriculum learning scheme. It exploits the inherent coherence of audio-visual signals from a large quantities of unlabeled videos. Quantitative and qualitative evaluations show that comparing to previous models that rely on visual appearance cues, our motion based system improves performance in separating musical instrument sounds. Furthermore, it separates sound components from duets of the same category of instruments, a challenging problem that has not been addressed before.

Curriculum Audiovisual Learning

Abstract: Associating sound and its producer in complex audiovisual scene is a challenging task, especially when we are lack of annotated training data. In this paper, we present a flexible audiovisual model that introduces a soft-clustering module as the audio and visual content detector, and regards the pervasive property of audiovisual concurrency as the latent supervision for inferring the correlation among detected contents. To ease the difficulty of audiovisual learning, we propose a novel curriculum learning strategy that trains the model from simple to complex scene. We show that such ordered learning procedure rewards the model the merits of easy training and fast convergence. Meanwhile, our audiovisual model can also provide effective unimodal representation and cross-modal alignment performance. We further deploy the well-trained model into practical audiovisual sound localization and separation task. We show that our localization model significantly outperforms existing methods, based on which we show comparable performance in sound separation without referring external visual supervision. Our video demo can be found at this https URL.

Overview of representative works for audio-visual correlation learning

Existing surveys introduce generic applications, core ideas and theoretical concepts of multimodal machine learning {{cite:fcb0044}}, {{cite:61715c8}}, and introduce audio-visual applications in the multimedia field {{cite:4c650aa}}. In contrast, this survey paper addresses the recent progress of deep learning models and their objective functions used for avcl. We address the natural complementary relationship between vision and audio that makes them suitable for exploiting underlying semantics. By analyzing the latest works on the field, we identified that the proliferation of attention-based models and objective functions based on semi-supervised learning have provided a new bloom for the field as it can be observed by the high performances demonstrated in Table {{table:89f4c31f-6f94-4958-8295-54cdf42c617e}} . Consequently, it is important to summarize new advances and discuss their pros and cons through a systematic comparison among existing deep avcl approaches.

In cross-modal correlation learning between audio and visual modalities , feature vectors of audio and visual sequences can be extracted directly by existing pre-trained models {{cite:ba8ba59}}, {{cite:52a1ac9}} or feature extractions are trained by following several DNN layers {{cite:6c92a64}}, {{cite:83ab17b}}. However, due to inconsistent distributions and representations between them, we have to learn a common space which can be utilized for bridging this gap, and further measuring and maximizing their correlation. In other words, since our input data ({{formula:ec011a2c-d2c7-4d70-9877-9167b0bcd04f}}  and {{formula:3adb4891-9d04-48a8-803c-f47b3f94d016}} ) belong to sub-spaces with different sizes, correlating audio-visual cross-modal data is done by using separate projections ({{formula:440d89b8-4738-4ab5-a3b6-c0ffa5131044}}  and {{formula:605edc88-54ac-475f-abe6-57e67397b7d4}} ). Generally, Deep avcl contains mainly two steps: i) To extract the temporal information from the frame-level audio and visual feature sequences, {{formula:2d1caf6a-e2cd-46c8-b456-7a545497b3f6}}  and {{formula:21a5f260-bc30-483f-a31a-f0f94b98d637}}  are fed into two separate sequence encoders. To better learn sequential dependencies between audio and visual data, recent works, such as attention mechanism, focus on synchronizing cross-modal semantic information and minimizing the heterogeneity gap simultaneously; ii) Objective functions are used to optimize these two sub-networks to maximize the correlation between audio and video in the joint space through back-propagation. According to different kinds of audio-visual data are exploited in deep learning stage, various objective functions are used in existing methods. In this section, we address the recent advances for step i) and ii) and discuss some tasks and paradigm in Deep avcl. In Table {{table:89f4c31f-6f94-4958-8295-54cdf42c617e}}  we provide a brief summary of these soa avcl methods.

As reflected in Table {{table:89f4c31f-6f94-4958-8295-54cdf42c617e}} , recent researches have intensively demonstrated that i) semi-supervised learning methods are able to achieve more discriminative features as it allows to leverage large volumes of (unlabeled) data, and ii) attention-based methods are designed to improve the synchronization between audio and visual sequences.


BioCreative V CDR task corpus: a resource for chemical disease relation extraction

Abstract: Community-run, formal evaluations and manually annotated text corpora are critically important for advancing biomedical text-mining research. Recently in BioCreative V, a new challenge was organized for the tasks of disease named entity recognition (DNER) and chemical-induced disease (CID) relation extraction. Given the nature of both tasks, a test collection is required to contain both disease/chemical annotations and relation annotations in the same set of articles. Despite previous efforts in biomedical corpus construction, none was found to be sufficient for the task. Thus, we developed our own corpus called BC5CDR during the challenge by inviting a team of Medical Subject Headings (MeSH) indexers for disease/chemical entity annotation and Comparative Toxicogenomics Database (CTD) curators for CID relation annotation. To ensure high annotation quality and productivity, detailed annotation guidelines and automatic annotation tools were provided. The resulting BC5CDR corpus consists of 1500 PubMed articles with 4409 annotated chemicals, 5818 diseases and 3116 chemical-disease interactions. Each entity annotation includes both the mention text spans and normalized concept identifiers, using MeSH as the controlled vocabulary. To ensure accuracy, the entities were first captured independently by two annotators followed by a consensus annotation: The average inter-annotator agreement (IAA) scores were 87.49% and 96.05% for the disease and chemicals, respectively, in the test set according to the Jaccard similarity coefficient. Our corpus was successfully used for the BioCreative V challenge tasks and should serve as a valuable resource for the text-mining research community. Database URL: http://www.biocreative.org/tasks/biocreative-v/track-3-cdr/

The SPECIES and ORGANISMS Resources for Fast and Accurate Identification of Taxonomic Names in Text

Abstract: The exponential growth of the biomedical literature is making the need for efficient, accurate text-mining tools increasingly clear. The identification of named biological entities in text is a central and difficult task. We have developed an efficient algorithm and implementation of a dictionary-based approach to named entity recognition, which we here use to identify names of species and other taxa in text. The tool, SPECIES, is more than an order of magnitude faster and as accurate as existing tools. The precision and recall was assessed both on an existing gold-standard corpus and on a new corpus of 800 abstracts, which were manually annotated after the development of the tool. The corpus comprises abstracts from journals selected to represent many taxonomic groups, which gives insights into which types of organism names are hard to detect and which are easy. Finally, we have tagged organism names in the entire Medline database and developed a web resource, ORGANISMS, that makes the results accessible to the broad community of biologists. The SPECIES software is open source and can be downloaded from http://species.jensenlab.org along with dictionary files and the manually annotated gold-standard corpus. The ORGANISMS web resource can be found at http://organisms.jensenlab.org.

List of downstream tasks.

Datasets.  We conduct FDAPT on the PubMed dataset, which contains abstracts and full-text research articles in the biomedical domain. The PubMed dataset adopted in our experiments {{cite:74b0aa4}} is smaller than the version used for BioBERT {{cite:1c0742f}}, which minimizes the experimental costs while maintaining pronounced results. Domain-specific PLMs are evaluated on 9 publicly available downstream tasks in the biomedical domain, including 6 Named Entity Recognition (NER), 2 Relation Extraction (RE) and 1 Question Answering (QA) datasets, as shown in Table {{table:b0ab2ef9-2b9c-41fc-b8e4-a499f27436b3}} .