Learning Gradient Fields for Molecular Conformation Generation

Abstract: We study a fundamental problem in computational chemistry known as molecular conformation generation, trying to predict stable 3D structures from 2D molecular graphs. Existing machine learning approaches usually first predict distances between atoms and then generate a 3D structure satisfying the distances, where noise in predicted distances may induce extra errors during 3D coordinate generation. Inspired by the traditional force field methods for molecular dynamics simulation, in this paper, we propose a novel approach called ConfGF by directly estimating the gradient fields of the log density of atomic coordinates. The estimated gradient fields allow directly generating stable conformations via Langevin dynamics. However, the problem is very challenging as the gradient fields are roto-translation equivariant. We notice that estimating the gradient fields of atomic coordinates can be translated to estimating the gradient fields of interatomic distances, and hence develop a novel algorithm based on recent score-based generative models to effectively estimate these gradients. Experimental results across multiple tasks show that ConfGF outperforms previous state-of-the-art baselines by a significant margin.

Pre-training Molecular Graph Representation with 3D Geometry

Abstract: Molecular graph representation learning is a fundamental problem in modern drug and material discovery. Molecular graphs are typically modeled by their 2D topological structures, but it has been recently discovered that 3D geometric information plays a more vital role in predicting molecular functionalities. However, the lack of 3D information in real-world scenarios has significantly impeded the learning of geometric graph representation. To cope with this challenge, we propose the Graph Multi-View Pre-training (GraphMVP) framework where self-supervised learning (SSL) is performed by leveraging the correspondence and consistency between 2D topological structures and 3D geometric views. GraphMVP effectively learns a 2D molecular graph encoder that is enhanced by richer and more discriminative 3D geometry. We further provide theoretical insights to justify the effectiveness of GraphMVP. Finally, comprehensive experiments show that GraphMVP can consistently outperform existing graph SSL methods.

E(n) Equivariant Graph Neural Networks

Abstract: This paper introduces a new model to learn graph neural networks equivariant to rotations, translations, reflections and permutations called E(n)-Equivariant Graph Neural Networks (EGNNs). In contrast with existing methods, our work does not require computationally expensive higher-order representations in intermediate layers while it still achieves competitive or better performance. In addition, whereas existing methods are limited to equivariance on 3 dimensional spaces, our model is easily scaled to higher-dimensional spaces. We demonstrate the effectiveness of our method on dynamical systems modelling, representation learning in graph autoencoders and predicting molecular properties.

SchNetPack: A Deep Learning Toolbox For Atomistic Systems

Abstract: SchNetPack is a toolbox for the development and application of deep neural networks to the prediction of potential energy surfaces and other quantum-chemical properties of molecules and materials. It contains basic building blocks of atomistic neural networks, manages their training and provides simple access to common benchmark datasets. This allows for an easy implementation and evaluation of new models. For now, SchNetPack includes implementations of (weighted) atomcentered symmetry functions and the deep tensor neural network SchNet as well as ready-to-use scripts that allow to train these models on molecule and material datasets. Based upon the PyTorch deep learning framework, SchNetPack allows to efficiently apply the neural networks to large datasets with millions of reference calculations as well as parallelize the model across multiple GPUs. Finally, SchNetPack provides an interface to the Atomic Simulation Environment in order to make trained models easily accessible to researchers that are not yet familiar with neural networks.

TorchMD-NET: Equivariant Transformers for Neural Network based Molecular Potentials

Abstract: The prediction of quantum mechanical properties is historically plagued by a trade-off between accuracy and speed. Machine learning potentials have previously shown great success in this domain, reaching increasingly better accuracy while maintaining computational efficiency comparable with classical force fields. In this work we propose TorchMD-NET, a novel equivariant transformer (ET) architecture, outperforming state-of-the-art on MD17, ANI-1, and many QM9 targets in both accuracy and computational efficiency. Through an extensive attention weight analysis, we gain valuable insights into the black box predictor and show differences in the learned representation of conformers versus conformations sampled from molecular dynamics or normal modes. Furthermore, we highlight the importance of datasets including off-equilibrium conformations for the evaluation of molecular potentials.

Fast Graph Representation Learning with PyTorch Geometric

Abstract: We introduce PyTorch Geometric, a library for deep learning on irregularly structured input data such as graphs, point clouds and manifolds, built upon PyTorch. In addition to general graph data structures and processing methods, it contains a variety of recently published methods from the domains of relational learning and 3D data processing. PyTorch Geometric achieves high data throughput by leveraging sparse GPU acceleration, by providing dedicated CUDA kernels and by introducing efficient mini-batch handling for input examples of different size. In this work, we present the library in detail and perform a comprehensive comparative study of the implemented methods in homogeneous evaluation scenarios.

List of open-source assets used.

Our code is available at https://anonymous.4open.science/r/3D-MGP-4931. The implementation and hyper-parameter settings are partially based on ConfGF {{cite:dd72f48}}, a generative model via denoising score matching. The implementation of baseline methods refers to GraphMVP {{cite:3189adb}}. The implementation of backbone equivariant geometric networks refers to the official codes of EGNN {{cite:494dc0b}}, SchNetPack {{cite:6ee1416}}, TorchMD-Net {{cite:afb68bf}} and Pytorch Geometric {{cite:4261cea}}. Detailed URLs and licenses are listed in Table {{table:968bd638-6486-4fea-ba44-b67cac06023f}} .


The first large-scale image-based dialogue dataset, including question-answer pairs about images.

A diagnostic dataset for studying multi-round reasoning and visual coreference resolution in image-based dialogue.

A dialogue dataset in which speakers are required to play roles with specific emotional mood or conversation style.

An open-domain dialogue dataset with image contexts extracted from movies and TV series.

The first video-based dialogue dataset, with conversations discussing the videos about indoor human activities.

A diagnostic video-grounded dialogue dataset containing annotations for different types of reasoning over videos.

Visual Dialog

Abstract: We introduce the task of Visual Dialog, which requires an AI agent to hold a meaningful dialog with humans in natural, conversational language about visual content. Specifically, given an image, a dialog history, and a question about the image, the agent has to ground the question in image, infer context from history, and answer the question accurately. Visual Dialog is disentangled enough from a specific downstream task so as to serve as a general test of machine intelligence, while being grounded in vision enough to allow objective evaluation of individual responses and benchmark progress. We develop a novel two-person chat data-collection protocol to curate a large-scale Visual Dialog dataset (VisDial). VisDial v0.9 has been released and contains 1 dialog with 10 question-answer pairs on ~120k images from COCO, with a total of ~1.2M dialog question-answer pairs. We introduce a family of neural encoder-decoder models for Visual Dialog with 3 encoders -- Late Fusion, Hierarchical Recurrent Encoder and Memory Network -- and 2 decoders (generative and discriminative), which outperform a number of sophisticated baselines. We propose a retrieval-based evaluation protocol for Visual Dialog where the AI agent is asked to sort a set of candidate answers and evaluated on metrics such as mean-reciprocal-rank of human response. We quantify gap between machine and human performance on the Visual Dialog task via human studies. Putting it all together, we demonstrate the first 'visual chatbot'! Our dataset, code, trained models and visual chatbot are available on https://visualdialog.org

CLEVR-Dialog: A Diagnostic Dataset for Multi-Round Reasoning in Visual Dialog

Abstract: Visual Dialog is a multimodal task of answering a sequence of questions grounded in an image (using the conversation history as context). It entails challenges in vision, language, reasoning, and grounding. However, studying these subtasks in isolation on large, real datasets is infeasible as it requires prohibitively-expensive complete annotation of the ‘state’ of all images and dialogs. We develop CLEVR-Dialog, a large diagnostic dataset for studying multi-round reasoning in visual dialog. Specifically, we construct a dialog grammar that is grounded in the scene graphs of the images from the CLEVR dataset. This combination results in a dataset where all aspects of the visual dialog are fully annotated. In total, CLEVR-Dialog contains 5 instances of 10-round dialogs for about 85k CLEVR images, totaling to 4.25M question-answer pairs. We use CLEVR-Dialog to benchmark performance of standard visual dialog models; in particular, on visual coreference resolution (as a function of the coreference distance). This is the first analysis of its kind for visual dialog models that was not possible without this dataset. We hope the findings from CLEVR-Dialog will help inform the development of future models for visual dialog. Our code and dataset are publicly available.

Image Chat: Engaging Grounded Conversations

Abstract: To achieve the long-term goal of machines being able to engage humans in conversation, our models should captivate the interest of their speaking partners. Communication grounded in images, whereby a dialogue is conducted based on a given photo, is a setup naturally appealing to humans (Hu et al., 2014). In this work we study large-scale architectures and datasets for this goal. We test a set of neural architectures using state-of-the-art image and text representations, considering various ways to fuse the components. To test such models, we collect a dataset of grounded human-human conversations, where speakers are asked to play roles given a provided emotional mood or style, as the use of such traits is also a key factor in engagingness (Guo et al., 2019). Our dataset, Image-Chat, consists of 202k dialogues over 202k images using 215 possible style traits. Automatic metrics and human evaluations of engagingness show the efficacy of our approach; in particular, we obtain state-of-the-art performance on the existing IGC task, and our best performing model is almost on par with humans on the Image-Chat test set (preferred 47.7% of the time).

OpenViDial: A Large-Scale, Open-Domain Dialogue Dataset with Visual Contexts

Abstract: When humans converse, what a speaker will say next significantly depends on what he sees. Unfortunately, existing dialogue models generate dialogue utterances only based on preceding textual contexts, and visual contexts are rarely considered. This is due to a lack of a large-scale multi-module dialogue dataset with utterances paired with visual contexts. In this paper, we release {\bf OpenViDial}, a large-scale multi-module dialogue dataset. The dialogue turns and visual contexts are extracted from movies and TV series, where each dialogue turn is paired with the corresponding visual context in which it takes place. OpenViDial contains a total number of 1.1 million dialogue turns, and thus 1.1 million visual contexts stored in images. Based on this dataset, we propose a family of encoder-decoder models leveraging both textual and visual contexts, from coarse-grained image features extracted from CNNs to fine-grained object features extracted from Faster R-CNNs. We observe that visual information significantly improves dialogue generation qualities, verifying the necessity of integrating multi-modal features for dialogue learning. Our work marks an important step towards large-scale multi-modal dialogue learning.

Audio Visual Scene-Aware Dialog

Abstract: We introduce the task of scene-aware dialog. Our goal is to generate a complete and natural response to a question about a scene, given video and audio of the scene and the history of previous turns in the dialog. To answer successfully, agents must ground concepts from the question in the video while leveraging contextual cues from the dialog history. To benchmark this task, we introduce the Audio Visual Scene-Aware Dialog (AVSD) Dataset. For each of more than 11,000 videos of human actions from the Charades dataset, our dataset contains a dialog about the video, plus a final summary of the video by one of the dialog participants. We train several baseline systems for this task and evaluate the performance of the trained models using both qualitative and quantitative metrics. Our results indicate that models must utilize all the available inputs (video, audio, question, and dialog history) to perform best on this dataset.

DVD: A Diagnostic Dataset for Multi-step Reasoning in Video Grounded Dialogue

Abstract: A video-grounded dialogue system is required to understand both dialogue, which contains semantic dependencies from turn to turn, and video, which contains visual cues of spatial and temporal scene variations. Building such dialogue systems is a challenging problem, involving various reasoning types on both visual and language inputs. Existing benchmarks do not have enough annotations to thoroughly analyze dialogue systems and understand their capabilities and limitations in isolation. These benchmarks are also not explicitly designed to minimise biases that models can exploit without actual reasoning. To address these limitations, in this paper, we present DVD, a Diagnostic Dataset for Video-grounded Dialogues. The dataset is designed to contain minimal biases and has detailed annotations for the different types of reasoning over the spatio-temporal space of video. Dialogues are synthesized over multiple question turns, each of which is injected with a set of cross-turn semantic relationships. We use DVD to analyze existing approaches, providing interesting insights into their abilities and limitations. In total, DVD is built from $11k$ CATER synthetic videos and contains $10$ instances of $10$-round dialogues for each video, resulting in more than $100k$ dialogues and $1M$ question-answer pairs. Our code and dataset are publicly available at https://github.com/facebookresearch/DVDialogues.

A review of available datasets.

Visual-context Augmented Dialogue Datasets

Massive data is undoubtedly an essential factor for training deep neural networks-based VAD. To promote researches in this field, many large-scale and high-quality datasets have been released, summarized in Table {{table:bc6cdfd1-5648-4a37-abcc-b40d7fe64306}} .


On the Origin of the Gamma-Ray Emission toward SNR CTB 37A with $Fermi$-LAT

Abstract: The middle-aged supernova remnant (SNR) CTB 37A is known to interact with several dense molecular clouds through the detection of shocked ${\rm H_{2}}$ and OH 1720 MHz maser emission. In the present work, we use eight years of $\textit Fermi$-LAT Pass 8 data, with an improved point-spread function and an increased acceptance, to perform detailed morphological and spectral studies of the $\gamma$-ray emission toward CTB 37A from 200 MeV to 200 GeV. The best fit of the source extension is obtained for a very compact Gaussian model with a significance of 5.75$\sigma$ and a 68\% containment radius of $0.116^{\circ}$ $\pm$ $0.014^{\circ}_{\rm stat}$ $\pm$ $0.017^{\circ}_{\rm sys}$ above 1 GeV, which is larger than the TeV emission size. The energy spectrum is modeled as a LogParabola, resulting in a spectral index $\alpha$ = 1.92 $\pm$ 0.19 at 1 GeV and a curvature $\beta$ = 0.18 $\pm$ 0.05, which becomes softer than the TeV spectrum above 10 GeV. The SNR properties, including a dynamical age of 6000 yr, are derived assuming the Sedov phase. From the multiwavelength modeling of emission toward the remnant, we conclude that the nonthermal radio and GeV emission is mostly due to the reacceleration of preexisting cosmic rays (CRs) by radiative shocks in the adjacent clouds. Furthermore, the observational data allow us to constrain the total kinetic energy transferred to the trapped CRs in the clouds. Based on these facts, we infer a composite nature for CTB 37A to explain the broadband spectrum and to elucidate the nature of the observed $\gamma$-ray emission.

HAWC J2227+610 and its association with G106.3+2.7, a new potential Galactic PeVatron

Abstract: We present the detection of VHE gamma-ray emission above 100 TeV from HAWC J2227+610 with the HAWC observatory. Combining our observations with previously published results by VERITAS, we interpret the gamma-ray emission from HAWC J2227+610 as emission from protons with a lower limit in their cutoff energy of 800 TeV. The most likely source of the protons is the associated supernova remnant G106.3+2.7, making it a good candidate for a Galactic PeVatron. However, a purely leptonic origin of the observed emission cannot be excluded at this time.

Expansion measurement of Supernova Remnant RX J1713.7-3946

Abstract: Supernova remnant (SNR) RX J1713.7-3946 is well known for its bright TeV gamma-ray emission with shell-like morphology. To constrain the hydrodynamical evolution, we have performed six times observations of the northwestern (NW) shell with the Chandra X-ray Observatory from 2005 to 2011, and measured the proper motion by using these data and the first epoch observation taken in 2000. The blast-wave shock speed at the NW shell is measured to be $(3900\pm 300) (d/{\rm kpc})\ {\rm km}\ {\rm s}^{-1}$ with an estimated distance of $d = 1$ kpc, and the proper motions of other structures within the NW shell are significantly less than that. Assuming that the measured blast-wave shock speed is the representative of the remnant's outer shock wave as a whole, we have confronted our measurements as well as a recent detection of thermal X-ray lines, with the analytic solution of the hydrodynamical properties of SNRs. Our hydrodynamical analysis indicates that the age of the remnant is 1580-2100 years, supporting the association with SN393. A model with SN kinetic energy of $E = 10^{51}\, \rm erg$, the ejecta mass of $M_{\rm ej} = 3\, M_{\odot}$, and the ambient density at the current blast wave location of $n_2 = 0.015\, {\rm cm}^{-3}$, provides reasonable explanation for our measurements and previous findings at the X-ray and gamma-ray wavelengths. We find that the transition to the Sedov-Taylor (ST) phase is incomplete for any reasonable set of parameters, implying that the current maximum energy of accelerated protons in RX J1713.7-3946 would not correspond to the maximum attainable energy for this remnant.

Cosmic-ray energy spectrum and composition up to the ankle - the case for a second Galactic component

Abstract: We have carried out a detailed study to understand the observed energy spectrum and composition of cosmic rays with energies up to ~10^18 eV. Our study shows that a single Galactic component with subsequent energy cut-offs in the individual spectra of different elements, optimised to explain the observed spectra below ~10^14 eV and the knee in the all-particle spectrum, cannot explain the observed all-particle spectrum above ~2x10^16 eV. We discuss two approaches for a second component of Galactic cosmic rays -- re-acceleration at a Galactic wind termination shock, and supernova explosions of Wolf-Rayet stars, and show that the latter scenario can explain almost all observed features in the all-particle spectrum and the composition up to ~10^18 eV, when combined with a canonical extra-galactic spectrum expected from strong radio galaxies or a source population with similar cosmological evolution. In this two-component Galactic model, the knee at ~ 3x10^15 eV and the second knee at ~10^17 eV in the all-particle spectrum are due to the cut-offs in the first and second components, respectively. We also discuss several variations of the extra-galactic component, from a minimal contribution to scenarios with a significant component below the ankle (at ~4x10^18 eV), and find that extra-galactic contributions in excess of regular source evolution are neither indicated nor in conflict with the existing data. Our main result is that the second Galactic component predicts a composition of Galactic cosmic rays at and above the second knee that largely consists of helium or a mixture of helium and CNO nuclei, with a weak or essentially vanishing iron fraction, in contrast to most common assumptions. This prediction is in agreement with new measurements from LOFAR and the Pierre Auger Observatory which indicate a strong light component and a rather low iron fraction between ~10^17 and 10^18 eV.

List of the chosen SNRs along with their type (Int. refers to SNRs interacting with the surrounding medium) and age.

After these considerations, we are left with the 9 SNRs listed in Table {{table:65da9d6e-149d-4a99-8610-87d02a81bc16}} . All the data points and their origin are listed in Table {{table:3cb196bd-8b71-47a7-aa32-602ec12163b9}}  in the appendix. For each SNR, we provide information about its type (Table {{table:741b3830-9146-4d1c-a067-cd3aae88140b}} ) which can be found in TeVCat and age (Table {{table:65da9d6e-149d-4a99-8610-87d02a81bc16}} ) along with the references.

In the following discussion, the information on the type and age of SNRs has been referred from papers shown in Table {{table:65da9d6e-149d-4a99-8610-87d02a81bc16}} .


Matching Networks for One Shot Learning

Abstract: Learning from a few examples remains a key challenge in machine learning. Despite recent advances in important domains such as vision and language, the standard supervised deep learning paradigm does not offer a satisfactory solution for learning new concepts rapidly from little data. In this work, we employ ideas from metric learning based on deep neural features and from recent advances that augment neural networks with external memories. Our framework learns a network that maps a small labelled support set and an unlabelled example to its label, obviating the need for fine-tuning to adapt to new class types. We then define one-shot learning problems on vision (using Omniglot, ImageNet) and language tasks. Our algorithm improves one-shot accuracy on ImageNet from 87.6% to 93.2% and from 88.0% to 93.8% on Omniglot compared to competing approaches. We also demonstrate the usefulness of the same model on language modeling by introducing a one-shot task on the Penn Treebank.

Meta-Learning for Semi-Supervised Few-Shot Classification

Abstract: In few-shot classification, we are interested in learning algorithms that train a classifier from only a handful of labeled examples. Recent progress in few-shot classification has featured meta-learning, in which a parameterized model for a learning algorithm is defined and trained on episodes representing different classification problems, each with a small labeled training set and its corresponding test set. In this work, we advance this few-shot classification paradigm towards a scenario where unlabeled examples are also available within each episode. We consider two situations: one where all unlabeled examples are assumed to belong to the same set of classes as the labeled examples of the episode, as well as the more challenging situation where examples from other distractor classes are also provided. To address this paradigm, we propose novel extensions of Prototypical Networks (Snell et al., 2017) that are augmented with the ability to use unlabeled examples when producing prototypes. These models are trained in an end-to-end way on episodes, to learn to leverage the unlabeled examples successfully. We evaluate these methods on versions of the Omniglot and miniImageNet benchmarks, adapted to this new framework augmented with unlabeled examples. We also propose a new split of ImageNet, consisting of a large set of classes, with a hierarchical structure. Our experiments confirm that our Prototypical Networks can learn to improve their predictions due to unlabeled examples, much like a semi-supervised algorithm would.

Information on the four datasets used.

A summary of dataset statistics is given in Table {{table:17236178-d702-40f9-9112-91254f633674}} , with details on each dataset below.


Stable clustering, the halo model and nonlinear cosmological power spectra

Abstract: We present the results of a large library of cosmological N-body simulations, using power-law initial spectra. The nonlinear evolution of the matter power spectra is compared with the predictions of existing analytic scaling formulae based on the work of Hamilton et al. The scaling approach has assumed that highly nonlinear structures obey `stable clustering' and are frozen in proper coordinates. Our results show that, when transformed under the self-similarity scaling, the scale-free spectra define a nonlinear locus that is clearly shallower than would be required under stable clustering. Furthermore, the small-scale nonlinear power increases as both the power-spectrum index n and the density parameter Omega decrease, and this evolution is not well accounted for by the previous scaling formulae. This breakdown of stable clustering can be understood as resulting from the modification of dark-matter haloes by continuing mergers. These effects are naturally included in the analytic `halo model' for nonlinear structure; using this approach we are able to fit both our scale-free results and also our previous CDM data. This approach is more accurate than the commonly-used Peacock--Dodds formula and should be applicable to more general power spectra. Code to evaluate nonlinear power spectra using this method is available from http://as1.chem.nottingham.ac.uk/~res/software.html Following publication, we will make the power-law simulation data available through the Virgo website http://www.mpa-garching.mpg.de/Virgo

An accurate halo model for fitting non-linear cosmological power spectra and baryonic feedback models

Abstract: We present an optimized variant of the halo model, designed to produce accurate matter power spectra well into the non-linear regime for a wide range of cosmological models. To do this, we introduce physically motivated free parameters into the halo-model formalism and fit these to data from high-resolution $N$-body simulations. For a variety of $\Lambda$CDM and $w$CDM models the halo-model power is accurate to $\simeq 5$ per cent for $k\leq 10h\,\mathrm{Mpc}$ and $z\leq 2$. An advantage of our new halo model is that it can be adapted to account for the effects of baryonic feedback on the power spectrum. We demonstrate this by fitting the halo model to power spectra from the OWLS hydrodynamical simulation suite via parameters that govern halo internal structure. We are able to fit all feedback models investigated at the 5 per cent level using only two free parameters, and we place limits on the range of these halo parameters for feedback models investigated by the OWLS simulations. Accurate predictions to high-$k$ are vital for weak lensing surveys, and these halo parameters could be considered nuisance parameters to marginalize over in future analyses to mitigate uncertainty regarding the details of feedback. Finally, we investigate how lensing observables predicted by our model compare to those from simulations and from HALOFIT for a range of $k$-cuts and feedback models and quantify the angular scales at which these effects become important. Code to calculate power spectra from the model presented in this paper can be found at https://github.com/alexander-mead/hmcode

hmcode-2020: improved modelling of non-linear cosmological power spectra with baryonic feedback

Abstract: We present an updated version of the hmcode augmented halo model that can be used to make accurate predictions of the non-linear matter power spectrum over a wide range of cosmologies. Major improvements include modelling of baryon-acoustic oscillation (BAO) damping in the power spectrum and an updated treatment of massive neutrinos. We fit our model to simulated power spectra and show that we can match the results with an root mean square (RMS) error of 2.5 per cent across a range of cosmologies, scales $k \lt 10\, h\, \mathrm{Mpc}^{-1}$, and redshifts z < 2. The error rarely exceeds 5 per cent and never exceeds 16 per cent. The worst-case errors occur at z ≃ 2, or for cosmologies with unusual dark energy equations of state. This represents a significant improvement over previous versions of hmcode, and over other popular fitting functions, particularly for massive-neutrino cosmologies with high neutrino mass. We also present a simple halo model that can be used to model the impact of baryonic feedback on the power spectrum. This six-parameter physical model includes gas expulsion by active galactic nuclei (AGN) feedback and encapsulates star formation. By comparing this model to data from hydrodynamical simulations, we demonstrate that the power spectrum response to feedback is matched at the <1 per cent level for z < 1 and $k\lt 20\, h\, \mathrm{Mpc}^{-1}$. We also present a single-parameter variant of this model, parametrized in terms of feedback strength, which is only slightly less accurate. We make code available for our non-linear and baryon models at https://github.com/alexander-mead/HMcode and it is also available within camb and soon within class.

On the road to percent accuracy: nonlinear reaction of the matter power spectrum to dark energy and modified gravity

Abstract: We present a general method to compute the nonlinear matter power spectrum for dark energy and modified gravity scenarios with percent-level accuracy. By adopting the halo model and nonlinear perturbation theory, we predict the reaction of a $\Lambda$CDM matter power spectrum to the physics of an extended cosmological parameter space. By comparing our predictions to $N$-body simulations we demonstrate that with no-free parameters we can recover the nonlinear matter power spectrum for a wide range of different $w_0$-$w_a$ dark energy models to better than 1% accuracy out to $k \approx 1 \, h \, {\rm Mpc}^{-1}$. We obtain a similar performance for both DGP and $f(R)$ gravity, with the nonlinear matter power spectrum predicted to better than 3% accuracy over the same range of scales. When including direct measurements of the halo mass function from the simulations, this accuracy improves to 1%. With a single suite of standard $\Lambda$CDM $N$-body simulations, our methodology provides a direct route to constrain a wide range of non-standard extensions to the concordance cosmology in the high signal-to-noise nonlinear regime.

The BACCO Simulation Project: Exploiting the full power of large-scale structure for cosmology

Abstract: We present the BACCO project, a simulation framework specially designed to provide highly-accurate predictions for the distribution of mass, galaxies, and gas as a function of cosmological parameters. In this paper, we describe our main suite of simulations (L $\sim2$ Gpc and $4320^3$ particles) and present various validation tests. Using a cosmology-rescaling technique, we predict the nonlinear mass power spectrum over the redshift range $0<z<1.5$ and over scales $10^{-2} < k/(h Mpc^{-1} ) < 5$ for 800 points in an 8-dimensional cosmological parameter space. For an efficient interpolation of the results, we build an emulator and compare its predictions against several widely-used methods. Over the whole range of scales considered, we expect our predictions to be accurate at the 2\% level for parameters in the minimal $\Lambda$ CDM model and to 3\% when extended to dynamical dark energy and massive neutrinos. We make our emulator publicly available under http://www.dipc.org/bacco

Euclid preparation: II. The EuclidEmulator -- A tool to compute the cosmology dependence of the nonlinear matter power spectrum

Abstract: We present a new power spectrum emulator named EuclidEmulator that estimates the nonlinear correction to the linear dark matter power spectrum. It is based on a spectral decomposition method called polynomial chaos expansion. All steps in the construction of the emulator have been tested and optimized: the large high-resolution N-body simulations carried out with PKDGRAV3 were validated using a simulation from the Euclid Flagship campaign and demonstrated to have converged up to wavenumbers $k\approx 5\,h\,{\rm Mpc}^{-1}$ for redshifts $z\leq 5$. The emulator is constructed using the uncertainty quantification software UQLab and it has been optimized first by creating mock emulators based on Takahashi's HALOFIT. We show that it is possible to successfully predict the performance of the final emulator in this way prior to performing any N-body simulations. We provide a C-code to calculate the nonlinear correction at a relative accuracy of $\sim0.3\%$ with respect to N-body simulations within 50 ms. The absolute accuracy of the final nonlinear power spectrum is comparable to one obtained with N-body simulations, i.e. $\sim 1\%$ for $k\lesssim 1\,h\,{\rm Mpc}^{-1}$ and $z\lesssim 3.5$. This enables efficient forward modeling in the nonlinear regime allowing for maximum likelihood estimation of cosmological parameters. EuclidEmulator has been compared to HALOFIT and CosmicEmu, an alternative emulator based on the Mira-Titan Universe, and shown to be more accurate than these other approaches. This work paves a new way for optimal construction of future emulators that also consider other cosmological observables, use higher resolution input simulations and investigate higher dimensional cosmological parameter spaces.

List of the nonlinear prescriptions of {{formula:f2e7d437-5421-4a17-9164-aa5e2b8b91b3}}  analysed in this work.

Modelling the nonlinear matter power spectrum

We briefly describe in this Sec. the nonlinear prescriptions considered in this work for the modelling of {{formula:849a11c4-fe69-4c6f-9f17-696a3c98fa7f}} .
In general, we can recognise three different categories to which they belong: fitting functions, halo-models and emulators.
We quickly review their main features and summarise them in Tab. {{table:8c1592d5-f706-4aad-b912-a7981a6bb64b}} .


The Kinetics Human Action Video Dataset

Abstract: We describe the DeepMind Kinetics human action video dataset. The dataset contains 400 human action classes, with at least 400 video clips for each action. Each clip lasts around 10s and is taken from a different YouTube video. The actions are human focussed and cover a broad range of classes including human-object interactions such as playing instruments, as well as human-human interactions such as shaking hands. We describe the statistics of the dataset, how it was collected, and give some baseline performance figures for neural network architectures trained and tested for human action classification on this dataset. We also carry out a preliminary analysis of whether imbalance in the dataset leads to bias in the classifiers.

AVA: A Video Dataset of Spatio-temporally Localized Atomic Visual Actions

Abstract: This paper introduces a video dataset of spatio-temporally localized Atomic Visual Actions (AVA). The AVA dataset densely annotates 80 atomic visual actions in 430 15-minute video clips, where actions are localized in space and time, resulting in 1.58M action labels with multiple labels per person occurring frequently. The key characteristics of our dataset are: (1) the definition of atomic visual actions, rather than composite actions; (2) precise spatio-temporal annotations with possibly multiple annotations for each person; (3) exhaustive annotation of these atomic actions over 15-minute video clips; (4) people temporally linked across consecutive segments; and (5) using movies to gather a varied set of action representations. This departs from existing datasets for spatio-temporal action recognition, which typically provide sparse annotations for composite actions in short video clips. We will release the dataset publicly. AVA, with its realistic scene and action complexity, exposes the intrinsic difficulty of action recognition. To benchmark this, we present a novel approach for action localization that builds upon the current state-of-the-art methods, and demonstrates better performance on JHMDB and UCF101-24 categories. While setting a new state of the art on existing datasets, the overall results on AVA are low at 15.6% mAP, underscoring the need for developing new approaches for video understanding.

Large-Scale Study of Perceptual Video Quality

Abstract: The great variations of videographic skills, camera designs, compression and processing protocols, and displays lead to an enormous variety of video impairments. Current no-reference (NR) video quality models are unable to handle this diversity of distortions. This is true in part because available video quality assessment databases contain very limited content, fixed resolutions, were captured using a small number of camera devices by a few videographers and have been subjected to a modest number of distortions. As such, these databases fail to adequately represent real world videos, which contain very different kinds of content obtained under highly diverse imaging conditions and are subject to authentic, often commingled distortions that are impossible to simulate. As a result, NR video quality predictors tested on real-world video data often perform poorly. Towards advancing NR video quality prediction, we constructed a large-scale video quality assessment database containing 585 videos of unique content, captured by a large number of users, with wide ranges of levels of complex, authentic distortions. We collected a large number of subjective video quality scores via crowdsourcing. A total of 4776 unique participants took part in the study, yielding more than 205000 opinion scores, resulting in an average of 240 recorded human opinions per video. We demonstrate the value of the new resource, which we call the LIVE Video Quality Challenge Database (LIVE-VQC), by conducting a comparison of leading NR video quality predictors on it. This study is the largest video quality assessment study ever conducted along several key dimensions: number of unique contents, capture devices, distortion types and combinations of distortions, study participants, and recorded subjective scores. The database is available for download on this link: http://live.ece.utexas.edu/research/LIVEVQC/index.html .

Subjective Quality Assessment For Youtube Ugc Dataset

Abstract: Due to the scale of social video sharing, User Generated Content (UGC) is getting more attention from academia and industry. To facilitate compression-related research on UGC, YouTube has released a large-scale dataset [1]. The initial dataset only provided videos, limiting its use in quality assessment. We used a crowd-sourcing platform to collect subjective quality scores for this dataset. We analyzed the distribution of Mean Opinion Score (MOS) in various dimensions, and investigated some fundamental questions in video quality assessment, like the correlation between full video MOS and corresponding chunk MOS, and the influence of chunk variation in quality score aggregation.

Patch-VQ: 'Patching Up' the Video Quality Problem

Abstract: No-reference (NR) perceptual video quality assessment (VQA) is a complex, unsolved, and important problem to social and streaming media applications. Efficient and accurate video quality predictors are needed to monitor and guide the processing of billions of shared, often imperfect, user-generated content (UGC). Unfortunately, current NR models are limited in their prediction capabilities on real-world, "in-the-wild" UGC video data. To advance progress on this problem, we created the largest (by far) subjective video quality dataset, containing 39, 000 realworld distorted videos and 117, 000 space-time localized video patches ('v-patches'), and 5.5M human perceptual quality annotations. Using this, we created two unique NR-VQA models: (a) a local-to-global region-based NR VQA architecture (called PVQ) that learns to predict global video quality and achieves state-of-the-art performance on 3 UGC datasets, and (b) a first-of-a-kind space-time video quality mapping engine (called PVQ Mapper) that helps localize and visualize perceptual distortions in space and time. We will make the new database and prediction models available immediately following the review process.

Common datasets in VQA and other video tasks. Most common VQA datasets are to small (noted in red) to learn sufficient quality representations independently.

Tab. {{table:ad11c008-2e24-460d-9b55-e62738e2d8cb}}  shows common VQA datasets, other video datasets and their sizes. The early VQA datasets {{cite:a4b4cd1}}, {{cite:505c1a8}} are synthesized with specialized distortion and have a very small volume. Some recent in-the-wild VQA datasets like KoNViD-1k {{cite:7e3e05f}}, YouTube-UGC {{cite:778e962}} and LIVE-VQC {{cite:313a67a}} are still small compared to datasets for other video tasks such as {{cite:1f3852a}}, {{cite:a62b786}}, {{cite:75948b2}}.
Recently, LSVQ{{cite:a749b75}}, a large-scale VQA dataset with 39,076 videos is publicly available. With end-to-end deep learning of the proposed FAST-VQA, the video-quality-related features learnt on large-scale LSVQ dataset can be transferred into smaller VQA datasets to reach better performance.


Using wearable proximity sensors to characterize social contact patterns in a village of rural Malawi

Abstract: Measuring close proximity interactions between individuals can provide key information on social contacts in human communities. With the present study, we report the quantitative assessment of contact patterns in a village in rural Malawi, based on proximity sensors technology that allows for high-resolution measurements of social contacts. The system provided information on community structure of the village, on social relationships and social assortment between individuals, and on daily contacts activity within the village. Our findings revealed that the social network presented communities that were highly correlated with household membership, thus confirming the importance of family ties within the village. Contacts within households occur mainly between adults and children, and adults and adolescents. This result suggests that the principal role of adults within the family is the care for the youngest. Most of the inter-household interactions occurred among caregivers and among adolescents. We studied the tendency of participants to interact with individuals with whom they shared similar attributes (i.e., assortativity). Age and gender assortativity were observed in inter-household network, showing that individuals not belonging to the same family group prefer to interact with people with whom they share similar age and gender. Age disassortativity is observed in intra-household networks. Family members congregate in the early morning, during lunch time and dinner time. In contrast, individuals not belonging to the same household displayed a growing contact activity from the morning, reaching a maximum in the afternoon. The data collection infrastructure used in this study seems to be very effective to capture the dynamics of contacts by collecting high resolution temporal data and to give access to the level of information needed to understand the social context of the village.

Can co-location be used as a proxy for face-to-face contacts?

Abstract: Technological advances have led to a strong increase in the number of data collection efforts aimed at measuring co-presence of individuals at different spatial resolutions. It is however unclear how much co-presence data can inform us on actual face-to-face contacts, of particular interest to study the structure of a population in social groups or for use in data-driven models of information or epidemic spreading processes. Here, we address this issue by leveraging data sets containing high resolution face-to-face contacts as well as a coarser spatial localisation of individuals, both temporally resolved, in various contexts. The co-presence and the face-to-face contact temporal networks share a number of structural and statistical features, but the former is (by definition) much denser than the latter. We thus consider several down-sampling methods that generate surrogate contact networks from the co-presence signal and compare them with the real face-to-face data. We show that these surrogate networks reproduce some features of the real data but are only partially able to identify the most central nodes of the face-to-face network. We then address the issue of using such down-sampled co-presence data in data-driven simulations of epidemic processes, and in identifying efficient containment strategies. We show that the performance of the various sampling methods strongly varies depending on context. We discuss the consequences of our results with respect to data collection strategies and methodologies.

Summary statistics for the 16 datasets used for homophily comparisons, 9 of which are also used for link prediction experiments.

   To understand the effect of different homophily definitions in empirical networks, we apply the definitions to the 16 publicly available datasets described in Table {{table:3501ffc2-238d-4000-85a9-92bdcfa8f1d2}} .
With the empirical networks, we are able to quantify the difference between the {{formula:89decf19-6414-4522-8683-bc889bcc4c1b}} -simplicial homophily score and the hypergraph homophily score for triadic interactions.
Using the definitions from Section REF , we compute the homogeneous homophily scores for all 16 datasets and display them in Figure {{figure:93daf115-e090-4e90-ba87-b0aca38905d5}} .
For all but one dataset (contact-hospital), the hypergraph homophily score is higher than the {{formula:12ef0089-54dd-4767-af0b-164e565a5b24}} -simplicial homophily score, consistent with synthetic experiments where {{formula:0373d12f-d96f-43d9-b2cb-42babdb4b616}} .
The result is particularly strong for retail-trivago, cont-high-school, bills-house, and coauth-dblp, for which {{formula:0ef6a463-de6f-46ba-b781-8bc8fadde267}} -simplicial homophily suggests anti-homophily in group formation.
In retail-trivago, which has the strongest tendency for anti-homophily, we posit that travelers headed to a specific destination might look at two hotels to compare cost and amenities, but if a traveler is browsing more than three hotels, they are likely taking a longer trip or have more flexibility for their search.
For the remaining three datasets, the tendency for anti-homophily is much smaller, but can still be explained by a desire for diversity in larger group sizes.


Extra regularization techniques are required

Limited to unsophisticated adversarial instances

Face-like features as adversarial perturbations

Efficient Decision-based Black-box Adversarial Attacks on Face Recognition

Abstract: Face recognition has obtained remarkable progress in recent years due to the great improvement of deep convolutional neural networks (CNNs). However, deep CNNs are vulnerable to adversarial examples, which can cause fateful consequences in real-world face recognition applications with security-sensitive purposes. Adversarial attacks are widely studied as they can identify the vulnerability of the models before they are deployed. In this paper, we evaluate the robustness of state-of-the-art face recognition models in the decision-based black-box attack setting, where the attackers have no access to the model parameters and gradients, but can only acquire hard-label predictions by sending queries to the target model. This attack setting is more practical in real-world face recognition systems. To improve the efficiency of previous methods, we propose an evolutionary attack algorithm, which can model the local geometries of the search directions and reduce the dimension of the search space. Extensive experiments demonstrate the effectiveness of the proposed method that induces a minimum perturbation to an input face image with fewer queries. We also apply the proposed method to attack a real-world face recognition system successfully.

Invisible Mask: Practical Attacks on Face Recognition with Infrared

Abstract: Accurate face recognition techniques make a series of critical applications possible: policemen could employ it to retrieve criminals' faces from surveillance video streams; cross boarder travelers could pass a face authentication inspection line without the involvement of officers. Nonetheless, when public security heavily relies on such intelligent systems, the designers should deliberately consider the emerging attacks aiming at misleading those systems employing face recognition. We propose a kind of brand new attack against face recognition systems, which is realized by illuminating the subject using infrared according to the adversarial examples worked out by our algorithm, thus face recognition systems can be bypassed or misled while simultaneously the infrared perturbations cannot be observed by raw eyes. Through launching this kind of attack, an attacker not only can dodge surveillance cameras. More importantly, he can impersonate his target victim and pass the face authentication system, if only the victim's photo is acquired by the attacker. Again, the attack is totally unobservable by nearby people, because not only the light is invisible, but also the device we made to launch the attack is small enough. According to our study on a large dataset, attackers have a very high success rate with a over 70\% success rate for finding such an adversarial example that can be implemented by infrared. To the best of our knowledge, our work is the first one to shed light on the severity of threat resulted from infrared adversarial examples against face recognition.

Adversarial Light Projection Attacks on Face Recognition Systems: A Feasibility Study

Abstract: Deep learning-based systems have been shown to be vulnerable to adversarial attacks in both digital and physical domains. While feasible, digital attacks have limited applicability in attacking deployed systems, including face recognition systems, where an adversary typically has access to the input and not the transmission channel. In such setting, physical attacks that directly provide a malicious input through the input channel pose a bigger threat. We investigate the feasibility of conducting real-time physical attacks on face recognition systems using adversarial light projections. A setup comprising a commercially available web camera and a projector is used to conduct the attack. The adversary uses a transformation-invariant adversarial pattern generation method to generate a digital adversarial pattern using one or more images of the target available to the adversary. The digital adversarial pattern is then projected onto the adversary's face in the physical domain to either impersonate a target (impersonation) or evade recognition (obfuscation). We conduct preliminary experiments using two open-source and one commercial face recognition system on a pool of 50 subjects. Our experimental results demonstrate the vulnerability of face recognition systems to light projection attacks in both white-box and black-box attack settings.

Improving Transferability of Adversarial Patches on Face Recognition with Generative Models

Abstract: Face recognition is greatly improved by deep convolutional neural networks (CNNs). Recently, these face recognition models have been used for identity authentication in security sensitive applications. However, deep CNNs are vulnerable to adversarial patches, which are physically realizable and stealthy, raising new security concerns on the real-world applications of these models. In this paper, we evaluate the robustness of face recognition models using adversarial patches based on transferability, where the attacker has limited accessibility to the target models. First, we extend the existing transfer-based attack techniques to generate transferable adversarial patches. However, we observe that the transferability is sensitive to initialization and degrades when the perturbation magnitude is large, indicating the overfitting to the substitute models. Second, we propose to regularize the adversarial patches on the low dimensional data manifold. The manifold is represented by generative models pre-trained on legitimate human face images. Using face-like features as adversarial perturbations through optimization on the manifold, we show that the gaps between the responses of substitute models and the target models dramatically decrease, exhibiting a better transferability. Extensive digital world experiments are conducted to demonstrate the superiority of the proposed method in the black-box setting. We apply the proposed method in the physical world as well.

Unravelling Robustness of Deep Learning based Face Recognition Against Adversarial Attacks

Abstract: Deep neural network (DNN) architecture based models have high expressive power and learning capacity. However, they are essentially a black box method since it is not easy to mathematically formulate the functions that are learned within its many layers of representation. Realizing this, many researchers have started to design methods to exploit the drawbacks of deep learning based algorithms questioning their robustness and exposing their singularities. In this paper, we attempt to unravel three aspects related to the robustness of DNNs for face recognition: (i) assessing the impact of deep architectures for face recognition in terms of vulnerabilities to attacks inspired by commonly observed distortions in the real world that are well handled by shallow learning methods along with learning based adversaries; (ii) detecting the singularities by characterizing abnormal filter response behavior in the hidden layers of deep networks; and (iii) making corrections to the processing pipeline to alleviate the problem. Our experimental evaluation using multiple open-source DNN-based face recognition networks, including OpenFace and VGG-Face, and two publicly available databases (MEDS and PaSC) demonstrates that the performance of deep learning based face recognition algorithms can suffer greatly in the presence of such distortions. The proposed method is also compared with existing detection algorithms and the results show that it is able to detect the attacks with very high accuracy by suitably designing a classifier using the response of the hidden layers in the network. Finally, we present several effective countermeasures to mitigate the impact of adversarial attacks and improve the overall robustness of DNN-based face recognition.

Amora: Black-box Adversarial Morphing Attack

Abstract: Nowadays, digital facial content manipulation has become ubiquitous and realistic with the success of generative adversarial networks (GANs), making face recognition (FR) systems suffer from unprecedented security concerns. In this paper, we investigate and introduce a new type of adversarial attack to evade FR systems by manipulating facial content, called adversarial morphing attack (a.k.a. Amora). In contrast to adversarial noise attack that perturbs pixel intensity values by adding human-imperceptible noise, our proposed adversarial morphing attack works at the semantic level that perturbs pixels spatially in a coherent manner. To tackle the black-box attack problem, we devise a simple yet effective joint dictionary learning pipeline to obtain a proprietary optical flow field for each attack. Our extensive evaluation on two popular FR systems demonstrates the effectiveness of our adversarial morphing attack at various levels of morphing intensity with smiling facial expression manipulations. Both open-set and closed-set experimental results indicate that a novel black-box adversarial attack based on local deformation is possible, and is vastly different from additive noise attacks. The findings of this work potentially pave a new research direction towards a more thorough understanding and investigation of image-based adversarial attacks and defenses.

Fast Geometrically-Perturbed Adversarial Faces

Abstract: The state-of-the-art performance of deep learning algorithms has led to a considerable increase in the utilization of machine learning in security-sensitive and critical applications. However, it has recently been shown that a small and carefully crafted perturbation in the input space can completely fool a deep model. In this study, we explore the extent to which face recognition systems are vulnerable to geometrically-perturbed adversarial faces. We propose a fast landmark manipulation method for generating adversarial faces, which is approximately 200 times faster than the previous geometric attacks and obtains 99.86% success rate on the state-of-the-art face recognition models. To further force the generated samples to be natural, we introduce a second attack constrained on the semantic structure of the face which has the half speed of the first attack with the success rate of 99.96%. Both attacks are extremely robust against the state-of-the-art defense methods with the success rate of equal or greater than 53.59%. Code is available at https://github.com/alldbi/FLM

Classification of previous works on adversarial attacks on face recognition systems. W stands for White Box attack and B stands for Black Box attacks.

In this section, we review some of the well-known adversarial attacks against face recognition models (FR) which can be categorized to three different types: (i) Adversarial instances are created based on simple noise or real world distortions.
(ii) Adversarial instances are created based on external resources like light projection or infrared.
(iii) Adversarial instances are created based on focusing on face-like features for adversarial perturbations. Despite the simplicity of adding real world distortion or optimized perturbation, they suffer two main problems : they often need large number of queries and they can easily be removed by pre-processing. On the other hand, relying on external light resources to impact the ability of FR models for accurate recognition limits the feasibility of such attacks. Our investigations show that, focusing on face-like features for adversarial perturbations has more capabilities for the next generation of adversarial attacks on FR models. Table {{table:9a447cbb-e590-472b-a252-d5f20a7d0fb0}}  summarizes the main previous adversarial attacks against FR models.


Predictive coding with Gaussian Process regression

The homunculus for proprioception: Toward learning the representation of a humanoid robot's joint space using self-organizing maps

Abstract: In primate brains, tactile and proprioceptive inputs are relayed to the somatosensory cortex which is known for somatotopic representations, or, "homunculi". Our research centers on understanding the mechanisms of the formation of these and more higher-level body representations (body schema) by using humanoid robots and neural networks to construct models. We specifically focus on how spatial representation of the body may be learned from somatosensory information in self-touch configurations. In this work, we target the representation of proprioceptive inputs, which we take to be joint angles in the robot. The inputs collected in different body postures serve as inputs to a Self-Organizing Map (SOM) with a 2D lattice on the output. With unrestricted, all-to-all connections, the map is not capable of representing the input space while preserving the topological relationships, because the intrinsic dimensionality of the body posture space is too large. Hence, we use a method we developed previously for tactile inputs (Hoffmann, Straka et al. 2018) called MRF-SOM, where the Maximum Receptive Field of output neurons is restricted so they only learn to represent specific parts of the input space. This is in line with the receptive fields of neurons in somatosensory areas representing proprioception that often respond to combination of few joints (e.g. wrist and elbow).

Adaptive robot body learning and estimation through predictive coding

Abstract: The predictive functions that permit humans to infer their body state by sensorimotor integration are critical to perform safe interaction in complex environments. These functions are adaptive and robust to non-linear actuators and noisy sensory information. This paper introduces a computational perceptual model based on predictive processing that enables any multisensory robot to learn, infer and update its body configuration when using arbitrary sensors with Gaussian additive noise. The proposed method integrates different sources of information (tactile, visual and proprioceptive) to drive the robot belief to its current body configuration. The motivation is to enable robots with the embodied perception needed for self-calibration and safe physical human-robot interaction. We formulate body learning as obtaining the forward model that encodes the sensor values depending on the body variables, and we solve it by Gaussian process regression. We model body estimation as minimizing the discrepancy between the robot body configuration belief and the observed posterior. We minimize the variational free energy using the sensory prediction errors (sensed vs expected). In order to evaluate the model we test it on a real multisensory robotic arm. We show how different sensor modalities contributions, included as additive errors, improve the refinement of the body estimation and how the system adapts itself to provide the most plausible solution even when injecting strong sensory visuo-tactile perturbations. We further analyse the reliability of the model when different sensor modalities are disabled. This provides grounded evidence about the correctness of the perceptual model and shows how the robot estimates and adjusts its body configuration just by means of sensory information.

Summary of models of body schema representations. Sensory information is coded as: visual—V, proprioception—P, tactile—T, audio—A

The body schema and the peripersonal space

As reviewed by {{cite:ee8a136}}, the representation of an agent's body can be distinguished into body schema—the representation for actionsAction-oriented representation is defined by the author as “it carries information about the bodily effector (and the bodily goal in reflective actions) that is used to guide bodily movements.”, and body image—other body-related representations for perception, conception and emotion (according to the dyadic taxonomy) (see also {{cite:40e8bb1}}). The body image can be further separated into two distinct representations, namely visuo-spatial body map—the structure description of body parts, and body semantics—the conceptual and linguistic level of body parts (according to the triadic taxonomy). However, with the perspective of the enactive approach {{cite:250cda9}}, in which the sensorimotor exploration gives rise to perceptual experiences, the distinction between the bodily action-oriented and perceptual representation is quite blurry.
For example, visual appearance and boundary of a limb would have an effect on the agent's perception of the length and position of the limb. Hence, it is reasonable to include the body structure description of the body image representation (and its sources of sensory information) when considering the body schema in action, especially from the computational perspective. Indeed, most robotics models of the so-called body schema fall into this category (see Section REF  and Table {{table:49f97fa4-0bea-4fba-9bd7-28d3e7cf0d3d}} ). Furthermore, from the technical point of view, it is difficult to model the mental level of the body image when the definition is unclear. Therefore, we will use the term “body schema” in an extended meaning including both “body schema” and “body structure description”.

Computational and robotic models of the body schema

In the following,
we present a survey of models on robotic body schema in an ascending
order of the amount of a priori knowledge provided in the learning problem. By organizing reviewed models in this order, we aim to emphasize one important aspect of autonomous systems: The ability to learn and adapt to dynamic environments. Ideally, an autonomous system should be able to learn to complete different tasks with only little provided information. A summary of the reviewed models is presented in the Table {{table:49f97fa4-0bea-4fba-9bd7-28d3e7cf0d3d}} .


Forward model learning by supervised learning

Reinforcement learning with intrinsic reward

Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Images

Abstract: We introduce Embed to Control (E2C), a method for model learning and control of non-linear dynamical systems from raw pixel images. E2C consists of a deep generative model, belonging to the family of variational autoencoders, that learns to generate image trajectories from a latent space in which the dynamics is constrained to be locally linear. Our model is derived directly from an optimal control formulation in latent space, supports long-term prediction of image sequences and exhibits strong performance on a variety of complex control problems.

Learning to Poke by Poking: Experiential Learning of Intuitive Physics

Abstract: We investigate an experiential learning paradigm for acquiring an internal model of intuitive physics. Our model is evaluated on a real-world robotic manipulation task that requires displacing objects to target locations by poking. The robot gathered over 400 hours of experience by executing more than 100K pokes on different objects. We propose a novel approach based on deep neural networks for modeling the dynamics of robot's interactions directly from images, by jointly estimating forward and inverse models of dynamics. The inverse model objective provides supervision to construct informative visual features, which the forward model can then predict and in turn regularize the feature space for the inverse model. The interplay between these two objectives creates useful, accurate models that can then be used for multi-step decision making. This formulation has the additional benefit that it is possible to learn forward models in an abstract feature space and thus alleviate the need of predicting pixels. Our experiments show that this joint modeling approach outperforms alternative methods.

Self-Supervised Exploration via Disagreement

Abstract: Efficient exploration is a long-standing problem in sensorimotor learning. Major advances have been demonstrated in noise-free, non-stochastic domains such as video games and simulation. However, most of these formulations either get stuck in environments with stochastic dynamics or are too inefficient to be scalable to real robotics setups. In this paper, we propose a formulation for exploration inspired by the work in active learning literature. Specifically, we train an ensemble of dynamics models and incentivize the agent to explore such that the disagreement of those ensembles is maximized. This allows the agent to learn skills by exploring in a self-supervised manner without any external reward. Notably, we further leverage the disagreement objective to optimize the agent's policy in a differentiable manner, without using reinforcement learning, which results in a sample-efficient exploration. We demonstrate the efficacy of this formulation across a variety of benchmark environments including stochastic-Atari, Mujoco and Unity. Finally, we implement our differentiable exploration on a real robot which learns to interact with objects completely from scratch. Project videos and code are at this https URL

Summary of active self models based on single sensory input. Sensory information is coded as: visual—V, proprioception—P, motor—M, tactile—T

In this section, we review robotics models of the active self or models owning a common feature, which is employing the predictive coding mechanism or the forward model. This focus roots from the idea that the feeling of agency can emerge in an agent with an ability to anticipate the effect of its own action (see detailed discussions in Section REF  and REF ). We first review models employing multisensory modalities (in Section REF  and Table {{table:3e64c6f6-86ba-447c-81b5-cc15a1ae021f}} ) then continue with using single sensory modality (mostly from visual input, in Section REF  and Table {{table:4687244c-5846-4607-be04-0f940fe6936a}} ). For the latter cases, they are possible to capture the dynamics of the whole system (including the agent and the interactive environment) via only single input due to the special design of input, i.e. the visual input is not taken from the first perspective viewer (as in human and other animals).


Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding

Abstract: Neural networks are both computationally intensive and memory intensive, making them difficult to deploy on embedded systems with limited hardware resources. To address this limitation, we introduce "deep compression", a three stage pipeline: pruning, trained quantization and Huffman coding, that work together to reduce the storage requirement of neural networks by 35x to 49x without affecting their accuracy. Our method first prunes the network by learning only the important connections. Next, we quantize the weights to enforce weight sharing, finally, we apply Huffman coding. After the first two steps we retrain the network to fine tune the remaining connections and the quantized centroids. Pruning, reduces the number of connections by 9x to 13x; Quantization then reduces the number of bits that represent each connection from 32 to 5. On the ImageNet dataset, our method reduced the storage required by AlexNet by 35x, from 240MB to 6.9MB, without loss of accuracy. Our method reduced the size of VGG-16 by 49x from 552MB to 11.3MB, again with no loss of accuracy. This allows fitting the model into on-chip SRAM cache rather than off-chip DRAM memory. Our compression method also facilitates the use of complex neural networks in mobile applications where application size and download bandwidth are constrained. Benchmarked on CPU, GPU and mobile GPU, compressed network has 3x to 4x layerwise speedup and 3x to 7x better energy efficiency.

Variational Dropout Sparsifies Deep Neural Networks

Abstract: We explore a recently proposed Variational Dropout technique that provided an elegant Bayesian interpretation to Gaussian Dropout. We extend Variational Dropout to the case when dropout rates are unbounded, propose a way to reduce the variance of the gradient estimator and report first experimental results with individual dropout rates per weight. Interestingly, it leads to extremely sparse solutions both in fully-connected and convolutional layers. This effect is similar to automatic relevance determination effect in empirical Bayes but has a number of advantages. We reduce the number of parameters up to 280 times on LeNet architectures and up to 68 times on VGG-like networks with a negligible decrease of accuracy.

Designing Energy-Efficient Convolutional Neural Networks Using Energy-Aware Pruning

Abstract: Deep convolutional neural networks (CNNs) are indispensable to state-of-the-art computer vision algorithms. However, they are still rarely deployed on battery-powered mobile devices, such as smartphones and wearable gadgets, where vision algorithms can enable many revolutionary real-world applications. The key limiting factor is the high energy consumption of CNN processing due to its high computational complexity. While there are many previous efforts that try to reduce the CNN model size or the amount of computation, we find that they do not necessarily result in lower energy consumption. Therefore, these targets do not serve as a good metric for energy cost estimation. To close the gap between CNN design and energy consumption optimization, we propose an energy-aware pruning algorithm for CNNs that directly uses the energy consumption of a CNN to guide the pruning process. The energy estimation methodology uses parameters extrapolated from actual hardware measurements. The proposed layer-by-layer pruning algorithm also prunes more aggressively than previously proposed pruning methods by minimizing the error in the output feature maps instead of the filter weights. For each layer, the weights are first pruned and then locally fine-tuned with a closed-form least-square solution to quickly restore the accuracy. After all layers are pruned, the entire network is globally fine-tuned using back-propagation. With the proposed pruning method, the energy consumption of AlexNet and GoogLeNet is reduced by 3.7x and 1.6x, respectively, with less than 1% top-5 accuracy loss. We also show that reducing the number of target classes in AlexNet greatly decreases the number of weights, but has a limited impact on energy consumption.

A Systematic DNN Weight Pruning Framework using Alternating Direction Method of Multipliers

Abstract: Weight pruning methods for deep neural networks (DNNs) have been investigated recently, but prior work in this area is mainly heuristic, iterative pruning, thereby lacking guarantees on the weight reduction ratio and convergence time. To mitigate these limitations, we present a systematic weight pruning framework of DNNs using the alternating direction method of multipliers (ADMM). We first formulate the weight pruning problem of DNNs as a nonconvex optimization problem with combinatorial constraints specifying the sparsity requirements, and then adopt the ADMM framework for systematic weight pruning. By using ADMM, the original nonconvex optimization problem is decomposed into two subproblems that are solved iteratively. One of these subproblems can be solved using stochastic gradient descent, the other can be solved analytically. Besides, our method achieves a fast convergence rate. The weight pruning results are very promising and consistently outperform the prior work. On the LeNet-5 model for the MNIST data set, we achieve 71.2 times weight reduction without accuracy loss. On the AlexNet model for the ImageNet data set, we achieve 21 times weight reduction without accuracy loss. When we focus on the convolutional layer pruning for computation reductions, we can reduce the total computation by five times compared with the prior work (achieving a total of 13.4 times weight reduction in convolutional layers). Our models and codes are released at https://github.com/KaiqiZhang/admm-pruning

NeST: A Neural Network Synthesis Tool Based on a Grow-and-Prune Paradigm

Abstract: Deep neural networks (DNNs) have begun to have a pervasive impact on various applications of machine learning. However, the problem of finding an optimal DNN architecture for large applications is challenging. Common approaches go for deeper and larger DNN architectures but may incur substantial redundancy. To address these problems, we introduce a network growth algorithm that complements network pruning to learn both weights and compact DNN architectures during training. We propose a DNN synthesis tool (NeST) that combines both methods to automate the generation of compact and accurate DNNs. NeST starts with a randomly initialized sparse network called the seed architecture. It iteratively tunes the architecture with gradient-based growth and magnitude-based pruning of neurons and connections. Our experimental results show that NeST yields accurate, yet very compact DNNs, with a wide range of seed architecture selection. For the LeNet-300-100 (LeNet-5) architecture, we reduce network parameters by $70.2\times$70.2× ($74.3\times$74.3×) and floating-point operations (FLOPs) by $79.4\times$79.4× ($43.7\times$43.7×). For the AlexNet, VGG-16, and ResNet-50 architectures, we reduce network parameters (FLOPs) by $15.7\times$15.7× ($4.6\times$4.6×), $33.2\times$33.2× ($8.9\times$8.9×), and $4.1\times$4.1× ($2.1\times$2.1×) respectively. NeST's grow-and-prune paradigm delivers significant additional parameter and FLOPs reduction relative to pruning-only methods.

Comparison of Non-structure pruning methods

Discussion:
Table {{table:a373a283-d7af-47c1-b1cb-6844f951ed0c}}  and {{table:3f6ecf43-d5e8-4992-8db4-d7884eadaaf3}}  summarize methods discussed in this section. Pruning is the well-studied topic in model compression and many methods have been proposed to advance the network pruning {{cite:b767aeb}}. These pruning methods show promising results, capable of compressing a DNN model, in some cases more than 40x, while only degrading the prediction accuracy slightly {{cite:67376cf}}.
To effectively adopt model pruning for edge intelligence systems, the existing works suffer from two flaws: 1)
the theoretical foundation behind pruning lacks, and thus there is a debate whether we should prune a complex model or directly train a compact model for resource constrained hardware platforms. Some recent research {{cite:8473f3a}}{{cite:008d43d}}{{cite:a1e4003}}{{cite:3a2542c}} strives to empirically or theoretically find an answer for this question.
Since pruning is a time-consuming procedure involving large model training and iterative pruning-retraining procedure, a theoretical foundation or proof may drastically change the way we use model pruning to design edge DNN models;
2) Almost all of pruning methods are hardware-agnostic and depend on hardware-independent metrics, like MACs and FLOPs. Since different hardware architectures demonstrate different degrees of parallelism, the state-of-the-art methods may unnecessarily prune models without performance improvement (e.g., latency reduction) while reducing the capacity of DNN models which is proven to have a significant impact on model's accuracy {{cite:3ac0327}}. Therefore, we need to devise pruning method with hardware-awareness such that the model can be tailored for various hardware.


Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Research

Abstract: The purpose of this technical report is two-fold. First of all, it introduces a suite of challenging continuous control tasks (integrated with OpenAI Gym) based on currently existing robotics hardware. The tasks include pushing, sliding and pick & place with a Fetch robotic arm as well as in-hand object manipulation with a Shadow Dexterous Hand. All tasks have sparse binary rewards and follow a Multi-Goal Reinforcement Learning (RL) framework in which an agent is told what to do using an additional input. The second part of the paper presents a set of concrete research ideas for improving RL algorithms, most of which are related to Multi-Goal RL and Hindsight Experience Replay.

Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations

Abstract: Dexterous multi-fingered hands are extremely versatile and provide a generic way to perform a multitude of tasks in human-centric environments. However, effectively controlling them remains challenging due to their high dimensionality and large number of potential contacts. Deep reinforcement learning (DRL) provides a model-agnostic approach to control complex dynamical systems, but has not been shown to scale to high-dimensional dexterous manipulation. Furthermore, deployment of DRL on physical systems remains challenging due to sample inefficiency. Consequently, the success of DRL in robotics has thus far been limited to simpler manipulators and tasks. In this work, we show that model-free DRL can effectively scale up to complex manipulation tasks with a high-dimensional 24-DoF hand, and solve them from scratch in simulated experiments. Furthermore, with the use of a small number of human demonstrations, the sample complexity can be significantly reduced, which enables learning with sample sizes equivalent to a few hours of robot experience. The use of demonstrations result in policies that exhibit very natural movements and, surprisingly, are also substantially more robust.

OpenAI Gym

Abstract: OpenAI Gym is a toolkit for reinforcement learning research. It includes a growing collection of benchmark problems that expose a common interface, and a website where people can share their results and compare the performance of algorithms. This whitepaper discusses the components of OpenAI Gym and the design decisions that went into the software.

OpenAI Gym

OpenAI Gym

OpenAI Gym

List of environments used in this work to benchmark.

In order to standardize evaluation of these algorithms, we have compiled a representative set of environments, which range from simple sanity checks to difficult high-dimensional parameter estimation problems. A complete list of characterized environments can be seen in Table {{table:492c2083-7847-488e-b1c3-3cd2d595c0de}}  and visualized in Figure {{figure:0b31cee2-007d-4c47-8131-3861ee19d08e}} .

To compare performance across environments and algorithms, we introduce the SiPE plot, inspired by bsuite {{cite:f378576}} and implemented with Matplotlib {{cite:1765e0e}}, shown in Figure REF . Each algorithm is run against all of the benchmark environments in Table {{table:492c2083-7847-488e-b1c3-3cd2d595c0de}} , and the score for each radial point represents the accuracy on that particular environment. Scores are calculated by averaging across all of the parameters.


High probability to achieve the same performance as DPC

High passive array gains provided by large-size RISs

Extended coverage and low-complexity beamformer design

Complicated joint optimal decoding order and beamformer design

Non-trivial CSI acquisition and limited coverage

Potentially high overhead for information exchange

Power Efficient IRS-Assisted NOMA

Abstract: In this paper, we propose a downlink multiple-input single-output (MISO) transmission scheme, which is assisted by an intelligent reflecting surface (IRS) consisting of a large number of passive reflecting elements. In the literature, it has been proved that nonorthogonal multiple access (NOMA) can achieve the capacity region when the channels are quasi-degraded. However, in a conventional communication scenario, it is difficult to guarantee the quasi-degradation, because the channels are determined by the propagation environments and cannot be reconfigured. To overcome this difficulty, we focus on an IRS-assisted MISO NOMA system, where the wireless channels can be effectively tuned. We optimize the beamforming vectors and the IRS phase shift matrix for minimizing transmission power. Furthermore, we propose an improved quasi-degradation condition by using IRS, which can ensure that NOMA achieves the capacity region with high possibility. For a comparison, we study zero-forcing beamforming (ZFBF) as well, where the beamforming vectors and the IRS phase shift matrix are also jointly optimized. Comparing NOMA with ZFBF, it is shown that, with the same IRS phase shift matrix and the improved quasi-degradation condition, NOMA always outperforms ZFBF. At the same time, we identify the condition under which ZFBF outperforms NOMA, which motivates the proposed hybrid NOMA transmission. Simulation results show that the proposed IRS-assisted MISO system outperforms the MISO case without IRS, and the hybrid NOMA transmission scheme always achieves better performance than orthogonal multiple access.

Intelligent Reflecting Surface Enhanced Millimeter-Wave NOMA Systems

Abstract: In this letter, a downlink intelligent reflecting surface (IRS) enhanced millimeter-wave (mmWave) non-orthogonal multiple access (NOMA) system is considered. A joint optimization problem over active beamforming, passive beamforming and power allocation is formulated. Due to the highly coupled variables, the formulated optimization problem is non-convex. To solve this problem, an alternative optimization and successive convex approximation based iterative algorithm is proposed. Numerical results illustrate that: 1) the proposed scheme offers significant sum-rate gains, which confirms the effectiveness of introducing IRS for mmWave-NOMA systems; 2) the proposed algorithm with discrete phase shifts can achieve close performance to that of continuous phase shifts.

A Simple Design of IRS-NOMA Transmission

Abstract: This letter proposes a simple design of intelligent reflecting surface (IRS) assisted non-orthogonal multiple access (NOMA) downlink transmission. In particular, conventional spatial division multiple access (SDMA) is used first at the base station to generate orthogonal beams by using the spatial directions of the near users’ channels. Then, IRS-assisted NOMA is used to ensure that additional cell-edge users can also be served on these beams by aligning the cell-edge users’ effective channel vectors with the predetermined spatial directions. Both analytical and simulation results are provided to demonstrate the performance of the proposed IRS-NOMA scheme and also study the impact of hardware impairments on IRS-NOMA.

blackSummary of RIS-aided Multiple-antenna NOMA Transmission Strategies.

blackIn Table {{table:96a90058-19a7-44c6-a57d-8d0e6a65de74}} , we summarize the respective advantages and disadvantages of the proposed RIS-aided multiple-antenna NOMA transmission strategies, where “BB-NOMA” and “CB-NOMA” refer to beamformer-based NOMA and cluster-based NOMA, respectively.


SemEval-2020 Task 12: Multilingual Offensive Language Identification in Social Media (OffensEval 2020)

Abstract: We present the results and main findings of SemEval-2020 Task 12 on Multilingual Offensive Language Identification in Social Media (OffensEval 2020). The task involves three subtasks corresponding to the hierarchical taxonomy of the OLID schema (Zampieri et al., 2019a) from OffensEval 2019. The task featured five languages: English, Arabic, Danish, Greek, and Turkish for Subtask A. In addition, English also featured Subtasks B and C. OffensEval 2020 was one of the most popular tasks at SemEval-2020 attracting a large number of participants across all subtasks and also across all languages. A total of 528 teams signed up to participate in the task, 145 teams submitted systems during the evaluation period, and 70 submitted system description papers.

Ex Machina: Personal Attacks Seen at Scale

Abstract: The damage personal attacks cause to online discourse motivates many platforms to try to curb the phenomenon. However, understanding the prevalence and impact of personal attacks in online platforms at scale remains surprisingly difficult. The contribution of this paper is to develop and illustrate a method that combines crowdsourcing and machine learning to analyze personal attacks at scale. We show an evaluation method for a classifier in terms of the aggregated number of crowd-workers it can approximate. We apply our methodology to English Wikipedia, generating a corpus of over 100k high quality human-labeled comments and 63M machine-labeled ones from a classifier that is as good as the aggregate of 3 crowd-workers, as measured by the area under the ROC curve and Spearman correlation. Using this corpus of machine-labeled scores, our methodology allows us to explore some of the open questions about the nature of online personal attacks. This reveals that the majority of personal attacks on Wikipedia are not the result of a few malicious users, nor primarily the consequence of allowing anonymous contributions from unregistered users.

BEEP! Korean Corpus of Online News Comments for Toxic Speech Detection

Abstract: Toxic comments in online platforms are an unavoidable social issue under the cloak of anonymity. Hate speech detection has been actively done for languages such as English, German, or Italian, where manually labeled corpus has been released. In this work, we first present 9.4K manually labeled entertainment news comments for identifying Korean toxic speech, collected from a widely used online news platform in Korea. The comments are annotated regarding social bias and hate speech since both aspects are correlated. The inter-annotator agreement Krippendorff's alpha score is 0.492 and 0.496, respectively. We provide benchmarks using CharCNN, BiLSTM, and BERT, where BERT achieves the highest score on all tasks. The models generally display better performance on bias identification, since the hate speech detection is a more subjective issue. Additionally, when BERT is trained with bias label for hate speech detection, the prediction score increases, implying that bias and hate are intertwined. We make our dataset publicly available and open competitions with the corpus and benchmarks.

Community evolution in retweet networks

Abstract: Communities in social networks often reflect close social ties between their members and their evolution through time. We propose an approach that tracks two aspects of community evolution in retweet networks: flow of the members in, out and between the communities, and their influence. We start with high resolution time windows, and then select several timepoints which exhibit large differences between the communities. For community detection, we propose a two-stage approach. In the first stage, we apply an enhanced Louvain algorithm, called Ensemble Louvain, to find stable communities. In the second stage, we form influence links between these communities, and identify linked super-communities. For the detected communities, we compute internal and external influence, and for individual users, the retweet h-index influence. We apply the proposed approach to three years of Twitter data of all Slovenian tweets. The analysis shows that the Slovenian tweetosphere is dominated by politics, that the left-leaning communities are larger, but that the right-leaning communities and users exhibit significantly higher impact. An interesting observation is that retweet networks change relatively gradually, despite such events as the emergence of the Covid-19 pandemic or the change of government.

Comment filtering datasets and their properties: number of examples, size of the split and class distribution inside subsets.

While comparable datasets exist across different languages for the NER and DP task, no such standard datasets exist for the CF task. For that reason, in our experiments on CF, we select languages for which adequate datasets exist, i.e. large, of sufficient quality, and reasonably balanced across classes. We provide a summary of the used datasets in Table {{table:25433d52-2801-4879-8ca7-cad646be4894}} .

Experimental settings for comment filtering

We evaluate the models using the macro {{formula:cea63947-e622-4468-937c-63c4a4c92731}}  score, computed as the unweighted mean of the {{formula:444a6312-a1bf-43d1-a7c2-20d7aa8ce77b}}  scores for each label. We use fixed training, validation and test sets, described in Table {{table:25433d52-2801-4879-8ca7-cad646be4894}} . As we have observed noticeable variance in the preliminary experiments, we report the mean metrics over five runs for each setting, along with the standard deviation.