Reevaluating Old Stellar Populations

Abstract: Determining the properties of old stellar populations (those with age >1 Gyr) has long involved the comparison of their integrated light, either in the form of photometry or spectroscopic indexes, with empirical or synthetic templates. Here we reevaluate the properties of old stellar populations using a new set of stellar population synthesis models, designed to incorporate the effects of binary stellar evolution pathways as a function of stellar mass and age. We find that single-aged stellar population models incorporating binary stars, as well as new stellar evolution and atmosphere models, can reproduce the colours and spectral indices observed in both globular clusters and quiescent galaxies. The best fitting model populations are often younger than those derived from older spectral synthesis models, and may also lie at slightly higher metallicities.

COSMIC Variance in Binary Population Synthesis

Abstract: The formation and evolution of binary stars are critical components of several fields in astronomy. The most numerous sources for gravitational wave observatories are inspiraling or merging compact binaries, while binary stars are present in nearly every electromagnetic survey regardless of the target population. Simulations of large binary populations serve to both predict and inform observations of electromagnetic and gravitational wave sources. Binary population synthesis is a tool that balances physical modeling with simulation speed to produce large binary populations on timescales of days. We present a community-developed binary population synthesis suite, COSMIC, which is designed to simulate compact-object binary populations and their progenitors. As a proof of concept, we simulate the Galactic population of compact binaries and their gravitational wave signals observable by the Laser Interferometer Space Antenna.

The impact of mass-transfer physics on the observable properties of field binary black hole populations

Abstract: We study the impact of mass-transfer physics on the observable properties of binary black hole populations formed through isolated binary evolution. We investigate the impact of mass-accretion efficiency onto compact objects and common-envelope efficiency on the observed distributions of $\chi_{eff}$, $M_{chirp}$ and $q$. We find that low common envelope efficiency translates to tighter orbits post common envelope and therefore more tidally spun up second-born black holes. However, these systems have short merger timescales and are only marginally detectable by current gravitational-waves detectors as they form and merge at high redshifts ($z\sim 2$), outside current detector horizons. Assuming Eddington-limited accretion efficiency and that the first-born black hole is formed with a negligible spin, we find that all non-zero $\chi_{eff}$ systems in the detectable population can come only from the common envelope channel as the stable mass-transfer channel cannot shrink the orbits enough for efficient tidal spin-up to take place. We find the local rate density ($z\simeq 0.01$) for the common envelope channel is in the range $\sim 17-113~Gpc^{-3}yr^{-1}$ considering a range of $\alpha_{CE} \in [0.2,5.0]$ while for the stable mass transfer channel the rate density is $\sim 25~Gpc^{-3}yr^{-1}$. The latter drops by two orders of magnitude if the mass accretion onto the black hole is not Eddington limited because conservative mass transfer does not shrink the orbit as efficiently as non-conservative mass transfer does. Finally, using GWTC-2 events, we constrain the lower bound of branching fraction from other formation channels in the detected population to be $\sim 0.2$. Assuming all remaining events to be formed through either stable mass transfer or common envelope channels, we find moderate to strong evidence in favour of models with inefficient common envelopes.

The mass spectrum of compact remnants from the PARSEC stellar evolution tracks

Abstract: The mass spectrum of stellar-mass black holes (BHs) is highly uncertain. Dynamical mass measurements are available only for few ($\sim{}10$) BHs in X-ray binaries, while theoretical models strongly depend on the hydrodynamics of supernova (SN) explosions and on the evolution of massive stars. In this paper, we present and discuss the mass spectrum of compact remnants that we obtained with SEVN, a new public population-synthesis code, which couples the PARSEC stellar evolution tracks with up-to-date recipes for SN explosion (depending on the Carbon-Oxygen mass of the progenitor, on the compactness of the stellar core at pre-SN stage, and on a recent two-parameter criterion based on the dimensionless entropy per nucleon at pre-SN stage). SEVN can be used both as a stand-alone code and in combination with direct-summation N-body codes (Starlab, HiGPUs). The PARSEC stellar evolution tracks currently implemented in SEVN predict significantly larger values of the Carbon-Oxygen core mass with respect to previous models. For most of the SN recipes we adopt, this implies substantially larger BH masses at low metallicity ($\leq{}2\times{}10^{-3}$), than other population-synthesis codes. The maximum BH mass found with SEVN is $\sim{}$25, 60 and 130 M$_{\odot}$ at metallicity $Z =2 \times{} 10^{-2}$ , $2 \times{}10^{-3}$ and $2\times{} 10^{-4}$ , respectively. Mass loss by stellar winds plays a major role in determining the mass of BHs for very massive stars ($\geq{}90$ M$_\odot{}$), while the remnant mass spectrum depends mostly on the adopted SN recipe for lower progenitor masses. We discuss the implications of our results for the transition between NS and BH mass, and for the expected number of massive BHs (with mass $>25$ M$_\odot{}$) as a function of metallicity.

Population synthesis codes used by the community at the time of writing. Binary_c, COSMIC, MOBSE are based on the BSE code {{cite:856f804}}.

Population synthesis codes are crucial for LISA science, both in order to make forecasts for source rates, but also to develop data analysis pipelines. Indeed, ongoing work on building fast and reliable waveforms relies on the knowledge of the expected properties of the sources (masses, spins, eccentricities) and the accuracies required to detect them and measure these properties. The codes currently in use by the LISA community are listed in Table {{table:1dfc2113-3b17-4761-8178-a622186a8214}} .
Further development of these codes, in particular the inclusion of additional physical processes as well as cross-checks between the codes will help to prepare for LISA observations.


MuMiN: A Large-Scale Multilingual Multimodal Fact-Checked Misinformation Social Network Dataset

Abstract: Misinformation is becoming increasingly prevalent on social media and in news articles. It has become so widespread that we require algorithmic assistance utilising machine learning to detect such content. Training these machine learning models require datasets of sufficient scale, diversity and quality. However, datasets in the field of automatic misinformation detection are predominantly monolingual, include a limited amount of modalities and are not of sufficient scale and quality. Addressing this, we develop a data collection and linking system (MuMiN-trawl), to build a public misinformation graph dataset (MuMiN), containing rich social media data (tweets, replies, users, images, articles, hashtags) spanning 21 million tweets belonging to 26 thousand Twitter threads, each of which have been semantically linked to 13 thousand fact-checked claims across dozens of topics, events and domains, in 41 different languages, spanning more than a decade. The dataset is made available as a heterogeneous graph via a Python package (mumin). We provide baseline results for two node classification tasks related to the veracity of a claim involving social media, and demonstrate that these are challenging tasks, with the highest macro-average F1-score being 62.55% and 61.45% for the two tasks, respectively. The MuMiN ecosystem is available at https://mumin-dataset.github.io/, including the data, documentation, tutorials and leaderboards.

Images and Misinformation in Political Groups: Evidence from WhatsApp in India

Abstract: WhatsApp is a key medium for the spread of news and rumors, often shared as images. We study a large collection of politically-oriented WhatsApp groups in India, focusing on the period leading up to the 2019 Indian national elections. By labeling samples of random and popular images, we find that around 13% of shared images are known misinformation and most fall into three types of images. Machine learning methods can be used to predict whether a viral image is misinformation, but are brittle to shifts in content over time.

DisinfoMeme: A Multimodal Dataset for Detecting Meme Intentionally Spreading Out Disinformation

Abstract: Disinformation has become a serious problem on social media. In particular, given their short format, visual attraction, and humorous nature, memes have a significant advantage in dissemination among online communities, making them an effective vehicle for the spread of disinformation. We present DisinfoMeme to help detect disinformation memes. The dataset contains memes mined from Reddit covering three current topics: the COVID-19 pandemic, the Black Lives Matter movement, and veganism/vegetarianism. The dataset poses multiple unique challenges: limited data and label imbalance, reliance on external knowledge, multimodal reasoning, layout dependency, and noise from OCR. We test multiple widely-used unimodal and multimodal models on this dataset. The experiments show that the room for improvement is still huge for current models.

COVID-VTS: Fact Extraction and Verification on Short Video Platforms

Abstract: We introduce a new benchmark, COVID-VTS, for fact-checking multi-modal information involving short-duration videos with COVID19- focused information from both the real world and machine generation. We propose, TwtrDetective, an effective model incorporating cross-media consistency checking to detect token-level malicious tampering in different modalities, and generate explanations. Due to the scarcity of training data, we also develop an efficient and scalable approach to automatically generate misleading video posts by event manipulation or adversarial matching. We investigate several state-of-the-art models and demonstrate the superiority of TwtrDetective.

Check Mate: Prioritizing User Generated Multi-Media Content for Fact-Checking

Abstract: Volume of content and misinformation on social media is rapidly increasing. There is a need for systems that can support fact checkers by prioritizing content that needs to be fact checked. Prior research on prioritizing content for fact-checking has focused on news media articles, predominantly in English language. Increasingly, misinformation is found in user-generated content. In this paper we present a novel dataset that can be used to prioritize check-worthy posts from multi-media content in Hindi. It is unique in its 1) focus on user generated content, 2) language and 3) accommodation of multi-modality in social media posts. In addition, we also provide metadata for each post such as number of shares and likes of the post on ShareChat, a popular Indian social media platform, that allows for correlative analysis around virality and misinformation. The data is accessible on Zenodo (https://zenodo.org/record/4032629) under Creative Commons Attribution License (CC BY 4.0).

Datasets for claim detection. Img, Txt, Vid, Aud, and Meta denote image, text, video, audio, and metadata, respectively. Output indicates the number classification labels. Mul indicates that the input has multiple languages.

Input.
Typical inputs to claim detection are unimodal, including image {{cite:1c88045}}, {{cite:116d2d9}}, audio {{cite:c5ca3f7}}, and video {{cite:d91d05f}}, {{cite:2da83c9}}, which are collected from social media platforms such as WhatsApp and TikTok (see Table {{table:2e7d0111-ccf1-49f3-a395-c3b57b115473}} ).
The written or spoken claim is extracted from the input at this stage before fact-checking it.


Performance Measures and a Data Set for Multi-Target, Multi-Camera Tracking

Abstract: To help accelerate progress in multi-target, multi-camera tracking systems, we present (i) a new pair of precision-recall measures of performance that treats errors of all types uniformly and emphasizes correct identification over sources of error; (ii) the largest fully-annotated and calibrated data set to date with more than 2 million frames of 1080p, 60fps video taken by 8 cameras observing more than 2,700 identities over 85 minutes; and (iii) a reference software system as a comparison baseline. We show that (i) our measures properly account for bottom-line identity match performance in the multi-camera setting; (ii) our data set poses realistic challenges to current trackers; and (iii) the performance of our system is comparable to the state of the art.

Person Transfer GAN to Bridge Domain Gap for Person Re-identification

Abstract: Although the performance of person Re-Identification (ReID) has been significantly boosted, many challenging issues in real scenarios have not been fully investigated, e.g., the complex scenes and lighting variations, viewpoint and pose changes, and the large number of identities in a camera network. To facilitate the research towards conquering those issues, this paper contributes a new dataset called MSMT171 with many important features, e.g., 1) the raw videos are taken by an 15-camera network deployed in both indoor and outdoor scenes, 2) the videos cover a long period of time and present complex lighting variations, and 3) it contains currently the largest number of annotated identities, i.e., 4,101 identities and 126,441 bounding boxes. We also observe that, domain gap commonly exists between datasets, which essentially causes severe performance drop when training and testing on different datasets. This results in that available training data cannot be effectively leveraged for new testing domains. To relieve the expensive costs of annotating new training samples, we propose a Person Transfer Generative Adversarial Network (PTGAN) to bridge the domain gap. Comprehensive experiments show that the domain gap could be substantially narrowed-down by the PTGAN.

Statistics of person re-ID datasets.

Datasets and Settings
Seven popular re-ID benchmarks are used, including Market1501 {{cite:df22ba3}}, CUHK03 {{cite:d39e918}}, DukeMTMC-reID (Duke) {{cite:a03b56d}}, {{cite:4c5ad02}}, MSMT17 {{cite:560b4e1}},Throughout this paper, we use the v1 version for MSMT17. VIPeR {{cite:7ac8da9}}, GRID {{cite:ebda2bd}} and CUHK01 {{cite:ba5641f}}. The overall dataset statistics are detailed in Table {{table:ca73cd08-2d36-4136-ace5-70b74ebfef70}} . The first four are typically considered as big re-ID datasets—even though their sizes are fairly moderate (around 30k training images for the largest dataset MSMT17). The rest three datasets are generally too small to train deep models without proper pre-training {{cite:92e956a}}, {{cite:6c3de72}}. For CUHK03, we use the 767/700 split {{cite:bca8e9e}} with the detected images. For VIPeR, GRID and CUHK01 (485/486 split {{cite:b0ac088}}), we follow {{cite:7a07ee3}}, {{cite:eac4e6e}}, {{cite:6c3de72}}, {{cite:92e956a}}, {{cite:7f6d38f}} to perform pre-training on large re-ID datasets and then fine-tune the model on the target dataset, where the results are averaged over 10 random splits. For evaluation metrics, we use cumulative matching characteristics (CMC) rank accuracy and mean average precision (mAP). The performance is reported in percentage.


Quantum Annealer; Universal Gate-Based Quantum Device

Construction of model Hamiltonians for adiabatic quantum computation and its application to finding low-energy conformations of lattice protein models

Abstract: In this report, we explore the use of a quantum optimization algorithm for obtaining low energy conformations of protein models. We discuss mappings between protein models and optimization variables, which are in turn mapped to a system of coupled quantum bits. General strategies are given for constructing Hamiltonians to be used to solve optimization problems of physical/chemical/biological interest via quantum computation by adiabatic evolution. As an example, we implement the Hamiltonian corresponding to the Hydrophobic-Polar (HP) model for protein folding. Furthermore, we present an approach to reduce the resulting Hamiltonian to two-body terms gearing towards an experimental realization.

Genome assembly using quantum and quantum-inspired annealing

Abstract: Recent advances in DNA sequencing open prospects to make whole-genome analysis rapid and reliable, which is promising for various applications including personalized medicine. However, existing techniques for {\it de novo} genome assembly, which is used for the analysis of genomic rearrangements, chromosome phasing, and reconstructing genomes without a reference, require solving tasks of high computational complexity. Here we demonstrate a method for solving genome assembly tasks with the use of quantum and quantum-inspired optimization techniques. Within this method, we present experimental results on genome assembly using quantum annealers both for simulated data and the $\phi$X 174 bacteriophage. Our results pave a way for an increase in the efficiency of solving bioinformatics problems with the use of quantum computing and, in particular, quantum annealing. We expect that the new generation of quantum annealing devices would outperform existing techniques for {\it de novo} genome assembly. To the best of our knowledge, this is the first experimental study of de novo genome assembly problems both for real and synthetic data on quantum annealing devices and quantum-inspired techniques.

Fast quantum search algorithms in protein sequence comparisons: quantum bioinformatics.

Abstract: Quantum search algorithms are considered in the context of protein sequence comparison in bioinformatics. Given a sample protein sequence of length m (i.e., m residues), the problem considered is to find an optimal match in a large database containing N residues. Initially, Grover's quantum search algorithm is applied to a simple illustrative case-namely, where the database forms a complete set of states over the 2(m) basis states of a m qubit register, and thus is known to contain the exact sequence of interest. This example demonstrates explicitly the typical O(square root of [N]) speedup on the classical O(N) requirements. An algorithm is then presented for the (more realistic) case where the database may contain repeat sequences, and may not necessarily contain an exact match to the sample sequence. In terms of minimizing the Hamming distance between the sample sequence and the database subsequences the algorithm finds an optimal alignment, in O(square root of [N]) steps, by employing an extension of Grover's algorithm, due to Boyer et al. for the case when the number of matches is not a priori known.

Quantum pattern matching fast on average

Abstract: The $d$-dimensional pattern matching problem is to find an occurrence of a pattern of length $m \times \dots \times m$ within a text of length $n \times \dots \times n$, with $n \ge m$. This task models various problems in text and image processing, among other application areas. This work describes a quantum algorithm which solves the pattern matching problem for random patterns and texts in time $\widetilde{O}((n/m)^{d/2} 2^{O(d^{3/2}\sqrt{\log m})})$. For large $m$ this is super-polynomially faster than the best possible classical algorithm, which requires time $\widetilde{\Omega}( (n/m)^d + n^{d/2} )$. The algorithm is based on the use of a quantum subroutine for finding hidden shifts in $d$ dimensions, which is a variant of algorithms proposed by Kuperberg.

Quantum Simulation of Phylogenetic Trees

Abstract: Quantum simulations constructing probability tensors of biological multi-taxa in phylogenetic trees are proposed, in terms of positive trace preserving maps, describing evolving systems of quantum walks with multiple walkers. Basic phylogenetic models applying on trees of various topologies are simulated following appropriate decoherent quantum circuits. Quantum simulations of statistical inference for aligned sequences of biological characters are provided in terms of a quantum pruning map operating on likelihood operator observables, utilizing state-observable duality and measurement theory.

Quantum annealing versus classical machine learning applied to a simplified computational biology problem

Abstract: Transcription factors regulate gene expression, but how these proteins recognize and specifically bind to their DNA targets is still debated. Machine learning models are effective means to reveal interaction mechanisms. Here we studied the ability of a quantum machine learning approach to predict binding specificity. Using simplified datasets of a small number of DNA sequences derived from actual binding affinity experiments, we trained a commercially available quantum annealer to classify and rank transcription factor binding. The results were compared to state-of-the-art classical approaches for the same simplified datasets, including simulated annealing, simulated quantum annealing, multiple linear regression, LASSO, and extreme gradient boosting. Despite technological limitations, we find a slight advantage in classification performance and nearly equal ranking performance using the quantum annealer for these fairly small training data sets. Thus, we propose that quantum annealing might be an effective method to implement machine learning for certain computational biology problems.

Selected quantum proof of principles relevant to biology and medicine. A variety of quantum approaches to addressing computational problems in biology and medicine exist, some of which have been experimentally demonstrated. Expected empirical advantages vary greatly. Among the most promising near term applications are ones that leverage quantum simulation and quantum machine learning techniques, such as quantum neural networks.

Future prospects in biology and medicine

In this section we describe a broad variety of potential applications for quantum algorithms. Our aim is to highlight the breadth of both existing quantum algorithms and the types of problems in biology and medicine that they may address. We leverage the quantum advantage framework described in Section  and note when a specific application may admit an empirical quantum advantage in the near- or medium-term (summarized in Table  {{table:f46c8ccb-7adf-4f56-b57a-00ec365d19f4}}). While some of the applications described are not expected to be feasible in the near term – indeed, in some cases, even an FTQC may not be the most appropriate tool for the target problem – it is our intent for the breadth of potential applications and research directions covered to be valuable to an interdisciplinary audience. As such, when possible, we have sought to provide i) quantum scientists with relevant details and references to develop targeted quantum algorithms for applications in biology and medicine and ii) domain computationalists with information on quantum algorithms relevant to applications in the biology and medicine and their prospects for operational quantum advantages in the near- or medium-term.

Prospects for bioinformatics. A small number of quantum algorithms for problems in bioinformatics have been proposed (Table  {{table:f46c8ccb-7adf-4f56-b57a-00ec365d19f4}}). These include theoretical algorithms developed for FTQC devices that target NP-hard problems, such as sequence alignment {{cite:462cfd1}}, {{cite:1b43929}}, {{cite:c43f186}} and the inference of phylogenetic trees {{cite:acee570}}, which leverage amplitude amplification and quantum walks {{cite:b55c84b}}. To be made practical, these theoretical quantum algorithms are expected to require both significant refinement and effort in translation. In the near term, these refinements could include i) recasting them for NISQ devices using the VQA, QAOA, or QA frameworks and ii) integrating greater biological context. Already, examples of this type of work exist for de novo assembly {{cite:b593c4e}}, {{cite:fddec33}}, sequence alignment {{cite:95dcbe6}}, and the inference of biological networks {{cite:562ada1}}, {{cite:a7ff242}}. Over the long term, operational advantages may be pursued by optimizing near term approaches and integrating fast quantum algorithm subroutines where possible. Known quantum algorithms that may be relevant to this work include ones for backtracking {{cite:9b820bf}}, dynamic programming {{cite:a333c01}}, {{cite:4bec599}}, operating on strings {{cite:e496bfd}}, {{cite:1552107}}, {{cite:5c0fec9}}, and differential equations {{cite:eac41e5}}, {{cite:be59c54}}, {{cite:168e745}}, {{cite:2b8b6b3}}, {{cite:124715a}}.


DAGs with NO TEARS: Continuous Optimization for Structure Learning

Abstract: Estimating the structure of directed acyclic graphs (DAGs, also known as Bayesian networks) is a challenging problem since the search space of DAGs is combinatorial and scales superexponentially with the number of nodes. Existing approaches rely on various local heuristics for enforcing the acyclicity constraint. In this paper, we introduce a fundamentally different strategy: we formulate the structure learning problem as a purely continuous optimization problem over real matrices that avoids this combinatorial constraint entirely. This is achieved by a novel characterization of acyclicity that is not only smooth but also exact. The resulting problem can be efficiently solved by standard numerical algorithms, which also makes implementation effortless. The proposed method outperforms existing ones, without imposing any structural assumptions on the graph such as bounded treewidth or in-degree.

Graphite: Iterative Generative Modeling of Graphs

Abstract: Graphs are a fundamental abstraction for modeling relational data. However, graphs are discrete and combinatorial in nature, and learning representations suitable for machine learning tasks poses statistical and computational challenges. In this work, we propose Graphite, an algorithmic framework for unsupervised learning of representations over nodes in large graphs using deep latent variable generative models. Our model parameterizes variational autoencoders (VAE) with graph neural networks, and uses a novel iterative graph refinement strategy inspired by low-rank approximations for decoding. On a wide variety of synthetic and benchmark datasets, Graphite outperforms competing approaches for the tasks of density estimation, link prediction, and node classification. Finally, we derive a theoretical connection between message passing in graph neural networks and mean-field variational inference.

DAG-GNN: DAG Structure Learning with Graph Neural Networks

Abstract: Learning a faithful directed acyclic graph (DAG) from samples of a joint distribution is a challenging combinatorial problem, owing to the intractable search space superexponential in the number of graph nodes. A recent breakthrough formulates the problem as a continuous optimization with a structural constraint that ensures acyclicity (Zheng et al., 2018). The authors apply the approach to the linear structural equation model (SEM) and the least-squares loss function that are statistically well justified but nevertheless limited. Motivated by the widespread success of deep learning that is capable of capturing complex nonlinear mappings, in this work we propose a deep generative model and apply a variant of the structural constraint to learn the DAG. At the heart of the generative model is a variational autoencoder parameterized by a novel graph neural network architecture, which we coin DAG-GNN. In addition to the richer capacity, an advantage of the proposed model is that it naturally handles discrete variables as well as vector-valued ones. We demonstrate that on synthetic data sets, the proposed method learns more accurate graphs for nonlinearly generated samples; and on benchmark data sets with discrete variables, the learned graphs are reasonably close to the global optima. The code is available at \url{this https URL}.

A Graph Autoencoder Approach to Causal Structure Learning

Abstract: Causal structure learning has been a challenging task in the past decades and several mainstream approaches such as constraint- and score-based methods have been studied with theoretical guarantees. Recently, a new approach has transformed the combinatorial structure learning problem into a continuous one and then solved it using gradient-based optimization methods. Following the recent state-of-the-arts, we propose a new gradient-based method to learn causal structures from observational data. The proposed method generalizes the recent gradient-based methods to a graph autoencoder framework that allows nonlinear structural equation models and is easily applicable to vector-valued variables. We demonstrate that on synthetic datasets, our proposed method outperforms other gradient-based methods significantly, especially on large causal graphs. We further investigate the scalability and efficiency of our method, and observe a near linear training time when scaling up the graph size.

Scaling structural learning with NO-BEARS to infer causal transcriptome networks

Abstract: Constructing gene regulatory networks is a critical step in revealing disease mechanisms from transcriptomic data. In this work, we present NO-BEARS, a novel algorithm for estimating gene regulatory networks. The NO-BEARS algorithm is built on the basis of the NOTEARS algorithm with two improvements. First, we propose a new constraint and its fast approximation to reduce the computational cost of the NO-TEARS algorithm. Next, we introduce a polynomial regression loss to handle non-linearity in gene expressions. Our implementation utilizes modern GPU computation that can decrease the time of hours-long CPU computation to seconds. Using synthetic data, we demonstrate improved performance, both in processing time and accuracy, on inferring gene regulatory networks from gene expression data.

A Meta-Transfer Objective for Learning to Disentangle Causal Mechanisms

Abstract: We propose to meta-learn causal structures based on how fast a learner adapts to new distributions arising from sparse distributional changes, e.g. due to interventions, actions of agents and other sources of non-stationarities. We show that under this assumption, the correct causal structural choices lead to faster adaptation to modified distributions because the changes are concentrated in one or just a few mechanisms when the learned knowledge is modularized appropriately. This leads to sparse expected gradients and a lower effective number of degrees of freedom needing to be relearned while adapting to the change. It motivates using the speed of adaptation to a modified distribution as a meta-learning objective. We demonstrate how this can be used to determine the cause-effect relationship between two observed variables. The distributional changes do not need to correspond to standard interventions (clamping a variable), and the learner has no direct knowledge of these interventions. We show that causal structures can be parameterized via continuous variables and learned end-to-end. We then explore how these ideas could be used to also learn an encoder that would map low-level observed variables to unobserved causal variables leading to faster adaptation out-of-distribution, learning a representation space where one can satisfy the assumptions of independent mechanisms and of small and sparse changes in these mechanisms due to actions and non-stationarities.

Causal Adversarial Network for Learning Conditional and Interventional Distributions

Abstract: We propose a generative Causal Adversarial Network (CAN) for learning and sampling from conditional and interventional distributions. In contrast to the existing CausalGAN which requires the causal graph to be given, our proposed framework learns the causal relations from the data and generates samples accordingly. The proposed CAN comprises a two-fold process namely Label Generation Network (LGN) and Conditional Image Generation Network (CIGN). The LGN is a GAN-based architecture which learns and samples from the causal model over labels. The sampled labels are then fed to CIGN, a conditional GAN architecture, which learns the relationships amongst labels and pixels and pixels themselves and generates samples based on them. This framework is equipped with an intervention mechanism which enables. the model to generate samples from interventional distributions. We quantitatively and qualitatively assess the performance of CAN and empirically show that our model is able to generate both interventional and conditional samples without having access to the causal graph for the application of face generation on CelebA data.

DAGs with No Fears: A Closer Look at Continuous Optimization for Learning Bayesian Networks

Abstract: This paper re-examines a continuous optimization framework dubbed NOTEARS for learning Bayesian networks. We first generalize existing algebraic characterizations of acyclicity to a class of matrix polynomials. Next, focusing on a one-parameter-per-edge setting, it is shown that the Karush-Kuhn-Tucker (KKT) optimality conditions for the NOTEARS formulation cannot be satisfied except in a trivial case, which explains a behavior of the associated algorithm. We then derive the KKT conditions for an equivalent reformulation, show that they are indeed necessary, and relate them to explicit constraints that certain edges be absent from the graph. If the score function is convex, these KKT conditions are also sufficient for local minimality despite the non-convexity of the constraint. Informed by the KKT conditions, a local search post-processing algorithm is proposed and shown to substantially and universally improve the structural Hamming distance of all tested algorithms, typically by a factor of 2 or more. Some combinations with local search are both more accurate and more efficient than the original NOTEARS.

On the Role of Sparsity and DAG Constraints for Learning Linear DAGs

Abstract: Learning graphical structure based on Directed Acyclic Graphs (DAGs) is a challenging problem, partly owing to the large search space of possible graphs. Recently, NOTEARS (Zheng et al., 2018) formulates the structure search problem as a continuous optimization task using the least squares objective and a proper characterization of DAGs. However, the formulation requires a hard DAG constraint and may lead to optimization difficulties. In this paper, we study the asymptotic roles of the sparsity and DAG constraints for learning DAG models in the linear Gaussian and non-Gaussian cases, and investigate their usefulness in the finite sample regime. Based on the theoretical results, we formulate a likelihood-based score function, and show that one only has to apply sparsity and DAG regularization terms to recover the underlying DAGs. This leads to an unconstrained optimization problem that is much easier to solve. Using gradient-based optimization and GPU acceleration, our procedure can easily handle thousand of nodes while retaining a high accuracy. Extensive experiments validate the effectiveness of our proposed method and show that the DAG-regularized likelihood objective is indeed favorable over the least squares one with the hard DAG constraint.

Differentiable Causal Discovery Under Unmeasured Confounding

Abstract: The data drawn from biological, economic, and social systems are often confounded due to the presence of unmeasured variables. Prior work in causal discovery has focused on discrete search procedures for selecting acyclic directed mixed graphs (ADMGs), specifically ancestral ADMGs, that encode ordinary conditional independence constraints among the observed variables of the system. However, confounded systems also exhibit more general equality restrictions that cannot be represented via these graphs, placing a limit on the kinds of structures that can be learned using ancestral ADMGs. In this work, we derive differentiable algebraic constraints that fully characterize the space of ancestral ADMGs, as well as more general classes of ADMGs, arid ADMGs and bow-free ADMGs, that capture all equality restrictions on the observed variables. We use these constraints to cast causal discovery as a continuous optimization problem and design differentiable procedures to find the best fitting ADMG when the data comes from a confounded linear system of equations with correlated errors. We demonstrate the efficacy of our method through simulations and application to a protein expression dataset.

Learning Neural Causal Models from Unknown Interventions

Abstract: Promising results have driven a recent surge of interest in continuous optimization methods for Bayesian network structure learning from observational data. However, there are theoretical limitations on the identifiability of underlying structures obtained from observational data alone. Interventional data provides much richer information about the underlying data-generating process. However, the extension and application of methods designed for observational data to include interventions is not straightforward and remains an open problem. In this paper we provide a general framework based on continuous optimization and neural networks to create models for the combination of observational and interventional data. The proposed method is even applicable in the challenging and realistic case that the identity of the intervened upon variable is unknown. We examine the proposed method in the setting of graph recovery both de novo and from a partially-known edge set. We establish strong benchmark results on several structure learning tasks, including structure recovery of both synthetic graphs as well as standard graphs from the Bayesian Network Repository.

A Critical View of the Structural Causal Model

Abstract: In the univariate case, we show that by comparing the individual complexities of univariate cause and effect, one can identify the cause and the effect, without considering their interaction at all. In our framework, complexities are captured by the reconstruction error of an autoencoder that operates on the quantiles of the distribution. Comparing the reconstruction errors of the two autoencoders, one for each variable, is shown to perform surprisingly well on the accepted causality directionality benchmarks. Hence, the decision as to which of the two is the cause and which is the effect may not be based on causality but on complexity. In the multivariate case, where one can ensure that the complexities of the cause and effect are balanced, we propose a new adversarial training method that mimics the disentangled structure of the causal model. We prove that in the multidimensional case, such modeling is likely to fit the data only in the direction of causality. Furthermore, a uniqueness result shows that the learned model is able to identify the underlying causal and residual (noise) components. Our multidimensional method outperforms the literature methods on both synthetic and real world datasets.

Amortized learning of neural causal representations

Abstract: Causal models can compactly and efficiently encode the data-generating process under all interventions and hence may generalize better under changes in distribution. These models are often represented as Bayesian networks and learning them scales poorly with the number of variables. Moreover, these approaches cannot leverage previously learned knowledge to help with learning new causal models. In order to tackle these challenges, we represent a novel algorithm called \textit{causal relational networks} (CRN) for learning causal models using neural networks. The CRN represent causal models using continuous representations and hence could scale much better with the number of variables. These models also take in previously learned information to facilitate learning of new causal models. Finally, we propose a decoding-based metric to evaluate causal models with continuous representations. We test our method on synthetic data achieving high accuracy and quick adaptation to previously unseen causal models.

Amortized Causal Discovery: Learning to Infer Causal Graphs from Time-Series Data

Abstract: On time-series data, most causal discovery methods fit a new model whenever they encounter samples from a new underlying causal graph. However, these samples often share relevant information which is lost when following this approach. Specifically, different samples may share the dynamics which describe the effects of their causal relations. We propose Amortized Causal Discovery, a novel framework that leverages such shared dynamics to learn to infer causal relations from time-series data. This enables us to train a single, amortized model that infers causal relations across samples with different underlying causal graphs, and thus leverages the shared dynamics information. We demonstrate experimentally that this approach, implemented as a variational model, leads to significant improvements in causal discovery performance, and show how it can be extended to perform well under added noise and hidden confounding.

Causal Discovery in Physical Systems from Videos

Abstract: Causal discovery is at the core of human cognition. It enables us to reason about the environment and make counterfactual predictions about unseen scenarios, that can vastly differ from our previous experiences. We consider the task of causal discovery from videos in an end-to-end fashion without supervision on the ground-truth graph structure. In particular, our goal is to discover the structural dependencies among environmental and object variables: inferring the type and strength of interactions that have a causal effect on the behavior of the dynamical system. Our model consists of (a) a perception module that extracts a semantically meaningful and temporally consistent keypoint representation from images, (b) an inference module for determining the graph distribution induced by the detected keypoints, and (c) a dynamics module that can predict the future by conditioning on the inferred graph. We assume access to different configurations and environmental conditions, i.e., data from unknown interventions on the underlying system; thus, we can hope to discover the correct underlying causal graph without explicit interventions. We evaluate our method in a planar multi-body interaction environment and scenarios involving fabrics of different shapes like shirts and pants. Experiments demonstrate that our model can correctly identify the interactions from a short sequence of images and make long-term future predictions. The causal structure assumed by the model also allows it to make counterfactual predictions and extrapolate to systems of unseen interaction graphs or graphs of various sizes.

CASTLE: Regularization via Auxiliary Causal Graph Discovery

Abstract: Regularization improves generalization of supervised models to out-of-sample data. Prior works have shown that prediction in the causal direction (effect from cause) results in lower testing error than the anti-causal direction. However, existing regularization methods are agnostic of causality. We introduce Causal Structure Learning (CASTLE) regularization and propose to regularize a neural network by jointly learning the causal relationships between variables. CASTLE learns the causal directed acyclical graph (DAG) as an adjacency matrix embedded in the neural network's input layers, thereby facilitating the discovery of optimal predictors. Furthermore, CASTLE efficiently reconstructs only the features in the causal DAG that have a causal neighbor, whereas reconstruction-based regularizers suboptimally reconstruct all input features. We provide a theoretical generalization bound for our approach and conduct experiments on a plethora of synthetic and real publicly available datasets demonstrating that CASTLE consistently leads to better out-of-sample predictions as compared to other popular benchmark regularizers.

Gradient-Based Neural DAG Learning

Abstract: We propose a novel score-based approach to learning a directed acyclic graph (DAG) from observational data. We adapt a recently proposed continuous constrained optimization formulation to allow for nonlinear relationships between variables using neural networks. This extension allows to model complex interactions while avoiding the combinatorial nature of the problem. In addition to comparing our method to existing continuous optimization methods, we provide missing empirical comparisons to nonlinear greedy search methods. On both synthetic and real-world data sets, this new method outperforms current continuous methods on most tasks, while being competitive with existing greedy search methods on important metrics for causal inference.

Causal Autoregressive Flows

Abstract: Two apparently unrelated fields -- normalizing flows and causality -- have recently received considerable attention in the machine learning community. In this work, we highlight an intrinsic correspondence between a simple family of autoregressive normalizing flows and identifiable causal models. We exploit the fact that autoregressive flow architectures define an ordering over variables, analogous to a causal ordering, to show that they are well-suited to performing a range of causal inference tasks, ranging from causal discovery to making interventional and counterfactual predictions. First, we show that causal models derived from both affine and additive autoregressive flows with fixed orderings over variables are identifiable, i.e. the true direction of causal influence can be recovered. This provides a generalization of the additive noise model well-known in causal discovery. Second, we derive a bivariate measure of causal direction based on likelihood ratios, leveraging the fact that flow models can estimate normalized log-densities of data. Third, we demonstrate that flows naturally allow for direct evaluation of both interventional and counterfactual queries, the latter case being possible due to the invertible nature of flows. Finally, throughout a series of experiments on synthetic and real data, the proposed method is shown to outperform current approaches for causal discovery as well as making accurate interventional and counterfactual predictions.

Learning DAGs without imposing acyclicity

Abstract: We explore if it is possible to learn a directed acyclic graph (DAG) from data without imposing explicitly the acyclicity constraint. In particular, for Gaussian distributions, we frame structural learning as a sparse matrix factorization problem and we empirically show that solving an $\ell_1$-penalized optimization yields to good recovery of the true graph and, in general, to almost-DAG graphs. Moreover, this approach is computationally efficient and is not affected by the explosion of combinatorial complexity as in classical structural learning algorithms.

Learning Sparse Nonparametric DAGs

Abstract: We develop a framework for learning sparse nonparametric directed acyclic graphs (DAGs) from data. Our approach is based on a recent algebraic characterization of DAGs that led to a fully continuous program for score-based learning of DAG models parametrized by a linear structural equation model (SEM). We extend this algebraic characterization to nonparametric SEM by leveraging nonparametric sparsity based on partial derivatives, resulting in a continuous optimization problem that can be applied to a variety of nonparametric and semiparametric models including GLMs, additive noise models, and index models as special cases. Unlike existing approaches that require specific modeling choices, loss functions, or algorithms, we present a completely general framework that can be applied to general nonlinear models (e.g. without additive noise), general differentiable loss functions, and generic black-box optimization routines. The code is available at https://github.com/xunzheng/notears.

Causal Discovery from Incomplete Data: A Deep Learning Approach

Abstract: As systems are getting more autonomous with the development of artificial intelligence, it is important to discover the causal knowledge from observational sensory inputs. By encoding a series of cause-effect relations between events, causal networks can facilitate the prediction of effects from a given action and analyze their underlying data generation mechanism. However, missing data are ubiquitous in practical scenarios. Directly performing existing casual discovery algorithms on partially observed data may lead to the incorrect inference. To alleviate this issue, we proposed a deep learning framework, dubbed Imputated Causal Learning (ICL), to perform iterative missing data imputation and causal structure discovery. Through extensive simulations on both synthetic and real data, we show that ICL can outperform state-of-the-art methods under different missing data mechanisms.

This table comprises a list of continuous optimization based approaches to causal discovery. `Data' indicates the dimensionality of the data the method has been demonstrated to handle. `Bi' indicates bivariate cause-effect pairs (possibly multivariate), `low' indicates <100 vertices, `medium' indicates >100, and `high' indicates either dimensionality >10,000 or data which are not already projected into a causal/semantic space (e.g., image data). `Acycl.' indicates whether the method enforces acyclicity, and `Interv.' indicates the use of interventions during learning. Please consult the main test for further details of each method. N.B. If the output is `DAG' this is not meant to necessarily imply the method identifies the true DAG.

Continuous Optimization Based Approaches

In this section we review the recent evolution of continuous optimization based approaches to structure learning, and Table {{table:6380337e-48ed-4a02-96f3-7595908a28ed}}  presents a non-exhaustive list of such methods.

One of the main advantages to combinatoric approaches to structure discovery relates to the provision of guarantees for identifying the true graph, or at least the true equivalence class. This advantage comes at a significant cost, however, because such approaches are limited to low-dimensional problems (or low-cardinality graphs) due to the super-exponential search space. One might expect, then, that even though the continuous optimization approaches are confronted with a non-trivial, non-convex solution space, they might at least scale to larger problems. Unfortunately, and as can be seen from Table {{table:6380337e-48ed-4a02-96f3-7595908a28ed}} , most continuous optimization approaches have only been evaluated on low-dimensional problems. This seems to be due to the fact that the most common acyclicity constraint, namely the one in Equation REF  from NO TEARS {{cite:709369c}}, contains a term that requires {{formula:280b611d-d74e-4be0-b210-a934587c141e}}  computations. This has motivated the development of higher-efficiency acyclicity constraints for continuous optimization approaches to structure discovery, such as the one in LEAST {{cite:9c00aac}}. One further way to alleviate the issues when confronted with high-dimensional problems is to encode the data into a lower-dimensional representation. This was undertaken in CausalVAE {{cite:ec4b4d5}}, who applied the NO TEARS constraint to a graph operating in low-dimensional representation space. Whilst this approach works well for non-semantic data (such as pixel data from images), it might not be useful in situations whereby the data are both high-dimensional and semantic (as with gene regulation data in the DREAM5 dataset {{cite:50e3a96}}). In the latter case, encoding semantic data into a new subspace may or may not be meaningful, and will likely depend on the domain of application.


Training Verifiers to Solve Math Word Problems

Abstract: State-of-the-art language models can match human performance on many tasks, but they still struggle to robustly perform multi-step mathematical reasoning. To diagnose the failures of current models and support research, we introduce GSM8K, a dataset of 8.5K high quality linguistically diverse grade school math word problems. We find that even the largest transformer models fail to achieve high test performance, despite the conceptual simplicity of this problem distribution. To increase performance, we propose training verifiers to judge the correctness of model completions. At test time, we generate many candidate solutions and select the one ranked highest by the verifier. We demonstrate that verification significantly improves performance on GSM8K, and we provide strong empirical evidence that verification scales more effectively with increased data than a finetuning baseline.

A Diverse Corpus for Evaluating and Developing English Math Word Problem Solvers

Abstract: We present ASDiv (Academia Sinica Diverse MWP Dataset), a diverse (in terms of both language patterns and problem types) English math word problem (MWP) corpus for evaluating the capability of various MWP solvers. Existing MWP corpora for studying AI progress remain limited either in language usage patterns or in problem types. We thus present a new English MWP corpus with 2,305 MWPs that cover more text patterns and most problem types taught in elementary school. Each MWP is annotated with its problem type and grade level (for indicating the level of difficulty). Furthermore, we propose a metric to measure the lexicon usage diversity of a given MWP corpus, and demonstrate that ASDiv is more diverse than existing corpora. Experiments show that our proposed corpus reflects the true capability of MWP solvers more faithfully.

Are NLP Models really able to Solve Simple Math Word Problems?

Abstract: The problem of designing NLP solvers for math word problems (MWP) has seen sustained research activity and steady gains in the test accuracy. Since existing solvers achieve high performance on the benchmark datasets for elementary level MWPs containing one-unknown arithmetic word problems, such problems are often considered "solved" with the bulk of research attention moving to more complex MWPs. In this paper, we restrict our attention to English MWPs taught in grades four and lower. We provide strong evidence that the existing MWP solvers rely on shallow heuristics to achieve high performance on the benchmark datasets. To this end, we show that MWP solvers that do not have access to the question asked in the MWP can still solve a large fraction of MWPs. Similarly, models that treat MWPs as bag-of-words can also achieve surprisingly high accuracy. Further, we introduce a challenge dataset, SVAMP, created by applying carefully chosen variations over examples sampled from existing datasets. The best accuracy achieved by state-of-the-art models is substantially lower on SVAMP, thus showing that much remains to be done even for the simplest of the MWPs.

Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problems

Abstract: Solving algebraic word problems requires executing a series of arithmetic operations---a program---to obtain a final answer. However, since programs can be arbitrarily complicated, inducing them directly from question-answer pairs is a formidable challenge. To make this task more feasible, we solve these problems by generating answer rationales, sequences of natural language and human-readable mathematical expressions that derive the final answer through a series of small steps. Although rationales do not explicitly specify programs, they provide a scaffolding for their structure via intermediate milestones. To evaluate our approach, we have created a new 100,000-sample dataset of questions, answers and rationales. Experimental results show that indirect supervision of program learning via answer rationales is a promising strategy for inducing arithmetic programs.

CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge

Abstract: When answering a question, people often draw upon their rich world knowledge in addition to the particular context. Recent work has focused primarily on answering questions given some relevant document or context, and required very little general background. To investigate question answering with prior knowledge, we present CommonsenseQA: a challenging new dataset for commonsense question answering. To capture common sense beyond associations, we extract from ConceptNet (Speer et al., 2017) multiple target concepts that have the same semantic relation to a single source concept. Crowd-workers are asked to author multiple-choice questions that mention the source concept and discriminate in turn between each of the target concepts. This encourages workers to create questions with complex semantics that often require prior knowledge. We create 12,247 questions through this procedure and demonstrate the difficulty of our task with a large number of strong baselines. Our best baseline is based on BERT-large (Devlin et al., 2018) and obtains 56% accuracy, well below human performance, which is 89%.

Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies

Abstract: A key limitation in current datasets for multi-hop reasoning is that the required steps for answering the question are mentioned in it explicitly. In this work, we introduce StrategyQA, a question answering (QA) benchmark where the required reasoning steps are implicit in the question, and should be inferred using a strategy. A fundamental challenge in this setup is how to elicit such creative questions from crowdsourcing workers, while covering a broad range of potential strategies. We propose a data collection procedure that combines term-based priming to inspire annotators, careful control over the annotator population, and adversarial filtering for eliminating reasoning shortcuts. Moreover, we annotate each question with (1) a decomposition into reasoning steps for answering it, and (2) Wikipedia paragraphs that contain the answers to each step. Overall, StrategyQA includes 2,780 examples, each consisting of a strategy question, its decomposition, and evidence paragraphs. Analysis shows that questions in StrategyQA are short, topic-diverse, and cover a wide range of strategies. Empirically, we show that humans perform well (87%) on this task, while our best baseline reaches an accuracy of $\sim$66%.

The statistics of the datasets used in this paper.# Ex. are the number of few-shot chain-of-thought exemplars used to prompt each task in evaluation.# Train and # Test denote the number of training data and test data, respectively.Note that in our experiments, we randomly sample 1000 data from the training set to reduce the computational cost and use the same test set as {{cite:8d9b59e}}.Trans.: A checkmark denotes that the exemplars are from other datasets and then transferred to this task.*: CSQA and StrategyQA do not have publicly available test set labels, so we simply follow the setting by {{cite:8d9b59e}} to evaluate the performance of the development set.

The statistics of these datasets are reported in Table {{table:40437068-29c4-4d03-a787-c3fbfdc531de}} .
Note that in our experiments,
we randomly sample 1000 data from the training set to reduce the computational cost.
This may affect the performance of the uncertainty estimation.
Intuitively, more training data will help capture the data distribution, leading to more precise uncertainty estimation.
Given more financial support, the performance of our model will continue to increase.
To make a fair comparison, we use the same test set as {{cite:8d9b59e}}.

In our implementation, the model could only access the training data {{formula:5e48d31f-7e0f-4895-bc91-6cb6e6ace088}}  before inference, and is evaluated on the test data {{formula:6c330490-2163-47f4-96e4-718ef9549413}} .
We apply the same number of exemplars as {{cite:8d9b59e}}, which is 8 for GSM8K, ASDiv, SVAMP, and SingleEq, 7 for CSQA, 6 for StrategyQA, 4 for AQuA and Letter (4).
Given that some datasets (i.e., ASDiv, SVAMP, and SingleEq) only have the test split, we adopt the annotation result of GSM8K and transfer it to these datasets for inference.
The transfer details are in Table {{table:40437068-29c4-4d03-a787-c3fbfdc531de}} .
In the inference stage, we set temperature {{formula:e92f8fb4-4625-43be-9a58-7531cdf09e2a}}  and infer 40 times for each question. We then take the most consistent answer.


Tiny-Inception-ResNet-v2: Using Deep Learning for Eliminating Bonded Labors of Brick Kilns in South Asia

Abstract: This paper proposes to employ a Inception-ResNet inspired deep learning architecture called Tiny-Inception-ResNet-v2 to eliminate bonded labor by identifying brick kilns within "Brick-Kiln-Belt" of South Asia. The framework is developed by training a network on the satellite imagery consisting of 11 different classes of South Asian region. The dataset developed during the process includes the geo-referenced images of brick kilns, houses, roads, tennis courts, farms, sparse trees, dense trees, orchards, parking lots, parks and barren lands. The dataset is made publicly available for further research. Our proposed network architecture with very fewer learning parameters outperforms all state-of-the-art architectures employed for recognition of brick kilns. Our proposed solution would enable regional monitoring and evaluation mechanisms for the Sustainable Development Goals.

Transfer Learning from Deep Features for Remote Sensing and Poverty Mapping

Abstract: The lack of reliable data in developing countries is a major obstacle to sustainable development, food security, and disaster relief. Poverty data, for example, is typically scarce, sparse in coverage, and labor-intensive to obtain. Remote sensing data such as high-resolution satellite imagery, on the other hand, is becoming increasingly available and inexpensive. Unfortunately, such data is highly unstructured and currently no techniques exist to automatically extract useful insights to inform policy decisions and help direct humanitarian efforts. We propose a novel machine learning approach to extract large-scale socioeconomic indicators from high-resolution satellite imagery. The main challenge is that training data is very scarce, making it difficult to apply modern techniques such as Convolutional Neural Networks (CNN). We therefore propose a transfer learning approach where nighttime light intensities are used as a data-rich proxy. We train a fully convolutional CNN model to predict nighttime lights from daytime imagery, simultaneously learning features that are useful for poverty prediction. The model learns filters identifying different terrains and man-made structures, including roads, buildings, and farmlands, without any supervision beyond nighttime lights. We demonstrate that these learned features are highly informative for poverty mapping, even approaching the predictive performance of survey data collected in the field.

Classifications of Proposals for Predicting Socio-Economic Indicators

In this paper, we propose a hybrid attention method to incorporate these relational inductive biases in remote sensing data. Instead of classifying each pixel, we propose a method based on Simple Linear Iterative Clustering (SLIC) image segmentation and Graph Attention GAT {{cite:2340e4a}} to detect socio-economic indicators from remote sensing data (see Table {{table:186060a2-ec1d-403b-91ea-d1d9e8e2dc03}} ). We first over-segment the image into superpixels. These superpixels become the nodes of our Graph Convolution Network (GCN). We then construct a region adjacency graph (RAG) where each superpixel is connected to every other adjacent superpixel in the image, enabling information to propagate globally. Finally, we classify each RAG via Spatially driven Attention Graph Neural Network (SAG-NN). We also propose an extension to our SAG-NN for spatio-temporal data named as Spatio-temporal Attention driven GNN (STAG-NN). Unlike, pixels or objects, superpixels are prone to change over time, to address this problem we propose a STAG-NN with Block diagonal Adjacency matrix (STAG-NN-BA) which enables us to incorporate both the spatial as well as temporal information in a single time-varying graph. The main novelty of this paper is the SAG-NN and STAG-NN-BA architectures for the prediction of spatio-temporal transition classes (such as construction, destruction, cultivation, and harvesting) from remote sensing data. We also show this approach incurs less computational cost compared with other deep learning methods. The details of our proposed approach, which is derived from vanilla GAT {{cite:2340e4a}}, are presented in Section .


Gated Convolution (T), Graph Convolution (S)

Gated Unit & Dilated Convolution (T), Graph Convolution (S)

Deep Spatio-Temporal Residual Networks for Citywide Crowd Flows Prediction

Abstract: Forecasting the flow of crowds is of great importance to traffic management and public safety, and very challenging as it is affected by many complex factors, such as inter-region traffic, events, and weather. We propose a deep-learning-based approach, called ST-ResNet, to collectively forecast the inflow and outflow of crowds in each and every region of a city. We design an end-to-end structure of ST-ResNet based on unique properties of spatio-temporal data. More specifically, we employ the residual neural network framework to model the temporal closeness, period, and trend properties of crowd traffic. For each property, we design a branch of residual convolutional units, each of which models the spatial properties of crowd traffic. ST-ResNet learns to dynamically aggregate the output of the three residual neural networks based on data, assigning different weights to different branches and regions. The aggregation is further combined with external factors, such as weather and day of the week, to predict the final traffic of crowds in each and every region. Experiments on two types of crowd flows in Beijing and New York City (NYC) demonstrate that the proposed ST-ResNet outperforms six well-known methods.

Spatio-temporal Graph Convolutional Neural Network: A Deep Learning Framework for Traffic Forecasting

Abstract: Timely accurate traffic forecast is crucial for urban traffic control and guidance. Due to the high nonlinearity and complexity of traffic flow, traditional methods cannot satisfy the requirements of mid-and-long term prediction tasks and often neglect spatial and temporal dependencies. In this paper, we propose a novel deep learning framework, Spatio-Temporal Graph Convolutional Networks (STGCN), to tackle the time series prediction problem in traffic domain. Instead of applying regular convolutional and recurrent units, we formulate the problem on graphs and build the model with complete convolutional structures, which enable much faster training speed with fewer parameters. Experiments show that our model STGCN effectively captures comprehensive spatio-temporal correlations through modeling multi-scale traffic networks and consistently outperforms state-of-the-art baselines on various real-world traffic datasets.

Graph WaveNet for Deep Spatial-Temporal Graph Modeling

Abstract: Spatial-temporal graph modeling is an important task to analyze the spatial relations and temporal trends of components in a system. Existing approaches mostly capture the spatial dependency on a fixed graph structure, assuming that the underlying relation between entities is pre-determined. However, the explicit graph structure (relation) does not necessarily reflect the true dependency and genuine relation may be missing due to the incomplete connections in the data. Furthermore, existing methods are ineffective to capture the temporal trends as the RNNs or CNNs employed in these methods cannot capture long-range temporal sequences. To overcome these limitations, we propose in this paper a novel graph neural network architecture, Graph WaveNet, for spatial-temporal graph modeling. By developing a novel adaptive dependency matrix and learn it through node embedding, our model can precisely capture the hidden spatial dependency in the data. With a stacked dilated 1D convolution component whose receptive field grows exponentially as the number of layers increases, Graph WaveNet is able to handle very long sequences. These two components are integrated seamlessly in a unified framework and the whole framework is learned in an end-to-end manner. Experimental results on two public traffic network datasets, METR-LA and PEMS-BAY, demonstrate the superior performance of our algorithm.

Adaptive Graph Convolutional Recurrent Network for Traffic Forecasting

Abstract: Modeling complex spatial and temporal correlations in the correlated time series data is indispensable for understanding the traffic dynamics and predicting the future status of an evolving traffic system. Recent works focus on designing complicated graph neural network architectures to capture shared patterns with the help of pre-defined graphs. In this paper, we argue that learning node-specific patterns is essential for traffic forecasting while the pre-defined graph is avoidable. To this end, we propose two adaptive modules for enhancing Graph Convolutional Network (GCN) with new capabilities: 1) a Node Adaptive Parameter Learning (NAPL) module to capture node-specific patterns; 2) a Data Adaptive Graph Generation (DAGG) module to infer the inter-dependencies among different traffic series automatically. We further propose an Adaptive Graph Convolutional Recurrent Network (AGCRN) to capture fine-grained spatial and temporal correlations in traffic series automatically based on the two modules and recurrent networks. Our experiments on two real-world traffic datasets show AGCRN outperforms state-of-the-art by a significant margin without pre-defined graphs about spatial connections.

GMAN: A Graph Multi-Attention Network for Traffic Prediction

Abstract: Long-term traffic prediction is highly challenging due to the complexity of traffic systems and the constantly changing nature of many impacting factors. In this paper, we focus on the spatio-temporal factors, and propose a graph multi-attention network (GMAN) to predict traffic conditions for time steps ahead at different locations on a road network graph. GMAN adapts an encoder-decoder architecture, where both the encoder and the decoder consist of multiple spatio-temporal attention blocks to model the impact of the spatio-temporal factors on traffic conditions. The encoder encodes the input traffic features and the decoder predicts the output sequence. Between the encoder and the decoder, a transform attention layer is applied to convert the encoded traffic features to generate the sequence representations of future time steps as the input of the decoder. The transform attention mechanism models the direct relationships between historical and future time steps that helps to alleviate the error propagation problem among prediction time steps. Experimental results on two real-world traffic prediction tasks (i.e., traffic volume prediction and traffic speed prediction) demonstrate the superiority of GMAN. In particular, in the 1 hour ahead prediction, GMAN outperforms state-of-the-art methods by up to 4% improvement in MAE measure. The source code is available at https://github.com/zhengchuanpan/GMAN.

Analysis of representative STTP approaches with STAnalytic.

As shown in Table {{table:4dc0ab86-fd08-4511-bfce-056fc79fb903}} , most of the first stream of works are much older, including Hamed et al. (1995) {{cite:3ed2a5d}}, Williams et al. (1998) {{cite:a6d5aae}}, and Williams et al. (2003) {{cite:f3369bc}}. All of them take the traffic prediction problem as time-series data analysis and thus solve the problem with ARIMA methods. The results from Williams et al. (1998, 2003) {{cite:a6d5aae}}, {{cite:f3369bc}} reveal that the daily and weekly periodicity widely exists and thus it is important to consider them in the temporal factors of the traffic prediction. More recently, researchers start leveraging advanced machine learning techniques, such as LSTM, to learn the temporal patterns in traffic prediction {{cite:9fe1404}}.

For the second stream of works, we list several representatives {{cite:6a215b7}}, {{cite:f30b2f5}}, {{cite:c165125}}, {{cite:7d09719}}, {{cite:75eecae}}, {{cite:cb5ac7b}}, {{cite:8c7eedc}}, {{cite:f21edcb}}, which are published in prestigious venues and highly cited.  All of these studies apply deep learning techniques into STTP with both temporal and spatial knowledge in consideration. Note that their modeling techniques are distinct (see `modeling technique' in Table {{table:4dc0ab86-fd08-4511-bfce-056fc79fb903}} ), thus not directly comparable from the technique perspective. However, with STAnalytic, we put more focus on analyzing which types of temporal and spatial knowledge are taken into consideration, and then these studies are clearly comparable as follows:

Compared with the existing methods listed in Table {{table:4dc0ab86-fd08-4511-bfce-056fc79fb903}} , STMeta is flexible to incorporate a variety of temporal and spatial factors. In particular, ST-MGCN {{cite:75eecae}} has also considered multiple temporal and spatial knowledge factors, but its network structure is different from STMeta in temporal modeling. More specifically, ST-MGCN directly combines the weekly, daily, and closeness historical records into one time-series sequence as the input. In comparison, the inputs of STMeta include three types of time-series data regarding closeness, daily periodicity, and weekly periodicity, respectively; thus, STMeta introduces a temporal aggregation unit to combine the three temporal patterns. We think that the design of STMeta may help to learn different temporal patterns more easily, as each temporal pattern (closeness, daily, and weekly periodicity) has its own time-series sequence for dedicated learning.

The hyperparameters of the above benchmark approaches follow the default settings as their original papers.We have also tested some other approaches in Table {{table:4dc0ab86-fd08-4511-bfce-056fc79fb903}} , such as ASTGCN {{cite:cb5ac7b}} and STSGCN {{cite:d53dd12}}; however, they are hard to converge under certain experiment scenarios.


Solving the Traveling Salesman's Problem Using the African Buffalo Optimization

Abstract: This paper proposes the African Buffalo Optimization (ABO) which is a new metaheuristic algorithm that is derived from careful observation of the African buffalos, a species of wild cows, in the African forests and savannahs. This animal displays uncommon intelligence, strategic organizational skills, and exceptional navigational ingenuity in its traversal of the African landscape in search for food. The African Buffalo Optimization builds a mathematical model from the behavior of this animal and uses the model to solve 33 benchmark symmetric Traveling Salesman's Problem and six difficult asymmetric instances from the TSPLIB. This study shows that buffalos are able to ensure excellent exploration and exploitation of the search space through regular communication, cooperation, and good memory of its previous personal exploits as well as tapping from the herd's collective exploits. The results obtained by using the ABO to solve these TSP cases were benchmarked against the results obtained by using other popular algorithms. The results obtained using the African Buffalo Optimization algorithm are very competitive.

Bat Algorithm: A Novel Approach for Global Engineering Optimization

Abstract: Nature-inspired algorithms are among the most powerful algorithms for optimization. In this study, a new nature-inspired metaheuristic optimization algorithm, called bat algorithm (BA), is introduced for solving engineering optimization tasks. The proposed BA is based on the echolocation behavior of bats. After a detailed formulation and explanation of its implementation, BA is verified using eight nonlinear engineering optimization problems reported in the specialized literature. BA has been carefully implemented and carried out optimization for eight well-known optimization tasks. Then, a comparison has been made between the proposed algorithm and other existing algorithms. The optimal solutions obtained by the proposed algorithm are better than the best solutions obtained by the existing methods. The unique search features used in BA are analyzed, and their implications for future research are also discussed in detail.

Intelligent Algorithm for Optimum Solutions Based on the Principles of Bat Sonar

Abstract: This paper presents a new intelligent algorithm that can solve the problems of finding the optimum solution in the state space among which the desired solution resides. The algorithm mimics the principles of bat sonar in finding its targets. The algorithm introduces three search approaches. The first search approach considers a single sonar unit (SSU) with a fixed beam length and a single starting point. In this approach, although the results converge toward the optimum fitness, it is not guaranteed to find the global optimum solution especially for complex problems; it is satisfied with finding 'acceptably good' solutions to these problems. The second approach considers multisonar units (MSU) working in parallel in the same state space. Each unit has its own starting point and tries to find the optimum solution. In this approach the probability that the algorithm converges toward the optimum solution is significantly increased. It is found that this approach is suitable for complex functions and for problems of wide state space. In the third approach, a single sonar unit with a moment (SSM) is used in order to handle the problem of convergence toward a local optimum rather than a global optimum. The momentum term is added to the length of the transmitted beams. This will give the chance to find the best fitness in a wider range within the state space. In this paper a comparison between the proposed algorithm and genetic algorithm (GA) has been made. It showed that both of the algorithms can catch approximately the optimum solutions for all of the testbed functions except for the function that has a local minimum, in which the proposed algorithm's result is much better than that of the GA algorithm. On the other hand, the comparison showed that the required execution time to obtain the optimum solution using the proposed algorithm is much less than that of the GA algorithm.

Novel Optimization Algorithm Inspired by Camel Traveling Behavior

Abstract: : This article presents a novel optimization algorithm inspired by camel traveling behavior that called Camel algorithm (CA). Camel is one of the extraordinary animals with many distinguish characters that allow it to withstand the severer desert environment. The Camel algorithm used to find the optimal solution for several different benchmark test functions. The results of CA and the results of GA and PSO algorithms are experimentally compared. The results indicate that the promising search ability of camel algorithm is useful, produce good results and outperform the others for different test functions.

Donkey and Smuggler Optimization Algorithm: A Collaborative Working Approach to Path Finding

Abstract: Swarm Intelligence is a metaheuristic optimization approach that has become very predominant over the last few decades. These algorithms are inspired by animals' physical behaviors and their evolutionary perceptions. The simplicity of these algorithms allows researchers to simulate different natural phenomena to solve various real-world problems. This paper suggests a novel algorithm called Donkey and Smuggler Optimization Algorithm (DSO). The DSO is inspired by the searching behavior of donkeys. The algorithm imitates transportation behavior such as searching and selecting routes for movement by donkeys in the actual world. Two modes are established for implementing the search behavior and route-selection in this algorithm. These are the Smuggler and Donkeys. In the Smuggler mode, all the possible paths are discovered and the shortest path is then found. In the Donkeys mode, several donkey behaviors are utilized such as Run, Face & Suicide, and Face & Support. Real world data and applications are used to test the algorithm. The experimental results consisted of two parts, firstly, we used the standard benchmark test functions to evaluate the performance of the algorithm in respect to the most popular and the state of the art algorithms. Secondly, the DSO is adapted and implemented on three real-world applications namely; traveling salesman problem, packet routing, and ambulance routing. The experimental results of DSO on these real-world problems are very promising. The results exhibit that the suggested DSO is appropriate to tackle other unfamiliar search spaces and complex problems.

Algorithms based on mammals behavior (part 1).

A more complex behavior can be observed in several primates like spider monkey ( {{cite:034dfb9}}), blue monkeys ( {{cite:c433ab4}}), ageist monkeys which age could bring a difference for better performance to solve problems ( {{cite:43562f6}}) and sexual behavior with individual intelligence of chimps ( {{cite:b1f7f6d}}). Humans are considered primates and have the same behavior but in a more sophisticated way to form tribes ( {{cite:aeba027}}), groups ( {{cite:a4e9558}}, {{cite:ded72fe}}) or aggregates as crowds ( {{cite:83e764a}}) (More complex behaviors unique for humans are not considered here because it is very difficult to be expressed by other species) All algorithms based on mammals behavior are shown in Tables {{table:885dfe6c-3ac5-40ba-94b3-1ea467147baf}}  and {{table:513c1e65-8c75-4450-9dad-2bd77e2896a9}} .


Adversarial Training Methods for Semi-Supervised Text Classification

Abstract: Adversarial training provides a means of regularizing supervised learning algorithms while virtual adversarial training is able to extend supervised learning algorithms to the semi-supervised setting. However, both methods require making small perturbations to numerous entries of the input vector, which is inappropriate for sparse high-dimensional inputs such as one-hot word representations. We extend adversarial and virtual adversarial training to the text domain by applying perturbations to the word embeddings in a recurrent neural network rather than to the original input itself. The proposed method achieves state of the art results on multiple benchmark semi-supervised and purely supervised tasks. We provide visualizations and analysis showing that the learned word embeddings have improved in quality and that while training, the model is less prone to overfitting.

Targeted Adversarial Training for Natural Language Understanding

Abstract: We present a simple yet effective Targeted Adversarial Training (TAT) algorithm to improve adversarial training for natural language understanding. The key idea is to introspect current mistakes and prioritize adversarial training steps to where the model errs the most. Experiments show that TAT can significantly improve accuracy over standard adversarial training on GLUE and attain new state-of-the-art zero-shot results on XNLI. Our code will be released upon acceptance of the paper.

Summary of Virtual Adversarial Training (VAT) based defense methods

The work proposed in {{cite:bdd3888}} extended the notion of virtual adversarial training and adversarial training for text classification and sequence models proposing this technique as a regularization method. For defending the models, they introduced perturbations in word embeddings of the text inputs, while minimizing the KL divergence of VAT.
In another work {{cite:ff38ee7}}, authors proposed a VAT method by performing adversarial steps on those examples which are predicted as wrong by the model and then regularise the model for this target direction in contrast with general adversarial training methods where the perturbation is done for all examples with variation from the gold label. In a targeted training manner, they try to steer the examples to a particular label {{formula:9d0775bc-a67f-4d11-ae1f-cb769257bd39}}  and presented a comparison with human-annotated data along with other adversarial training algorithms.
In the same direction, the work {{cite:f5370ce}} authors proposed a novel adversarial robust model “Adversarial training for large neural LangUage Models(ALUM)" for defending BERT-based pretraining language models. It’s a general model for adversarial training in pretraining and fine tuning which regularizes the training objective by applying perturbations in the embedding space that maximizes the adversarial loss. The model is regularized using Virtual Adversarial Training.
Experimenting with different word embeddings using VAT, {{cite:33781b2}} extended the adversarial training regularization for semi-supervised tasks. They used continuous bag of words (CBOW) model for generating word embeddings and restricted perturbation directions for creating adversaries.
Targeting specifically sequence labelling tasks in NLP, {{cite:1eba1dc}} proposed VAT for sequence labelling task combining CRF, making sequence labelling task more robust. They use CNN layer for extracting character and word embeddings, LSTM for sequence encoding, and CRF decoder layer to incorporate the probabilities of label transition.
Introducing more variations to VAT, {{cite:18970f7}} proposed Token aware virtual adversarial training. In contrast with conventional virtual adversarial training, TAVAT generated token aware perturbations instead of random perturbations to avoid unnecessary noise and take important information carried by tokens into consideration. Table {{table:51d34ef5-0681-42cf-bead-e713c54e1f64}}  describes the summary of Virtual Adversarial Training (VAT) based methods along with the specified NLP task for their design and granularity of the perturbation.