Very Deep Convolutional Networks for Large-Scale Image Recognition

Abstract: In this work we investigate the effect of the convolutional network depth on its accuracy in the large-scale image recognition setting. Our main contribution is a thorough evaluation of networks of increasing depth using an architecture with very small (3x3) convolution filters, which shows that a significant improvement on the prior-art configurations can be achieved by pushing the depth to 16-19 weight layers. These findings were the basis of our ImageNet Challenge 2014 submission, where our team secured the first and the second places in the localisation and classification tracks respectively. We also show that our representations generalise well to other datasets, where they achieve state-of-the-art results. We have made our two best-performing ConvNet models publicly available to facilitate further research on the use of deep visual representations in computer vision.

MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications

Abstract: We present a class of efficient models called MobileNets for mobile and embedded vision applications. MobileNets are based on a streamlined architecture that uses depth-wise separable convolutions to build light weight deep neural networks. We introduce two simple global hyper-parameters that efficiently trade off between latency and accuracy. These hyper-parameters allow the model builder to choose the right sized model for their application based on the constraints of the problem. We present extensive experiments on resource and accuracy tradeoffs and show strong performance compared to other popular models on ImageNet classification. We then demonstrate the effectiveness of MobileNets across a wide range of applications and use cases including object detection, finegrain classification, face attributes and large scale geo-localization.

Densely Connected Convolutional Networks

Abstract: Recent work has shown that convolutional networks can be substantially deeper, more accurate, and efficient to train if they contain shorter connections between layers close to the input and those close to the output. In this paper, we embrace this observation and introduce the Dense Convolutional Network (DenseNet), which connects each layer to every other layer in a feed-forward fashion. Whereas traditional convolutional networks with L layers have L connections—one between each layer and its subsequent layer—our network has L(L+1)/2 direct connections. For each layer, the feature-maps of all preceding layers are used as inputs, and its own feature-maps are used as inputs into all subsequent layers. DenseNets have several compelling advantages: they alleviate the vanishing-gradient problem, strengthen feature propagation, encourage feature reuse, and substantially reduce the number of parameters. We evaluate our proposed architecture on four highly competitive object recognition benchmark tasks (CIFAR-10, CIFAR-100, SVHN, and ImageNet). DenseNets obtain significant improvements over the state-of-the-art on most of them, whilst requiring less memory and computation to achieve high performance. Code and pre-trained models are available at https://github.com/liuzhuang13/DenseNet.

Rethinking the Inception Architecture for Computer Vision

Abstract: Convolutional networks are at the core of most state-of-the-art computer vision solutions for a wide variety of tasks. Since 2014 very deep convolutional networks started to become mainstream, yielding substantial gains in various benchmarks. Although increased model size and computational cost tend to translate to immediate quality gains for most tasks (as long as enough labeled data is provided for training), computational efficiency and low parameter count are still enabling factors for various use cases such as mobile vision and big-data scenarios. Here we explore ways to scale up networks in ways that aim at utilizing the added computation as efficiently as possible by suitably factorized convolutions and aggressive regularization. We benchmark our methods on the ILSVRC 2012 classification challenge validation set demonstrate substantial gains over the state of the art: 21.2% top-1 and 5.6% top-5 error for single frame evaluation using a network with a computational cost of 5 billion multiply-adds per inference and with using less than 25 million parameters. With an ensemble of 4 models and multi-crop evaluation, we report 3.5% top-5 error on the validation set (3.6% error on the test set) and 17.3% top-1 error on the validation set.

Evaluated DNN models.

Preparing Datasets {{formula:fc7eb85c-5ef1-4fbf-b65e-fcef06b490f5}}  and {{formula:e0e1d5ce-2c44-4255-b4b5-1d77579fd120}} . We prepare an
image set {{formula:8a7b995d-d58d-48c9-9c79-149aca847479}}  by randomly selecting images from a dataset listed in
{{table:d94c56b3-88db-46fa-9d7e-091b48d16b4c}} . These three datasets are the most widely-used datasets
for DNN testing. For each {{formula:284ac086-3b95-462e-b70b-867ac397e804}} , we mutate it with different {{formula:1af17a15-35da-48a3-8967-6c9793cf41d2}} , as presented in {{table:4f93dccd-938c-4a21-9284-00c8030d9c71}} , and produce {{formula:0c291615-759f-4f23-93e9-886d394f7d04}} .Note that a {{formula:0f2e7792-b0ac-4273-8dae-6a20d5d8db80}}  {{formula:9cdfe140-1d11-4e4a-a69e-1c9170308c05}}  may be used for multiple times, each with a randomly selected
configuration, e.g., rotation degree or brightness level. Then, for each {{formula:4e61dbd9-fed8-4def-8ea5-216fe7bc8396}}
and its mutated {{formula:15b6fea8-f976-444c-9c6a-1542509f036a}} , we construct a set {{formula:1fc8cb1a-e72b-4c8f-bfcd-7de20ea0873f}}  with image
pairs {{formula:14c81915-7f73-4d53-8254-f84f7d191af5}} , namely, {{formula:c65d49e7-c648-4410-80f6-e7e0d43b0f17}} . We further divide {{formula:b252665b-f8f6-4882-a664-b5303dfe4f55}}  into two
collections: 1) {{formula:88ceabf3-50c3-4521-9cd9-751658a087e3}}  where the DNN {{formula:a5a6ae0e-631f-4870-a08b-0f4be42afa08}}  has the same prediction label
for {{formula:1bf40951-3764-428c-8375-b51b6310cdaa}}  and {{formula:3aa5dab8-8315-4902-ac88-0b8c2c24f9c0}} , i.e., {{formula:bf56a904-a1c7-4ec2-a2ad-7225b1d5e6c5}} , and 2) {{formula:fdd702b5-4a33-455b-af93-f93d78bc2224}}  where the DNN {{formula:6745bc8c-5ba5-40a2-86d6-81a403948a9f}}  yields inconsistent labels for
{{formula:1ef20530-3d93-477a-a8b1-05c66ffef5c5}}  and {{formula:a6a6db50-09aa-4e4e-b102-84351e96de6c}} , i.e., {{formula:ffe3464f-4a22-41e9-b78b-f1e279784db2}} .
For each tested DNN, we keep generating {{formula:2a1b7662-ce24-4cdf-9187-93aeb573a73e}}  until both {{formula:b5d86275-7d4a-46fd-8c52-38468e06a193}}  and
{{formula:67a5da84-6c0b-4834-8bc7-17a54a42f3c7}}  has {{formula:734fdd15-928b-4375-a486-86ee09e2ef2c}}  pairs.

Preparing Tested DNNs. We list all DNN models tested in this paper in
{{table:d94c56b3-88db-46fa-9d7e-091b48d16b4c}} . Note that for the first three models, we use two instances
trained over two different datasets, ImageNet and
CIFAR10. All ImageNet-trained models are officially
provided by PyTorch and other models are well-trained (over 94% test accuracy).
They are commonly used in daily DNN tasks and testings.

We use five popular and representative DNNs trained on ImageNet, as listed in
{{table:d94c56b3-88db-46fa-9d7e-091b48d16b4c}} . For each model, we randomly select 500 pairs of images,
which are evenly generated by each type of {{formula:69cfa11d-b02c-4263-b580-a4b1688f29fc}}  we introduced in
Sec. , from {{formula:7cccb8d8-1b7e-4ede-a032-679194d70797}}  and collect a set of total
{{formula:d8f8a846-4f6e-4960-9237-eed1ebb82004}}  pairs. We then repeat each pair three times so that three participants
can analyze each pair of decisions. The totaling 7,500 images are evenly
assigned to 15 participants. Each participant may halt the human evaluation
anytime to avoid fatigue. On average, each participant requires {{formula:5175390f-bb06-47ed-8acd-e925569a1c7b}}  hours to
finish the entire evaluation.


Manhattan Room Layout Reconstruction from a Single 360 image: A Comparative Study of State-of-the-art Methods

Abstract: Recent approaches for predicting layouts from 360 panoramas produce excellent results. These approaches build on a common framework consisting of three steps: a pre-processing step based on edge-based alignment, prediction of layout elements, and a post-processing step by fitting a 3D layout to the layout elements. Until now, it has been difficult to compare the methods due to multiple different design decisions, such as the encoding network (e.g. SegNet or ResNet), type of elements predicted (e.g. corners, wall/floor boundaries, or semantic segmentation), or method of fitting the 3D layout. To address this challenge, we summarize and describe the common framework, the variants, and the impact of the design decisions. For a complete evaluation, we also propose extended annotations for the Matterport3D dataset [3], and introduce two depth-based evaluation metrics.

LayoutNet: Reconstructing the 3D Room Layout from a Single RGB Image

Abstract: We propose an algorithm to predict room layout from a single image that generalizes across panoramas and perspective images, cuboid layouts and more general layouts (e.g. L-shape room). Our method operates directly on the panoramic image, rather than decomposing into perspective images as do recent works. Our network architecture is similar to that of RoomNet, but we show improvements due to aligning the image based on vanishing points, predicting multiple layout elements (corners, boundaries, size and translation), and fitting a constrained Manhattan layout to the resulting predictions. Our method compares well in speed and accuracy to other existing work on panoramas, achieves among the best accuracy for perspective images, and can handle both cuboid-shaped and more general Manhattan layouts.

Dataset statistics.

Setting 1: labeled pre-training data + unlabeled target data.

In this setting, we sample the same amount of training data from both the pre-training dataset and the target dataset, as shown in Table {{table:e39a3aa1-c1db-49f0-b2b3-5f0b27e974aa}} . We start from the pre-trained model, and then generate pseudo-labels for unlabeled data. Then, during model finetuning, we include both labeled pre-trained data and pseudo-labeled data from MP3D-FPE.
In Table {{table:75a2764f-6966-419b-b3c1-6fd71ac585ec}} , we show that our method consistently performs favorably against SSLayout360{{formula:4f6b0c4b-7263-4b68-9849-6327af54f376}}  across most metrics, in which SSLayout360{{formula:d02f0cb7-2101-4534-9ad0-fa3c8f8ad18b}}  does not consider the usage of multi-view layout consistency as our approach does.


Evaluating Pronominal Anaphora in Machine Translation: An Evaluation Measure and a Test Suite

Abstract: The ongoing neural revolution in machine translation has made it easier to model larger contexts beyond the sentence-level, which can potentially help resolve some discourse-level ambiguities such as pronominal anaphora, thus enabling better translations. Unfortunately, even when the resulting improvements are seen as substantial by humans, they remain virtually unnoticed by traditional automatic evaluation measures like BLEU, as only a few words end up being affected. Thus, specialized evaluation measures are needed. With this aim in mind, we contribute an extensive, targeted dataset that can be used as a test suite for pronoun translation, covering multiple source languages and different pronoun errors drawn from real system translations, for English. We further propose an evaluation measure to differentiate good and bad pronoun translations. We also conduct a user study to report correlations with human judgments.

Optimal Complexity in Decentralized Training

Abstract: Decentralization is a promising method of scaling up parallel machine learning systems. In this paper, we provide a tight lower bound on the iteration complexity for such methods in a stochastic non-convex setting. Our lower bound reveals a theoretical gap in known convergence rates of many existing decentralized training algorithms, such as D-PSGD. We prove by construction this lower bound is tight and achievable. Motivated by our insights, we further propose DeTAG, a practical gossip-style decentralized algorithm that achieves the lower bound with only a logarithm gap. Empirically, we compare DeTAG with other decentralized algorithms on image classification tasks, and we show DeTAG enjoys faster convergence compared to baselines, especially on unshuffled data and in sparse networks.

Targeted test sets used for evaluating discourse phenomena.

Targeted tests have been developed in order to evaluate performance on discourse phenomena. Table {{table:d4dd81b1-dbe6-4aea-ac7e-0c8d7f96868a}}  lists the test sets used in this paper. While highly relevant, data created by {{cite:9df2fa9}} has not been released at the time of writing this paper. These test sets contain contrastive translation pairs, consisting of a correct human-generated translation, and a variant of it where a pronoun, or another linguistic unit of interest, is swapped with an incorrect one. Table {{table:246454bf-d75d-42ca-82de-eb3983ce2989}}  shows examples from these data sets.


Multilingual Knowledge Graph Embeddings for Cross-lingual Knowledge Alignment

Abstract: Many recent works have demonstrated the benefits of knowledge graph embeddings in completing monolingual knowledge graphs. Inasmuch as related knowledge bases are built in several different languages, achieving cross-lingual knowledge alignment will help people in constructing a coherent knowledge base, and assist machines in dealing with different expressions of entity relationships across diverse human languages. Unfortunately, achieving this highly desirable crosslingual alignment by human labor is very costly and errorprone. Thus, we propose MTransE, a translation-based model for multilingual knowledge graph embeddings, to provide a simple and automated solution. By encoding entities and relations of each language in a separated embedding space, MTransE provides transitions for each embedding vector to its cross-lingual counterparts in other spaces, while preserving the functionalities of monolingual embeddings. We deploy three different techniques to represent cross-lingual transitions, namely axis calibration, translation vectors, and linear transformations, and derive five variants for MTransE using different loss functions. Our models can be trained on partially aligned graphs, where just a small portion of triples are aligned with their cross-lingual counterparts. The experiments on cross-lingual entity matching and triple-wise alignment verification show promising results, with some variants consistently outperforming others on different tasks. We also explore how MTransE preserves the key properties of its monolingual counterpart TransE.

Cross-lingual Entity Alignment via Joint Attribute-Preserving Embedding

Abstract: Entity alignment is the task of finding entities in two knowledge bases (KBs) that represent the same real-world object. When facing KBs in different natural languages, conventional cross-lingual entity alignment methods rely on machine translation to eliminate the language barriers. These approaches often suffer from the uneven quality of translations between languages. While recent embedding-based techniques encode entities and relationships in KBs and do not need machine translation for cross-lingual entity alignment, a significant number of attributes remain largely unexplored. In this paper, we propose a joint attribute-preserving embedding model for cross-lingual entity alignment. It jointly embeds the structures of two KBs into a unified vector space and further refines it by leveraging attribute correlations in the KBs. Our experimental results on real-world datasets show that this approach significantly outperforms the state-of-the-art embedding approaches for cross-lingual entity alignment and could be complemented with methods based on machine translation.

Co-training Embeddings of Knowledge Graphs and Entity Descriptions for Cross-lingual Entity Alignment

Abstract: Multilingual knowledge graph (KG) embeddings provide latent semantic representations of entities and structured knowledge with cross-lingual inferences, which benefit various knowledge-driven cross-lingual NLP tasks. However, precisely learning such cross-lingual inferences is usually hindered by the low coverage of entity alignment in many KGs. Since many multilingual KGs also provide literal descriptions of entities, in this paper, we introduce an embedding-based approach which leverages a weakly aligned multilingual KG for semi-supervised cross-lingual learning using entity descriptions. Our approach performs co-training of two embedding models, i.e. a multilingual KG embedding model and a multilingual literal description embedding model. The models are trained on a large Wikipedia-based trilingual dataset where most entity alignment is unknown to training. Experimental results show that the performance of the proposed approach on the entity alignment task improves at each iteration of co-training, and eventually reaches a stage at which it significantly surpasses previous approaches. We also show that our approach has promising abilities for zero-shot entity alignment, and cross-lingual KG completion.

Learning to Exploit Long-term Relational Dependencies in Knowledge Graphs

Abstract: We study the problem of knowledge graph (KG) embedding. A widely-established assumption to this problem is that similar entities are likely to have similar relational roles. However, existing related methods derive KG embeddings mainly based on triple-level learning, which lack the capability of capturing long-term relational dependencies of entities. Moreover, triple-level learning is insufficient for the propagation of semantic information among entities, especially for the case of cross-KG embedding. In this paper, we propose recurrent skipping networks (RSNs), which employ a skipping mechanism to bridge the gaps between entities. RSNs integrate recurrent neural networks (RNNs) with residual learning to efficiently capture the long-term relational dependencies within and between KGs. We design an end-to-end framework to support RSNs on different tasks. Our experimental results showed that RSNs outperformed state-of-the-art embedding-based methods for entity alignment and achieved competitive performance for KG completion.

TransEdge: Translating Relation-contextualized Embeddings for Knowledge Graphs

Abstract: Learning knowledge graph (KG) embeddings has received increasing attention in recent years. Most embedding models in literature interpret relations as linear or bilinear mapping functions to operate on entity embeddings. However, we find that such relation-level modeling cannot capture the diverse relational structures of KGs well. In this paper, we propose a novel edge-centric embedding model TransEdge, which contextualizes relation representations in terms of specific head-tail entity pairs. We refer to such contextualized representations of a relation as edge embeddings and interpret them as translations between entity embeddings. TransEdge achieves promising performance on different prediction tasks. Our experiments on benchmark datasets indicate that it obtains the state-of-the-art results on embedding-based entity alignment. We also show that TransEdge is complementary with conventional entity alignment methods. Moreover, it shows very competitive performance on link prediction.

Multi-Channel Graph Neural Network for Entity Alignment

Abstract: Entity alignment typically suffers from the issues of structural heterogeneity and limited seed alignments. In this paper, we propose a novel Multi-channel Graph Neural Network model (MuGNN) to learn alignment-oriented knowledge graph (KG) embeddings by robustly encoding two KGs via multiple channels. Each channel encodes KGs via different relation weighting schemes with respect to self-attention towards KG completion and cross-KG attention for pruning exclusive entities respectively, which are further combined via pooling techniques. Moreover, we also infer and transfer rule knowledge for completing two KGs consistently. MuGNN is expected to reconcile the structural differences of two KGs, and thus make better use of seed alignments. Extensive experiments on five publicly available datasets demonstrate our superior performance (5% Hits@1 up on average).

Relation-Aware Entity Alignment for Heterogeneous Knowledge Graphs

Abstract: Entity alignment is the task of linking entities with the same real-world identity from different knowledge graphs (KGs), which has been recently dominated by embedding-based methods. Such approaches work by learning KG representations so that entity alignment can be performed by measuring the similarities between entity embeddings. While promising, prior works in the field often fail to properly capture complex relation information that commonly exists in multi-relational KGs, leaving much room for improvement. In this paper, we propose a novel Relation-aware Dual-Graph Convolutional Network (RDGCN) to incorporate relation information via attentive interactions between the knowledge graph and its dual relation counterpart, and further capture neighboring structures to learn better entity representations. Experiments on three real-world cross-lingual datasets show that our approach delivers better and more robust results over the state-of-the-art alignment methods by learning better KG representations.

Cross-lingual Knowledge Graph Alignment via Graph Matching Neural Network

Abstract: Previous cross-lingual knowledge graph (KG) alignment studies rely on entity embeddings derived only from monolingual KG structural information, which may fail at matching entities that have different facts in two KGs. In this paper, we introduce the topic entity graph, a local sub-graph of an entity, to represent entities with their contextual information in KG. From this view, the KB-alignment task can be formulated as a graph matching problem; and we further propose a graph-attention based solution, which first matches all entities in two topic entity graphs, and then jointly model the local matching information to derive a graph-level matching vector. Experiments show that our model outperforms previous state-of-the-art methods by a large margin.

Knowledge Association with Hyperbolic Knowledge Graph Embeddings

Abstract: Capturing associations for knowledge graphs (KGs) through entity alignment, entity type inference and other related tasks benefits NLP applications with comprehensive knowledge representations. Recent related methods built on Euclidean embeddings are challenged by the hierarchical structures and different scales of KGs. They also depend on high embedding dimensions to realize enough expressiveness. Differently, we explore with low-dimensional hyperbolic embeddings for knowledge association. We propose a hyperbolic relational graph neural network for KG embedding and capture knowledge associations with a hyperbolic transformation. Extensive experiments on entity alignment and type inference demonstrate the effectiveness and efficiency of our method.

Relational Reflection Entity Alignment

Abstract: Entity alignment aims to identify equivalent entity pairs from different Knowledge Graphs (KGs), which is essential in integrating multi-source KGs. Recently, with the introduction of GNNs into entity alignment, the architectures of recent models have become more and more complicated. We even find two counter-intuitive phenomena within these methods: (1) The standard linear transformation in GNNs is not working well. (2) Many advanced KG embedding models designed for link prediction task perform poorly in entity alignment. In this paper, we abstract existing entity alignment methods into a unified framework, Shape-Builder & Alignment, which not only successfully explains the above phenomena but also derives two key criteria for an ideal transformation operation. Furthermore, we propose a novel GNNs-based method, Relational Reflection Entity Alignment (RREA). RREA leverages Relational Reflection Transformation to obtain relation specific embeddings for each entity in a more efficient way. The experimental results on real-world datasets show that our model significantly outperforms the state-of-the-art methods, exceeding by 5.8%-10.9% on Hits@1.

Boosting the Speed of Entity Alignment 10*: Dual Attention Matching Network with Normalized Hard Sample Mining

Abstract: Seeking the equivalent entities among multi-source Knowledge Graphs (KGs) is the pivotal step to KGs integration, also known as \emph{entity alignment} (EA). However, most existing EA methods are inefficient and poor in scalability. A recent summary points out that some of them even require several days to deal with a dataset containing 200,000 nodes (DWY100K). We believe over-complex graph encoder and inefficient negative sampling strategy are the two main reasons. In this paper, we propose a novel KG encoder -- Dual Attention Matching Network (Dual-AMN), which not only models both intra-graph and cross-graph information smartly, but also greatly reduces computational complexity. Furthermore, we propose the Normalized Hard Sample Mining Loss to smoothly select hard negative samples with reduced loss shift. The experimental results on widely used public datasets indicate that our method achieves both high accuracy and high efficiency. On DWY100K, the whole running process of our method could be finished in 1,100 seconds, at least 10* faster than previous work. The performances of our method also outperform previous works across all datasets, where Hits@1 and MRR have been improved from 6% to 13%.

ClusterEA: Scalable Entity Alignment with Stochastic Training and Normalized Mini-batch Similarities

Abstract: Entity alignment (EA) aims at finding equivalent entities in different knowledge graphs (KGs). Embedding-based approaches have dominated the EA task in recent years. Those methods face problems that come from the geometric properties of embedding vectors, including hubness and isolation. To solve these geometric problems, many normalization approaches have been adopted for EA. However, the increasing scale of KGs renders it hard for EA models to adopt the normalization processes, thus limiting their usage in real-world applications. To tackle this challenge, we present ClusterEA, a general framework that is capable of scaling up EA models and enhancing their results by leveraging normalization methods on mini-batches with a high entity equivalent rate. ClusterEA contains three components to align entities between large-scale KGs, including stochastic training, ClusterSampler, and SparseFusion. It first trains a large-scale Siamese GNN for EA in a stochastic fashion to produce entity embeddings. Based on the embeddings, a novel ClusterSampler strategy is proposed for sampling highly overlapped mini-batches. Finally, ClusterEA incorporates SparseFusion, which normalizes local and global similarity and then fuses all similarity matrices to obtain the final similarity matrix. Extensive experiments with real-life datasets on EA benchmarks offer insight into the proposed framework, and suggest that it is capable of outperforming the state-of-the-art scalable EA framework by up to 8 times in terms of Hits@1.

Categorization of some popular EA methods.

Besides the modifications on encoders, some EA methods adopt iterative strategies to generate semi-supervised data due to the lack of labeled entities.
Some EA methods propose that introducing literal information (e.g., entity names) could provide a multi-aspect view for alignment models.
However, it should be noted that not all KGs contain literal information, especially in practical applications.
Table {{table:4eecf571-48e0-4004-979a-23ba43180c1b}}  categorizes some popular EA methods based on their encoders and whether using iterative strategies or literal information.

According to the categorization in Table {{table:4eecf571-48e0-4004-979a-23ba43180c1b}} , we compare LightEA against the following three groups of advanced EA methods:
(1) Basic:
MtransE {{cite:b4ce21e}}, GCN-Align {{cite:5739122}}, RSNs {{cite:2dabe03}}, HyperKA {{cite:a62c096}}.
(2) Iterative:
BootEA {{cite:569519b}}, TransEdge {{cite:cf2f5fa}}, MRAEA {{cite:4d603b4}}, Dual-AMN {{cite:b5b9e8a}}, LargeEA {{cite:a745c2d}}, ClusterEA {{cite:765b489}}.
(3) Literal:
GM-Align {{cite:1221a94}}, RDGCN {{cite:4bf6cf8}}, SEU {{cite:ffe8620}}, EASY {{cite:502232d}}.


Distribution-Aware Coordinate Representation for Human Pose Estimation

Abstract: While being the de facto standard coordinate representation in human pose estimation, heatmap is never systematically investigated in the literature, to our best knowledge. This work fills this gap by studying the coordinate representation with a particular focus on the heatmap. Interestingly, we found that the process of decoding the predicted heatmaps into the final joint coordinates in the original image space is surprisingly significant for human pose estimation performance, which nevertheless was not recognised before. In light of the discovered importance, we further probe the design limitations of the standard coordinate decoding method widely used by existing methods, and propose a more principled distribution-aware decoding method. Meanwhile, we improve the standard coordinate encoding process (i.e. transforming ground-truth coordinates to heatmaps) by generating accurate heatmap distributions for unbiased model training. Taking the two together, we formulate a novel Distribution-Aware coordinate Representation of Keypoint (DARK) method. Serving as a model-agnostic plug-in, DARK significantly improves the performance of a variety of state-of-the-art human pose estimation models. Extensive experiments show that DARK yields the best results on two common benchmarks, MPII and COCO, consistently validating the usefulness and effectiveness of our novel coordinate representation idea.

InfAnFace: Bridging the Infant–Adult Domain Gap in Facial Landmark Estimation in the Wild

Abstract: We lay the groundwork for research in the algorithmic comprehension of infant faces, in anticipation of applications from healthcare to psychology, especially in the early prediction of developmental disorders. Specifically, we introduce the first-ever dataset of infant faces annotated with facial landmark coordinates and pose attributes, demonstrate the inadequacies of existing facial landmark estimation algorithms in the infant domain, and train new state-of-the-art models that significantly improve upon those algorithms using domain adaptation techniques. We touch on the closely related task of facial detection for infants, and also on a challenging case study of infrared baby monitor images gathered by our lab as part of in-field research into the aforementioned developmental issues1

Invariant Representation Learning for Infant Pose Estimation with Small Data

Abstract: Infant motion analysis is a topic with critical importance in early childhood development studies. However, while the applications of human pose estimation have become more and more broad, models trained on large-scale adult pose datasets are barely successful in estimating infant poses due to the significant differences in their body ratio and the versatility of their poses. Moreover, the privacy and security considerations hinder the availability of adequate infant pose data required for training of a robust model from scratch. To address this problem, this paper presents (1) building and publicly releasing a hybrid synthetic and real infant pose (SyRIP) dataset with small yet diverse real infant images as well as generated synthetic infant poses and (2) a multi-stage invariant representation learning strategy that could transfer the knowledge from the adjacent domains of adult poses and synthetic infant images into our fine-tuned domain-adapted infant pose (FiDIP) estimation model. In our ablation study, with identical network structure, models trained on SyRIP dataset show noticeable improvement over the ones trained on the only other public infant pose datasets. Integrated with pose estimation backbone networks with varying complexity, FiDIP performs consistently better than the fine-tuned versions of those models. One of our best infant pose estimation performers on the state-of-the-art DarkPose model shows mean average precision (mAP) of 93.6.

Configuration of our landmark estimation models

To study the effectiveness of algorithmic assessment of our geometric measures, we perform experiments with two sets of pose estimation models, the first established models trained largely on adult data, and the second recent models designed specifically for the infant domain. The models and training data are summarized in Table {{table:b25b3d4c-3cc4-4b21-97ee-2f05a9549942}} .


Time-coupling sub-gradients; sum-minimization problems where certain parameters are to be estimated

Obtaining local optimum in a fast and efficient way

Converging to global optimum for (pseudo)convex functions under certain conditions

Only locally optimal for non-(pseudo)convex functions; iterative computation

Approximation and learning by greedy algorithms

Abstract: We consider the problem of approximating a given element $f$ from a Hilbert space $\mathcal{H}$ by means of greedy algorithms and the application of such procedures to the regression problem in statistical learning theory. We improve on the existing theory of convergence rates for both the orthogonal greedy algorithm and the relaxed greedy algorithm, as well as for the forward stepwise projection algorithm. For all these algorithms, we prove convergence results for a variety of function classes and not simply those that are related to the convex hull of the dictionary. We then show how these bounds for convergence rates lead to a new theory for the performance of greedy algorithms in learning. In particular, we build upon the results in [IEEE Trans. Inform. Theory 42 (1996) 2118--2132] to construct learning algorithms based on greedy approximations which are universally consistent and provide provable convergence rates for large classes of functions. The use of greedy algorithms in the context of learning is very appealing since it greatly reduces the computational burden when compared with standard model selection using general dictionaries.

Robust Smart-Grid Powered Cooperative Multipoint Systems

Abstract: A framework is introduced to integrate renewable energy sources (RES) and dynamic pricing capabilities of the smart grid into beamforming designs for coordinated multi-point (CoMP) downlink communication systems. To this end, novel models are put forth to account for harvesting, storage of nondispatchable RES, time-varying energy pricing, as well as stochastic wireless channels. Building on these models, robust energy management and transmit-beamforming designs are developed to minimize the worst-case energy cost subject to the worst-case user QoS guarantees for the CoMP downlink. Leveraging pertinent tools, this task is formulated as a convex problem. A Lagrange dual based subgradient iteration is then employed to find the desired optimal energy-management strategy and transmit-beamforming vectors. Numerical results are provided to demonstrate the merits of the proposed robust designs.

Energy Group Buying With Loading Sharing for Green Cellular Networks

Abstract: In the emerging hybrid electricity market, mobile network operators (MNOs) of cellular networks can make day-ahead energy purchase commitments at low prices and real-time flexible energy purchase at high prices. To minimize electricity bills, it is essential for MNOs to jointly optimize the day-ahead and real-time energy purchase based on their time-varying wireless traffic load. In this paper, we consider two different MNOs coexisting in the same area, and exploit their collaboration in both energy purchase and wireless load sharing for energy cost saving. Specifically, we propose a new approach named energy group buying with load sharing, in which the two MNOs are aggregated as a single group to make the day-ahead and real-time energy purchase, and their base stations (BSs) share the wireless traffic to maximally turn lightly-loaded BSs into sleep mode. When the two MNOs belong to the same entity and aim to minimize their total energy cost, we use the two-stage stochastic programming to obtain the optimal day-ahead and real-time energy group buying jointly with wireless load sharing. When the two MNOs belong to different entities and are self-interested in minimizing their individual energy costs, we propose a novel repeated Nash bargaining scheme for them to negotiate and share their energy costs under energy group buying and load sharing. Our proposed repeated Nash bargaining scheme is shown to achieve Pareto-optimal and fair energy cost reductions for both MNOs.

Analysis of algorithms

Featuring (part of) the system model in Fig. {{figure:60640b8f-94a4-4b02-8e6c-d7d6521011c2}} , many recent works focus on enhancing system performances by
minimizing energy consumption and operational cost {{cite:5527a95}}, {{cite:e3039bb}}, {{cite:aa5e277}}, {{cite:afc396a}}, {{cite:73d96d6}}, {{cite:fab8764}}, {{cite:dd95e50}}, {{cite:d30e17b}},
or maximizing system's efficiency and operator's utility {{cite:ff9db75}}, {{cite:14065be}}, {{cite:d348617}}, {{cite:321e0d5}}.
State-of-the-art techniques and methodologies, such as game theory,
DP {{cite:1143ec0}}, {{cite:af04c50}}, {{cite:4efc7f6}}, and stochastic gradient descent (SGD) {{cite:3bf5a18}}, {{cite:3527aaf}}, {{cite:c1a7a4f}}, {{cite:c7ea831}}, can be leveraged.
The objectives and the solving techniques of the optimization problems are categorized and summarized in Table {{table:1f052ef9-fc51-4b53-86c5-469d32f9e1fe}} .
Game theory is applied when competition or cooperation exists in the system consisting of BSs and electricity retailers.
DP usually decomposes a sophisticated problem into a set of much easier subproblems with recurrence expressions.
This relationship is known as Bellman optimality equation.
SGD is usually applied to achieve the asymptotically minimum time-average expenditure of a network
in no need of any prior knowledge
of system's randomness (such as the EH amounts and electricity prices).
The real gradient value of the objective is replaced with the gradient from the training set,
which helps to decouple the optimization and constraints over time.
It is proven that the optimality loss of the objective can asymptotically diminish by reducing the stepsize of SGD {{cite:eb11ce2}}.
The application scenarios, merits and disadvantages of some popular algorithms are summarized in Table {{table:f876c687-b8c4-4902-897a-fd01a6cf3da8}} .
Conditions and auxiliary methods applied to solve the problems are summarized in Table {{table:23fe929d-7dec-4c0d-b3cf-f400d676b8bd}} .


Vehicular Edge Computing via Deep Reinforcement Learning

Abstract: The smart vehicles construct Vehicle of Internet which can execute various intelligent services. Although the computation capability of the vehicle is limited, multi-type of edge computing nodes provide heterogeneous resources for vehicular services.When offloading the complicated service to the vehicular edge computing node, the decision should consider numerous factors.The offloading decision work mostly formulate the decision to a resource scheduling problem with single or multiple objective function and some constraints, and explore customized heuristics algorithms. However, offloading multiple data dependency tasks in a service is a difficult decision, as an optimal solution must understand the resource requirement, the access network, the user mobility, and importantly the data dependency. Inspired by recent advances in machine learning, we propose a knowledge driven (KD) service offloading decision framework for Vehicle of Internet, which provides the optimal policy directly from the environment. We formulate the offloading decision of multi-task in a service as a long-term planning problem, and explores the recent deep reinforcement learning to obtain the optimal solution. It considers the future data dependency of the following tasks when making decision for a current task from the learned offloading knowledge. Moreover, the framework supports the pre-training at the powerful edge computing node and continually online learning when the vehicular service is executed, so that it can adapt the environment changes and learns policy that are sensible in hindsight. The simulation results show that KD service offloading decision converges quickly, adapts to different conditions, and outperforms the greedy offloading decision algorithm.

SDN-Based Resource Management for Autonomous Vehicular Networks: A Multi-Access Edge Computing Approach

Abstract: Enabling high-definition (HD)-map-assisted cooperative driving among autonomous vehicles (AVs) to improve the navigation safety faces technical challenges due to increased communication traffic volume for data dissemination and increased number of computing/storing tasks on AVs. In this article, a new architecture that combines multi-access edge computing (MEC) and software-defined networking (SDN) is proposed for flexible resource management and enhanced resource utilization. With MEC, the interworking of multiple wireless access technologies can be realized to exploit the diversity gain over a wide range of radio spectrum, and at the same time, computing/storing tasks of an AV are collaboratively processed by servers and other AVs. Moreover, by enabling SDN and network function virtualization (NFV) control modules at each cloud-computing and MEC server, an efficient resource allocation framework is proposed to enhance global resource sharing among different network infrastructures. A case study is presented to demonstrate the effectiveness of the proposed resource allocation framework.

VEC edge nodes

The edge nodes are known for the nodes that promptly compute, store and transmit data located at distributed edge networks. In general, the vehicles, which are over utilised or faced a slow computation, offload (i.e., upload) their data to nearby edge nodes. On one hand, there is a case of contents like movies caching and delivery from the edge nodes instead of directly fetching from the centre cloud. Table {{table:f439a5ba-31fb-416e-945d-8145bfbafb42}}  presents the types of edge nodes employed in the ComOf and CachDel for VEC. The edge nodes are commonly characterised into two types which are:

VENs are smart vehicles equipped with communication modules and OBUs with computing and low storage capabilities as the concept of Vehicle as a Resource (VaaR) {{cite:307df16}}. As shown in Table {{table:f439a5ba-31fb-416e-945d-8145bfbafb42}} , the types of VENs considered in the offloading and caching works are cars, UAVs {{cite:9b4a6e1}} or buses {{cite:50fd930}}. The mobility of fog nodes explicitly expands new opportunities, such as on-demand computing where the moving vehicles may offer ubiquitously their available computational resources, particularly at the area without any SEN, like in rural areas. Leveraging Parked Vehicles (PV) {{cite:c37670b}} at the parking area, for examples in the airport or shopping mall, as a primary VEN for computing and caching is also a promising solution for VEC. Also, parked vehicles integrated with fog node controller as a data centre can expand the storage capacity for improving the performance of delivery services in vehicular networks{{cite:9696555}}, {{cite:c37670b}}. However, the key limitation is that a cluster of parked cars cannot be fully used as supplemental resources when their power are turned off. Although VEN has a small storage capacity compared with SEN, the vehicles are likely to offer a cheaper computational cost. Another significant issue is that not all VENs including the PV are keen to offer their computational resources to other vehicles. To solve this, a good reputation VEN is necessary to receive a token or reward based on the acquired utility.

In general, the MEC in non-vehicular networks, e.g., mobile cellular networks, served by MEC servers with several options to deploy, such as co-located with the BS {{cite:41b9af0}}, radio network controllers {{cite:4ee2726}} and can be farther from the UE at the core network {{cite:b154058}}. On the other hand, considerable VEC offloading works assumed deploying the server at the RSU (see Table {{table:f439a5ba-31fb-416e-945d-8145bfbafb42}} ) besides the BS. Another primary different is various kind of vehicles with specific mobility are used as edge nodes instead of mobile devices, which owing to a fast fading channel that affects the VEC performance. In terms of the communication protocol, DSRC is mostly used for the V2I communication for the case of RSU as edge nodes. Nevertheless, other cellular networks as highlighted previously are also evaluated in the VEC works. In VEC, the requesting vehicles are surrounded by several edge nodes (i.e., SEN and VEN) for the offloading decision. Identifying a reliable edge node for consistent connectivity is a big challenge because of the vehicle speed. Therefore, the edge nodes are currently characterised and selected in terms of available workload, central processing unit (CPU) processing rate, energy consumption, radio transmission rate, offloading cost, security and reputations.


Maximise network throughput with guaranteed QoS

minimise the total computation overhead (i.e. weighted-sum of task completion time and monetary cost for using cloud resources)

Linear programming relaxation and first-order Taylor series approximation and an alternate concave search (ACS) algorithm

Karush-Kuhn-Tucker (KKT) conditions and branch-and-bound algorithm

Optimal transmit power and network slicing

Cost of resources (vehicles and core cloud)

No resource allocation from other vehicles

Spectrum Resource Management for Multi-Access Edge Computing in Autonomous Vehicular Networks

Abstract: In this paper, a dynamic spectrum management framework is proposed to improve spectrum resource utilization in a multi-access edge computing (MEC) in autonomous vehicular network (AVNET). To support the increasing data traffic and guarantee quality-of-service (QoS), spectrum slicing, spectrum allocating, and transmit power controlling are jointly considered. Accordingly, three non-convex network utility maximization problems are formulated to slice spectrum among BSs, allocate spectrum among autonomous vehicles (AVs) associated with a BS, and control transmit powers of BSs, respectively. Via linear programming relaxation and first-order Taylor series approximation, these problems are transformed into tractable forms and then are jointly solved through an alternate concave search (ACS) algorithm. As a result, optimal spectrum slicing ratios among BSs, optimal BS-vehicle association patterns, optimal fractions of spectrum resources allocated to AVs, and optimal transmit powers of BSs are obtained. Based on our simulation, a high aggregate network utility is achieved by the proposed spectrum management scheme compared with two existing schemes.

Joint Node Selection and Resource Allocation for Task Offloading in Scalable Vehicle-Assisted Multi-Access Edge Computing

Abstract: The resource limitation of multi-access edge computing (MEC) is one of the major issues in order to provide low-latency high-reliability computing services for Internet of Things (IoT) devices. Moreover, with the steep rise of task requests from IoT devices, the requirement of computation tasks needs dynamic scalability while using the potential of offloading tasks to mobile volunteer nodes (MVNs). We, therefore, propose a scalable vehicle-assisted MEC (SVMEC) paradigm, which cannot only relieve the resource limitation of MEC but also enhance the scalability of computing services for IoT devices and reduce the cost of using computing resources. In the SVMEC paradigm, a MEC provider can execute its users’ tasks by choosing one of three ways: (i) Do itself on local MEC, (ii) offload to the remote cloud, and (iii) offload to the MVNs. We formulate the problem of joint node selection and resource allocation as a Mixed Integer Nonlinear Programming (MINLP) problem, whose major objective is to minimize the total computation overhead in terms of the weighted-sum of task completion time and monetary cost for using computing resources. In order to solve it, we adopt alternative optimization techniques by decomposing the original problem into two sub-problems: Resource allocation sub-problem and node selection sub-problem. Simulation results demonstrate that our proposed scheme outperforms the existing schemes in terms of the total computation overhead.

Hybrid Optimisation in VEC

The authors in {{cite:1b97a95}}, {{cite:951f196}} proposed a joint optimisation for selection, computation and offloading algorithm (JSCO) to maximise QoS-based system utility. The system model consists of multiple VEC servers and multiple vehicles as illustrated in Figure {{figure:f486da48-75b7-4ab5-b2e9-1a47c832a60c}} . The problem is solved using relaxation and rounding method and Lagrange multipliers. The proposed JSCO demonstrated the fair allocation of VEC servers to the requesting vehicles besides optimum computation resource and offloading ratio. Similarly, {{cite:de61158}} presented joint offloading proportion and resource allocation optimisation (JOPRAO) for all vehicles to minimise the task completion time. The key difference of both works is the offloading ratio whereby {{cite:1b97a95}} introduced the amount of data offloads to the VEC server and computes the remaining locally, while {{cite:de61158}} offloaded the sub-task between a vehicle edge node and MEC server. The offloading proportion for vehicle increases with the computation capability of the vehicle (i.e., MHz) and is enabled to achieve lower delay {{cite:de61158}}, but is not specifically discussed in {{cite:1b97a95}}. Another work {{cite:fa49048}} formulated a network utility maximization consisted of two-level resource allocation, system bandwidth slicing, and resources partitioning for AVs. The work is then extended to three aggregate network-wide utility maximization problems (refer to Table {{table:78aeed9a-f22c-401e-841f-6da4ff18bffe}} ) focusing on transmit power control and solved by Linear programming relaxation and first-order Taylor series approximation and an alternate concave search algorithm {{cite:b9c9062}}. However, both works {{cite:fa49048}}, {{cite:b9c9062}} do not consider AVs as computing edge nodes. Unlike others, {{cite:307df16}} solved the offloading problem by joining two different QoS parameters, which were latency and quality task allocation using Linear Programming based Optimization and Binary Particle Swarm Optimization prior to dynamic task allocation approach {{cite:3d10130}}. The study might far more convincing if the authors had attempted the multi vehicular cloudlets.


HoloClean: Holistic Data Repairs with Probabilistic Inference

Abstract: We introduce HoloClean, a framework for holistic data repairing driven by probabilistic inference. HoloClean unifies existing qualitative data repairing approaches, which rely on integrity constraints or external data sources, with quantitative data repairing methods, which leverage statistical properties of the input data. Given an inconsistent dataset as input, HoloClean automatically generates a probabilistic program that performs data repairing. Inspired by recent theoretical advances in probabilistic inference, we introduce a series of optimizations which ensure that inference over HoloClean's probabilistic model scales to instances with millions of tuples. We show that HoloClean scales to instances with millions of tuples and find data repairs with an average precision of ~90% and an average recall of above ~76% across a diverse array of datasets exhibiting different types of errors. This yields an average F1 improvement of more than 2x against state-of-the-art methods.

Advancements of Outlier Detection: A Survey

Abstract: Outlier detection is an important research problem in data mining that aims to discover useful abnormal and irregular patterns hidden in large datasets. In this paper, we present a survey of outlier detection techniques to reflect the recent advancements in this field. The survey will not only cover the traditional outlier detection methods for static and low dimensional datasets but also review the more recent developments that deal with more complex outlier detection problems for dynamic/streaming and high-dimensional datasets.

Error detection and repair tools used as baselines

We compare the performance of AutoCure to a wide collection of baseline methods composed of different error detection and repair tools. Table {{table:f1afecea-04f8-4b3b-a7e5-b5c946adce71}}  lists the error detection and repair methods. The combination of each detection and repair method represents a baseline method. For instance, the baseline B2 implies detecting errors in a dirty data set using dBoost and generating repair candidates using an ML-based imputation algorithm. In the list of repair methods, we also include the ground truth and dirty cases to show the performance upper- and lower-bound. The standard imputer replaces the erroneous data instances with the mean values of the numerical columns and with a dummy string for the categorical columns. Alternatively, the ML-based imputer employs a decision tree model to predict the erroneous numerical instances, while it employs missForest {{cite:98c1f9b}} for categorical data. The evaluation metrics used in the evaluation comprise the detection accuracy, i.e., expressed in terms of the detection precision, recall, and F1 score, the training time, and the predictive accuracy.


Spatio-Temporal Graph Convolutional Networks: A Deep Learning Framework for Traffic Forecasting

Abstract: Timely accurate traffic forecast is crucial for urban traffic control and guidance. Due to the high nonlinearity and complexity of traffic flow, traditional methods cannot satisfy the requirements of mid-and-long term prediction tasks and often neglect spatial and temporal dependencies. In this paper, we propose a novel deep learning framework, Spatio-Temporal Graph Convolutional Networks (STGCN), to tackle the time series prediction problem in traffic domain. Instead of applying regular convolutional and recurrent units, we formulate the problem on graphs and build the model with complete convolutional structures, which enable much faster training speed with fewer parameters. Experiments show that our model STGCN effectively captures comprehensive spatio-temporal correlations through modeling multi-scale traffic networks and consistently outperforms state-of-the-art baselines on various real-world traffic datasets.

Adaptive Graph Convolutional Recurrent Network for Traffic Forecasting

Abstract: Modeling complex spatial and temporal correlations in the correlated time series data is indispensable for understanding the traffic dynamics and predicting the future status of an evolving traffic system. Recent works focus on designing complicated graph neural network architectures to capture shared patterns with the help of pre-defined graphs. In this paper, we argue that learning node-specific patterns is essential for traffic forecasting while the pre-defined graph is avoidable. To this end, we propose two adaptive modules for enhancing Graph Convolutional Network (GCN) with new capabilities: 1) a Node Adaptive Parameter Learning (NAPL) module to capture node-specific patterns; 2) a Data Adaptive Graph Generation (DAGG) module to infer the inter-dependencies among different traffic series automatically. We further propose an Adaptive Graph Convolutional Recurrent Network (AGCRN) to capture fine-grained spatial and temporal correlations in traffic series automatically based on the two modules and recurrent networks. Our experiments on two real-world traffic datasets show AGCRN outperforms state-of-the-art by a significant margin without pre-defined graphs about spatial connections.

Graph WaveNet for Deep Spatial-Temporal Graph Modeling

Abstract: Spatial-temporal graph modeling is an important task to analyze the spatial relations and temporal trends of components in a system. Existing approaches mostly capture the spatial dependency on a fixed graph structure, assuming that the underlying relation between entities is pre-determined. However, the explicit graph structure (relation) does not necessarily reflect the true dependency and genuine relation may be missing due to the incomplete connections in the data. Furthermore, existing methods are ineffective to capture the temporal trends as the RNNs or CNNs employed in these methods cannot capture long-range temporal sequences. To overcome these limitations, we propose in this paper a novel graph neural network architecture, Graph WaveNet, for spatial-temporal graph modeling. By developing a novel adaptive dependency matrix and learn it through node embedding, our model can precisely capture the hidden spatial dependency in the data. With a stacked dilated 1D convolution component whose receptive field grows exponentially as the number of layers increases, Graph WaveNet is able to handle very long sequences. These two components are integrated seamlessly in a unified framework and the whole framework is learned in an end-to-end manner. Experimental results on two public traffic network datasets, METR-LA and PEMS-BAY, demonstrate the superior performance of our algorithm.

Spatial-Temporal Graph ODE Networks for Traffic Flow Forecasting

Abstract: Spatial-temporal forecasting has attracted tremendous attention in a wide range of applications, and traffic flow prediction is a canonical and typical example. The complex and long-range spatial-temporal correlations of traffic flow bring it to a most intractable challenge. Existing works typically utilize shallow graph convolution networks (GNNs) and temporal extracting modules to model spatial and temporal dependencies respectively. However, the representation ability of such models is limited due to: (1) shallow GNNs are incapable to capture long-range spatial correlations, (2) only spatial connections are considered and a mass of semantic connections are ignored, which are of great importance for a comprehensive understanding of traffic networks. To this end, we propose Spatial-Temporal Graph Ordinary Differential Equation Networks (STGODE). Specifically, we capture spatial-temporal dynamics through a tensor-based ordinary differential equation (ODE), as a result, deeper networks can be constructed and spatial-temporal features are utilized synchronously. To understand the network more comprehensively, semantical adjacency matrix is considered in our model, and a well-design temporal dialated convolution structure is used to capture long term temporal dependencies. We evaluate our model on multiple real-world traffic datasets and superior performance is achieved over state-of-the-art baselines.

Graph Neural Controlled Differential Equations for Traffic Forecasting

Abstract: Traffic forecasting is one of the most popular spatio-temporal tasks in the field of machine learning. A prevalent approach in the field is to combine graph convolutional networks and recurrent neural networks for the spatio-temporal processing. There has been fierce competition and many novel methods have been proposed. In this paper, we present the method of spatio-temporal graph neural controlled differential equation (STG-NCDE). Neural controlled differential equations (NCDEs) are a breakthrough concept for processing sequential data. We extend the concept and design two NCDEs: one for the temporal processing and the other for the spatial processing. After that, we combine them into a single framework. We conduct experiments with 6 benchmark datasets and 20 baselines. STG-NCDE shows the best accuracy in all cases, outperforming all those 20 baselines by non-trivial margins.

Social-STGCNN: A Social Spatio-Temporal Graph Convolutional Neural Network for Human Trajectory Prediction

Abstract: Better machine understanding of pedestrian behaviors enables faster progress in modeling interactions between agents such as autonomous vehicles and humans. Pedestrian trajectories are not only influenced by the pedestrian itself but also by interaction with surrounding objects. Previous methods modeled these interactions by using a variety of aggregation methods that integrate different learned pedestrians states. We propose the Social Spatio-Temporal Graph Convolutional Neural Network (Social-STGCNN), which substitutes the need of aggregation methods by modeling the interactions as a graph. Our results show an improvement over the state of art by 20% on the Final Displacement Error (FDE) and an improvement on the Average Displacement Error (ADE) with 8.5 times less parameters and up to 48 times faster inference speed than previously reported methods. In addition, our model is data efficient, and exceeds previous state of the art on the ADE metric with only 20% of the training data. We propose a kernel function to embed the social interactions between pedestrians within the adjacency matrix. Through qualitative analysis, we show that our model inherited social behaviors that can be expected between pedestrians trajectories. Code is available at https://github.com/abduallahmohamed/Social-STGCNN.

Recursive Social Behavior Graph for Trajectory Prediction

Abstract: Social interaction is an important topic in human trajectory prediction to generate plausible paths. In this paper, we present a novel insight of group-based social interaction model to explore relationships among pedestrians. We recursively extract social representations supervised by group-based annotations and formulate them into a social behavior graph, called Recursive Social Behavior Graph. Our recursive mechanism explores the representation power largely. Graph Convolutional Neural Network then is used to propagate social interaction information in such a graph. With the guidance of Recursive Social Behavior Graph, we surpass state-of-the-art methods on ETH and UCY dataset for 11.1% in ADE and 10.8% in FDE in average, and successfully predict complex social behaviors.

Deep Multi-View Spatiotemporal Virtual Graph Neural Network for Significant Citywide Ride-hailing Demand Prediction

Abstract: Urban ride-hailing demand prediction is a crucial but challenging task for intelligent transportation system construction. Predictable ride-hailing demand can facilitate more reasonable vehicle scheduling and online car-hailing platform dispatch. Conventional deep learning methods with no external structured data can be accomplished via hybrid models of CNNs and RNNs by meshing plentiful pixel-level labeled data, but spatial data sparsity and limited learning capabilities on temporal long-term dependencies are still two striking bottlenecks. To address these limitations, we propose a new virtual graph modeling method to focus on significant demand regions and a novel Deep Multi-View Spatiotemporal Virtual Graph Neural Network (DMVST-VGNN) to strengthen learning capabilities of spatial dynamics and temporal long-term dependencies. Specifically, DMVST-VGNN integrates the structures of 1D Convolutional Neural Network, Multi Graph Attention Neural Network and Transformer layer, which correspond to short-term temporal dynamics view, spatial dynamics view and long-term temporal dynamics view respectively. In this paper, experiments are conducted on two large-scale New York City datasets in fine-grained prediction scenes. And the experimental results demonstrate effectiveness and superiority of DMVST-VGNN framework in significant citywide ride-hailing demand prediction.

Summary of graph models for traffic analysis.

In this section, we first conclude the main GNN research directions in traffic domain, and then we summarize the typical graph construction processes in different traffic scenes and datasets. Finally, we list the classical GNN workflows for dealing with tasks in traffic networks. A summary is provided in Table {{table:23ffdaca-4489-485e-bfc1-32d103147dfb}} .


Biases can arise from prioritizing certain use cases or designing user interfaces for specific demographics or industries, inadvertently reinforcing existing biases and excluding different perspectives.

Developers might implement policies that prevent (or encourage) a given model behavior. For example, guardrails that modulate the behavior of ChatGPT and Bing-AI aimed to mitigate unintended toxic model behaviors or prevent malicious abuse.

Inherent Trade-Offs in the Fair Determination of Risk Scores

Abstract: Recent discussion in the public sphere about algorithmic classification has involved tension between competing notions of what it means for a probabilistic classification to be fair to different groups. We formalize three fairness conditions that lie at the heart of these debates, and we prove that except in highly constrained special cases, there is no method that can satisfy these three conditions simultaneously. Moreover, even satisfying all three conditions approximately requires that the data lie in an approximate version of one of the constrained special cases identified by our theorem. These results suggest some of the ways in which key notions of fairness are incompatible with each other, and hence provide a framework for thinking about the trade-offs between them.

Towards A Rigorous Science of Interpretable Machine Learning

Abstract: As machine learning systems become ubiquitous, there has been a surge of interest in interpretable machine learning: systems that provide explanation for their outputs. These explanations are often used to qualitatively assess other criteria such as safety or non-discrimination. However, despite the interest in interpretability, there is very little consensus on what interpretable machine learning is and how it should be measured. In this position paper, we first define interpretability and describe when interpretability is needed (and when it is not). Next, we suggest a taxonomy for rigorous evaluation and expose open questions towards a more rigorous science of interpretable machine learning.

Factors Contributing to Bias in AI Models

Factors contributing to bias in Large Language Models

Bias, in the context of large language models like GPT-4 {{cite:d60bfa6}}, {{cite:d8ceaa0}}, {{cite:b848a03}} and predecessors {{cite:4636cf2}}, {{cite:fbf2b46}}, {{cite:85f2053}}, or other state-of-the-art alternatives {{cite:b2564e6}}, {{cite:49320a5}} (including multimodal variants {{cite:0385559}}), can be defined as the presence of systematic misrepresentations, attribution errors, or factual distortions that result in favoring certain groups or ideas, perpetuating stereotypes, or making incorrect assumptions based on learned patterns. Biases in such models can arise due to several factors (cf., Table {{table:b39bef2b-cbc5-437a-84b0-26f31a6e0e0c}} ).


Large language models may learn and perpetuate cultural stereotypes or biases, as they are often present in the data used for training. This can result in the model producing outputs that reinforce or exacerbate existing cultural prejudices.

Since the majority of the internet's content is in English or a few other dominant languages, large language models tend to be more proficient in these languages. This can lead to biased performance and a lack of support for low-resource languages or minority dialects.

Identifying and Reducing Gender Bias in Word-Level Language Models

Abstract: Many text corpora exhibit socially problematic biases, which can be propagated or amplified in the models trained on such data. For example, doctor cooccurs more frequently with male pronouns than female pronouns. In this study we (i) propose a metric to measure gender bias; (ii) measure bias in a text corpus and the text generated from a recurrent neural network language model trained on the text corpus; (iii) propose a regularization loss term for the language model that minimizes the projection of encoder-trained embeddings onto an embedding subspace that encodes gender; (iv) finally, evaluate efficacy of our proposed method on reducing gender bias. We find this regularization method to be effective in reducing gender bias up to an optimal weight assigned to the loss term, beyond which the model becomes unstable as the perplexity increases. We replicate this study on three training corpora---Penn Treebank, WikiText-2, and CNN/Daily Mail---resulting in similar conclusions.

Word Translation Without Parallel Data

Abstract: State-of-the-art methods for learning cross-lingual word embeddings have relied on bilingual dictionaries or parallel corpora. Recent studies showed that the need for parallel data supervision can be alleviated with character-level information. While these methods showed encouraging results, they are not on par with their supervised counterparts and are limited to pairs of languages sharing a common alphabet. In this work, we show that we can build a bilingual dictionary between two languages without using any parallel corpora, by aligning monolingual word embedding spaces in an unsupervised way. Without using any character information, our model even outperforms existing supervised methods on cross-lingual tasks for some language pairs. Our experiments demonstrate that our method works very well also for distant language pairs, like English-Russian or English-Chinese. We finally describe experiments on the English-Esperanto low-resource language pair, on which there only exists a limited amount of parallel data, to show the potential impact of our method in fully unsupervised machine translation. Our code, embeddings and dictionaries are publicly available.

Types of Biases in Large Language Models

Types of biases in Large Language Models

Large language models, which are commonly trained from vast amounts of text data present on the Internet, inevitably absorb the biases present in such data sources. These biases can take various forms (cf., Table {{table:ba32206b-2852-4edd-bbb6-376c66cfe6d7}} ).


Models generalize knowledge from training data to new inputs, potentially leading to biased behavior if the data contains biases. This raises concerns about perpetuating biases, even if training data has been cleaned and filtered.

Unanticipated capabilities and biases may emerge in large language models due to complex interactions between model parameters and biased training data. It has been proven difficult to predict or control these emergent biases.

Biases in AI systems may have non-linear real-world impact, making it difficult to predict their consequences: small model biases may have massive negative effects, whereas large model biases might not cause significant consequences.

Annotation Artifacts in Natural Language Inference Data

Abstract: Large-scale datasets for natural language inference are created by presenting crowd workers with a sentence (premise), and asking them to generate three new sentences (hypotheses) that it entails, contradicts, or is logically neutral with respect to. We show that, in a significant portion of such data, this protocol leaves clues that make it possible to identify the label by looking only at the hypothesis, without observing the premise. Specifically, we show that a simple text categorization model can correctly classify the hypothesis alone in about 67% of SNLI (Bowman et. al, 2015) and 53% of MultiNLI (Williams et. al, 2017). Our analysis reveals that specific linguistic phenomena such as negation and vagueness are highly correlated with certain inference classes. Our findings suggest that the success of natural language inference models to date has been overestimated, and that the task remains a hard open problem.

Chain of Thought Prompting Elicits Reasoning in Large Language Models

Abstract: We explore how generating a chain of thought -- a series of intermediate reasoning steps -- significantly improves the ability of large language models to perform complex reasoning. In particular, we show how such reasoning abilities emerge naturally in sufficiently large language models via a simple method called chain of thought prompting, where a few chain of thought demonstrations are provided as exemplars in prompting. Experiments on three large language models show that chain of thought prompting improves performance on a range of arithmetic, commonsense, and symbolic reasoning tasks. The empirical gains can be striking. For instance, prompting a 540B-parameter language model with just eight chain of thought exemplars achieves state of the art accuracy on the GSM8K benchmark of math word problems, surpassing even finetuned GPT-3 with a verifier.

Path-Specific Counterfactual Fairness

Abstract: We consider the problem of learning fair decision systems from data in which a sensitive attribute might affect the decision along both fair and unfair pathways. We introduce a counterfactual approach to disregard effects along unfair pathways that does not incur in the same loss of individual-specific information as previous approaches. Our method corrects observations adversely affected by the sensitive attribute, and uses these to form a decision. We leverage recent developments in deep learning and approximate inference to develop a VAE-type method that is widely applicable to complex nonlinear models.

Types of biases in large language models and their descriptions.

During the training process, generative language models are exposed to billions of sentences and phrases, allowing them to learn the intricate relationships between words, grammar, context, and meaning {{cite:378c11c}}, {{cite:54c8642}}. As they process the text data, they gradually acquire natural language generation capabilities, enabling them to produce coherent and contextually relevant responses to various inputs. However, some capabilities of these models can lead to bias (cf., Table {{table:09bbfc71-08b8-481d-adbf-5c5b24796785}} ):


Hyperbolic Graph Convolutional Neural Networks

Abstract: Graph convolutional neural networks (GCNs) embed nodes in a graph into Euclidean space, which has been shown to incur a large distortion when embedding real-world graphs with scale-free or hierarchical structure. Hyperbolic geometry offers an exciting alternative, as it enables embeddings with much smaller distortion. However, extending GCNs to hyperbolic geometry presents several unique challenges because it is not clear how to define neural network operations, such as feature transformation and aggregation, in hyperbolic space. Furthermore, since input features are often Euclidean, it is unclear how to transform the features into hyperbolic embeddings with the right amount of curvature. Here we propose Hyperbolic Graph Convolutional Neural Network (HGCN), the first inductive hyperbolic GCN that leverages both the expressiveness of GCNs and hyperbolic geometry to learn inductive node representations for hierarchical and scale-free graphs. We derive GCN operations in the hyperboloid model of hyperbolic space and map Euclidean input features to embeddings in hyperbolic spaces with different trainable curvature at each layer. Experiments demonstrate that HGCN learns embeddings that preserve hierarchical structure, and leads to improved performance when compared to Euclidean analogs, even with very low dimensional embeddings: compared to state-of-the-art GCNs, HGCN achieves an error reduction of up to 63.1% in ROC AUC for link prediction and of up to 47.5% in F1 score for node classification, also improving state-of-the art on the Pubmed dataset.

Hyperbolic Graph Attention Network

Abstract: Graph neural network (GNN) has shown superior performance in dealing with structured graphs, which has attracted considerable research attention recently. Most of the existing GNNs are designed in euclidean spaces; however, real-world spatial structured data can be non-euclidean surfaces (e.g., hyperbolic spaces). For example, biologists may inspect the geometric shape of a protein surface to determine its interaction with other biomolecules for drug discovery. Although there is growing research on generalizing GNNs to non-euclidean surfaces, the works in these fields are still scarce. In this article, we exploit the graph attention network to learn robust node representations of graphs in hyperbolic spaces. As the gyrovector space framework provides an elegant algebraic formalism for hyperbolic geometry, we utilize this framework to learn the graph representations in hyperbolic spaces. Specifically, we first use the operations defined in the framework to transform the features in a graph; and we exploit the proximity in the product of hyperbolic spaces to model the multi-head attention mechanism in the non-Euclidean setting; afterward, we further devise a parallel strategy using logarithmic and exponential maps to improve the efficiency of our proposed model. The comprehensive experimental results demonstrate the effectiveness of the proposed model, compared with state-of-the-art methods.

Structured Sequence Modeling with Graph Convolutional Recurrent Networks

Abstract: This paper introduces Graph Convolutional Recurrent Network (GCRN), a deep learning model able to predict structured sequences of data. Precisely, GCRN is a generalization of classical recurrent neural networks (RNN) to data structured by an arbitrary graph. Such structured sequences can represent series of frames in videos, spatio-temporal measurements on a network of sensors, or random walks on a vocabulary graph for natural language modeling. The proposed model combines convolutional neural networks (CNN) on graphs to identify spatial structures and RNN to find dynamic patterns. We study two possible architectures of GCRN, and apply the models to two practical problems: predicting moving MNIST data, and modeling natural language with the Penn Treebank dataset. Experiments show that exploiting simultaneously graph spatial and dynamic information about data can improve both precision and learning speed.

Temporal Graph Networks for Deep Learning on Dynamic Graphs

Abstract: Graph Neural Networks (GNNs) have recently become increasingly popular due to their ability to learn complex systems of relations or interactions arising in a broad spectrum of problems ranging from biology and particle physics to social networks and recommendation systems. Despite the plethora of different models for deep learning on graphs, few approaches have been proposed thus far for dealing with graphs that present some sort of dynamic nature (e.g. evolving features or connectivity over time). In this paper, we present Temporal Graph Networks (TGNs), a generic, efficient framework for deep learning on dynamic graphs represented as sequences of timed events. Thanks to a novel combination of memory modules and graph-based operators, TGNs are able to significantly outperform previous approaches being at the same time more computationally efficient. We furthermore show that several previous models for learning on dynamic graphs can be cast as specific instances of our framework. We perform a detailed ablation study of different components of our framework and devise the best configuration that achieves state-of-the-art performance on several transductive and inductive prediction tasks for dynamic graphs.

Discrete-time Temporal Network Embedding via Implicit Hierarchical Learning in Hyperbolic Space

Abstract: Representation learning over temporal networks has drawn considerable attention in recent years. Efforts are mainly focused on modeling structural dependencies and temporal evolving regularities in Euclidean space which, however, underestimates the inherent complex and hierarchical properties in many real-world temporal networks, leading to sub-optimal embeddings. To explore these properties of a complex temporal network, we propose a hyperbolic temporal graph network (HTGN) that fully takes advantage of the exponential capacity and hierarchical awareness of hyperbolic geometry. More specially, HTGN maps the temporal graph into hyperbolic space, and incorporates hyperbolic graph neural network and hyperbolic gated recurrent neural network, to capture the evolving behaviors and implicitly preserve hierarchical information simultaneously. Furthermore, in the hyperbolic space, we propose two important modules that enable HTGN to successfully model temporal networks: (1) hyperbolic temporal contextual self-attention (HTA) module to attend to historical states and (2) hyperbolic temporal consistency (HTC) module to ensure stability and generalization. Experimental results on multiple real-world datasets demonstrate the superiority of HTGN for temporal graph embedding, as it consistently outperforms competing methods by significant margins in various temporal link prediction tasks. Specifically, HTGN achieves AUC improvement up to 9.98% for link prediction and 11.4% for new link prediction. Moreover, the ablation study further validates the representational ability of hyperbolic geometry and the effectiveness of the proposed HTA and HTC modules.

Baselines.

Considering that HGWaveNet is constructed in the hyperbolic space for dynamic graphs, we choose seven competing baselines either in hyperbolic spaces or built for dynamic graphs to verify the superiority of our model.
The baselines are summarized in Table {{table:75359f93-e155-434e-b130-489497f37da4}} , where HTGN on Poincaré ball shows state-of-the-art performance in most evaluations.


PaLM {{cite:acb89ae}} + InstructGPT {{cite:8281e5f}}

ALFRED {{cite:2f6b7c9}} + BEHAVIOR {{cite:a096ca5}} + Physical

ProgPrompt: Generating Situated Robot Task Plans using Large Language Models

Abstract: Task planning can require defining myriad domain knowledge about the world in which a robot needs to act. To ameliorate that effort, large language models (LLMs) can be used to score potential next actions during task planning, and even generate action sequences directly, given an instruction in natural language with no additional domain information. However, such methods either require enumerating all possible next steps for scoring, or generate free-form text that may contain actions not possible on a given robot in its current context. We present a programmatic LLM prompt structure that enables plan generation functional across situated environments, robot capabilities, and tasks. Our key insight is to prompt the LLM with program-like specifications of the available actions and objects in an environment, as well as with example programs that can be executed. We make concrete recommendations about prompt structure and generation constraints through ablation experiments, demonstrate state of the art success rates in VirtualHome household tasks, and deploy our method on a physical robot arm for tabletop tasks. Website at progprompt.github.io

Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents

Abstract: Can world knowledge learned by large language models (LLMs) be used to act in interactive environments? In this paper, we investigate the possibility of grounding high-level tasks, expressed in natural language (e.g. "make breakfast"), to a chosen set of actionable steps (e.g. "open fridge"). While prior work focused on learning from explicit step-by-step examples of how to act, we surprisingly find that if pre-trained LMs are large enough and prompted appropriately, they can effectively decompose high-level tasks into mid-level plans without any further training. However, the plans produced naively by LLMs often cannot map precisely to admissible actions. We propose a procedure that conditions on existing demonstrations and semantically translates the plans to admissible actions. Our evaluation in the recent VirtualHome environment shows that the resulting method substantially improves executability over the LLM baseline. The conducted human evaluation reveals a trade-off between executability and correctness but shows a promising sign towards extracting actionable knowledge from language models. Website at https://huangwl18.github.io/language-planner

Do As I Can, Not As I Say: Grounding Language in Robotic Affordances

Abstract: Large language models can encode a wealth of semantic knowledge about the world. Such knowledge could be extremely useful to robots aiming to act upon high-level, temporally extended instructions expressed in natural language. However, a significant weakness of language models is that they lack real-world experience, which makes it difficult to leverage them for decision making within a given embodiment. For example, asking a language model to describe how to clean a spill might result in a reasonable narrative, but it may not be applicable to a particular agent, such as a robot, that needs to perform this task in a particular environment. We propose to provide real-world grounding by means of pretrained skills, which are used to constrain the model to propose natural language actions that are both feasible and contextually appropriate. The robot can act as the language model's "hands and eyes," while the language model supplies high-level semantic knowledge about the task. We show how low-level skills can be combined with large language models so that the language model provides high-level knowledge about the procedures for performing complex and temporally-extended instructions, while value functions associated with these skills provide the grounding necessary to connect this knowledge to a particular physical environment. We evaluate our method on a number of real-world robotic tasks, where we show the need for real-world grounding and that this approach is capable of completing long-horizon, abstract, natural language instructions on a mobile manipulator. The project's website and the video can be found at https://say-can.github.io/.

Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language

Abstract: Large pretrained (e.g.,"foundation") models exhibit distinct capabilities depending on the domain of data they are trained on. While these domains are generic, they may only barely overlap. For example, visual-language models (VLMs) are trained on Internet-scale image captions, but large language models (LMs) are further trained on Internet-scale text with no images (e.g., spreadsheets, SAT questions, code). As a result, these models store different forms of commonsense knowledge across different domains. In this work, we show that this diversity is symbiotic, and can be leveraged through Socratic Models (SMs): a modular framework in which multiple pretrained models may be composed zero-shot i.e., via multimodal-informed prompting, to exchange information with each other and capture new multimodal capabilities, without requiring finetuning. With minimal engineering, SMs are not only competitive with state-of-the-art zero-shot image captioning and video-to-text retrieval, but also enable new applications such as (i) answering free-form questions about egocentric video, (ii) engaging in multimodal assistive dialogue with people (e.g., for cooking recipes) by interfacing with external APIs and databases (e.g., web search), and (iii) robot perception and planning.

Summarization of recent effort to leverage LLM prompting techniques for Embodied AI.

Prior designs for EAI deployments have largely converged on a common architecture which leverages hard prompt learning techniques in conjunction with object detectors to generate an agent's next action prediction. A summarization of prior work is provided in Table {{table:47c0cba9-d70a-44b7-a479-d00d8d7a0740}} .


SemEval-2023 Task 10: Explainable Detection of Online Sexism

Abstract: Online sexism is a widespread and harmful phenomenon. Automated tools can assist the detection of sexism at scale. Binary detection, however, disregards the diversity of sexist content, and fails to provide clear explanations for why something is sexist. To address this issue, we introduce SemEval Task 10 on the Explainable Detection of Online Sexism (EDOS). We make three main contributions: i) a novel hierarchical taxonomy of sexist content, which includes granular vectors of sexism to aid explainability; ii) a new dataset of 20,000 social media comments with fine-grained labels, along with larger unlabelled datasets for model adaptation; and iii) baseline models as well as an analysis of the methods, results and errors for participant submissions to our task.

CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge

Abstract: When answering a question, people often draw upon their rich world knowledge in addition to the particular context. Recent work has focused primarily on answering questions given some relevant document or context, and required very little general background. To investigate question answering with prior knowledge, we present CommonsenseQA: a challenging new dataset for commonsense question answering. To capture common sense beyond associations, we extract from ConceptNet (Speer et al., 2017) multiple target concepts that have the same semantic relation to a single source concept. Crowd-workers are asked to author multiple-choice questions that mention the source concept and discriminate in turn between each of the target concepts. This encourages workers to create questions with complex semantics that often require prior knowledge. We create 12,247 questions through this procedure and demonstrate the difficulty of our task with a large number of strong baselines. Our best baseline is based on BERT-large (Devlin et al., 2018) and obtains 56% accuracy, well below human performance, which is 89%.

Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering

Abstract: We present a new kind of question answering dataset, OpenBookQA, modeled after open book exams for assessing human understanding of a subject. The open book that comes with our questions is a set of 1329 elementary level science facts. Roughly 6000 questions probe an understanding of these facts and their application to novel situations. This requires combining an open book fact (e.g., metals conduct electricity) with broad common knowledge (e.g., a suit of armor is made of metal) obtained from other sources. While existing QA datasets over documents or knowledge bases, being generally self-contained, focus on linguistic understanding, OpenBookQA probes a deeper understanding of both the topic---in the context of common knowledge---and the language it is expressed in. Human performance on OpenBookQA is close to 92%, but many state-of-the-art pre-trained QA methods perform surprisingly poorly, worse than several simple neural baselines we develop. Our oracle experiments designed to circumvent the knowledge retrieval bottleneck demonstrate the value of both the open book and additional facts. We leave it as a challenge to solve the retrieval problem in this multi-hop setting and to close the large gap to human performance.

PIQA: Reasoning about Physical Commonsense in Natural Language

Abstract: To apply eyeshadow without a brush, should I use a cotton swab or a toothpick? Questions requiring this kind of physical commonsense pose a challenge to today's natural language understanding systems. While recent pretrained models (such as BERT) have made progress on question answering over more abstract domains - such as news articles and encyclopedia entries, where text is plentiful - in more physical domains, text is inherently limited due to reporting bias. Can AI systems learn to reliably answer physical common-sense questions without experiencing the physical world? In this paper, we introduce the task of physical commonsense reasoning and a corresponding benchmark dataset Physical Interaction: Question Answering or PIQA. Though humans find the dataset easy (95% accuracy), large pretrained models struggle (77%). We provide analysis about the dimensions of knowledge that existing models lack, which offers significant opportunities for future research.

Statistics of datasets

We present statistics of the five commonsense reasoning (CSR) dataset we use in our experiments in Table {{table:616debc1-250a-4c49-81af-067ca666a45b}} .


Political inclination of YouTube channels and tags to characterize channel.

YouTube Video Dataset of Conspiracy Theory Videos from YouTube's Watch Next engine

Tweets related to COVID-19 misinformation with 17 different labels which include topics like Irrelevant, Conspiracy, True Treatment, Politics, Commercial Activity, etc.

7,085 channels with 2,365 of the channels with the tag conspiracy

Conspiracy (542) and non-conspiracy (568) in training set

4,573 annotated Tweets. Conspiracy (924)

Conspiracy Videos from a Book {{cite:cc84406}} or Reddit and non-conspiracy videos from random scraping

Annotation class determined by 1 single annotator.

Algorithmic Extremism: Examining YouTube's Rabbit Hole of Radicalization

Abstract: The role that YouTube and its behind-the-scenes recommendation algorithm plays in encouraging online radicalization has been suggested by both journalists and academics alike. This study directly quantifies these claims by examining the role that YouTube's algorithm plays in suggesting radicalized content. After categorizing nearly 800 political channels, we were able to differentiate between political schemas in order to analyze the algorithm traffic flows out and between each group. After conducting a detailed analysis of recommendations received by each channel type, we refute the popular radicalization claims. To the contrary, these data suggest that YouTube's recommendation algorithm actively discourages viewers from visiting radicalizing or extremist content. Instead, the algorithm is shown to favor mainstream media and cable news content over independent YouTube channels with slant towards left-leaning or politically neutral channels. Our study thus suggests that YouTube's recommendation algorithm fails to promote inflammatory or radicalized content, as previously claimed by several outlets.

A Longitudinal Analysis of YouTube's Promotion of Conspiracy Videos

Abstract: Conspiracy theories have flourished on social media, raising concerns that such content is fueling the spread of disinformation, supporting extremist ideologies, and in some cases, leading to violence. Under increased scrutiny and pressure from legislators and the public, YouTube announced efforts to change their recommendation algorithms so that the most egregious conspiracy videos are demoted and demonetized. To verify this claim, we have developed a classifier for automatically determining if a video is conspiratorial (e.g., the moon landing was faked, the pyramids of Giza were built by aliens, end of the world prophecies, etc.). We coupled this classifier with an emulation of YouTube's watch-next algorithm on more than a thousand popular informational channels to obtain a year-long picture of the videos actively promoted by YouTube. We also obtained trends of the so-called filter-bubble effect for conspiracy theories.

Characterizing COVID-19 Misinformation Communities Using a Novel Twitter Dataset

Abstract: From conspiracy theories to fake cures and fake treatments, COVID-19 has become a hot-bed for the spread of misinformation online. It is more important than ever to identify methods to debunk and correct false information online. In this paper, we present a methodology and analyses to characterize the two competing COVID-19 misinformation communities online: (i) misinformed users or users who are actively posting misinformation, and (ii) informed users or users who are actively spreading true information, or calling out misinformation. The goals of this study are two-fold: (i) collecting a diverse set of annotated COVID-19 Twitter dataset that can be used by the research community to conduct meaningful analysis; and (ii) characterizing the two target communities in terms of their network structure, linguistic patterns, and their membership in other communities. Our analyses show that COVID-19 misinformed communities are denser, and more organized than informed communities, with a possibility of a high volume of the misinformation being part of disinformation campaigns. Our analyses also suggest that a large majority of misinformed users may be anti-vaxxers. Finally, our sociolinguistic analyses suggest that COVID-19 informed users tend to use more narratives than misinformed users.

Related Datasets for Conspiracy Detection

Table {{table:e2f044ae-0e86-438c-b01c-1ef06e84d218}}  highlights several existing datasets that have been used for conspiracy theory detection research.
Existing literature often focuses on misinformation {{cite:9d185e0}}, {{cite:c3a64e5}} or
specific conspiracy theories related to COVID-19, alien visitation, anti-vaccination, white genocide, climate change, or Jeffery Epstein {{cite:587aff0}}, {{cite:c002298}}, {{cite:1eb3593}}. Most works focus mainly on Tweets as the unit of the study {{cite:587aff0}}, {{cite:aaf7b82}}, {{cite:1eb3593}}, {{cite:961256c}}.
For example, {{cite:aaf7b82}} study Tweets explicitly containing the word “conspiracy.” {{cite:1eb3593}} have compiled a dataset consisting of four types of conspiracy, namely climate change, COVID-19 origins, COVID-19 vaccine, Epstein Maxwell.
{{cite:587aff0}} study COVID-19-related conspiracies by training a BERT-based classifier to distinguish conspiracy Tweets.